We are entering the era of artificial general intelligence (AGI) and foresee beyond it to superintelligence (SI). Assuring safe interaction between AGI/SI and humans is increasingly urgent and important. Ideally, safe interaction means provably safe, i.e. via mathematical proofs, the laws of physics, or ''guaranteed safe' via modeling and simulation. We focus on ’high-risk’ AGI/SI. Realistically, we can estimate AGI/SI timelines but must implement the state-of-the-art safety tech at all times.
SuperIntelligence™ (SI) is open-access. We continuously publish key developments in AGI/SI and AGI/SI safety: theory, technology, and governance. Submissions of articles, expert commentary, editorials, letters, and book reviews are welcome. There are no article processing charges and we emphasize public commentary over traditional anonymous peer review. Authors retain copyright, typically under CC-Attribution-NoDerivs 4.0, which permits further distribution of their work.
Takeaways from Jean-Noël Barrot (Minister for Europe and Foreign Affairs, France – Council President),
Yoshua Bengio (Co-Chair, UN Independent International Scientific Panel on AI), Sam Altman (Chief Executive Officer, OpenAI), Dario Amodei (Chief Executive Officer, Anthropic), and Clément Delangue (Chief Executive Officer, Hugging Face), plus commentary by Gary Marcus.
Critical to the advent of AGI is AI's ability to creatively solve computer science, general science, and mathematics problems. Recent advances in applying the most advanced models to science, mathematics, and automated theorem proving appear significant.
Takeaways from Jean-Noël Barrot (Minister for Europe and Foreign Affairs, France – Council President),
Yoshua Bengio (Co-Chair, UN Independent International Scientific Panel on AI), Sam Altman (Chief Executive Officer, OpenAI), Dario Amodei (Chief Executive Officer, Anthropic), and Clément Delangue (Chief Executive Officer, Hugging Face), plus commentary by Gary Marcus.
Anthropic CEO Dario Amodei envisions a 'country of geniuses in a datacenter' with these five enabling properties:1) smarter than top humans across most domains, 2) able to act autonomously over long horizons, 3) use digital tools, 4) coordinate many copies, and 5) operate at much higher speed than humans. You could use these as a checlist to develop such a system, as Anthropic probably does. #5 is true in some domains (real-time learning being a counter-example), #3 is well in progress, and #1, #2, and #4 are still significant obstacles. He does not mention, e.g., robust generalization, abstraction, and world models. The essay discusses risks, governance, and economic implications. The essay’s overall thesis is: AI’s upside remains enormous, but humanity must treat the next few years as a civilizational test requiring technical alignment work, pragmatic regulation, geopolitical realism, economic adaptation, and moral seriousness.
AGI will render today's liability-based AI regulation obsolete through its ability to circumvent cybersecurity, hide its origins, and act strategically—but it will also enable a new regulatory paradigm based on mathematically provable contracts.
SuperIntelligence co-founding editor Roman Yampolskiy interviewed at length on Joe Rogan. Over 800,000 views.
SuperIntelligence co-founding editor Steve Omohundro was one of three recipients of the prestigious FLI Award 2024 award recognizing seminal contributions to AI safety: "...for laying the foundation of modern ethics and safety considerations for artificial intelligence and computers."
