Paradigms of Artificial General Intelligence and Their Associated Risks


Funded by the Future of Life Institute, through the FLI International AI Safety Grants Competition.

Humans are considered to have general intelligence. Endowing machines with artificial general intelligence (AGI) would allow them to adapt to a variety of situations, maximising their potential. Looking ahead, there are likely to be areas of scientific and intellectual progress that will require the types of planning, abstract reasoning, and meaningful understanding of the world that we associate with general intelligence.

The big question is whether the potential given by further generality also implies higher risks and unknowns in comparison with the more specialised, constrained, systems we are already used to. Different issues are raised by a machine translation system, specialised for a task, compared to a versatile personal assistant at home, aimed at covering more and more daily tasks. If this association between risk and generality exists, can we find trade-offs or limitations that ensure flexibility and robustness at the same time?

This project investigates this and other related questions for several scenarios, including the profiles of generalised automation that are both safe and efficient, and the understanding of social dominance through general mind modelling, according to several AGI paradigms. The result is a systematic cartography of the risks of AGI from the perspective of AGI’s definitional concept: generality.

Contact: Principal Investigators: José Hernández-Orallo and Seán Ó hÉigeartaigh.


Many paradigms exist, and more will be created, for developing and understanding AI. Under these paradigms, the key benefits and risks materialise very differently. One dimension pervading all these paradigms is the notion of generality, which plays a central role, and provides the middle letter, in AGI, artificial general intelligence. This project explores the safety issues of present and future AGI paradigms from the perspective of measures of generality, as a complementary dimension to performance. We investigate the following research questions:

  1. Should we define generality in terms of tasks, goals or dominance? How does generality relate to capability, to computational resources, and ultimately to risks?
  2. What are the safe trade-offs between general systems with limited capability or less general systems with higher capability? How is this related to the efficiency and risks of automation?
  3. Can we replace the monolithic notion of performance explosion with breadth growth? How can this help develop safe pathways for more powerful AGI systems?

These questions are analysed for paradigms such as reinforcement learning, inverse reinforcement learning, adversarial settings (Turing learning), oracles, cognition as a service, learning by demonstration, control or traces, teaching scenarios, curriculum and transfer learning, naturalised induction, cognitive architectures, brain-inspired AI, among others.

You can find a more detailed description of the project in the abridged project proposal PDF.


John Burden was appointed to the postdoc position starting 1 July 2020. John has a background in Computer Science and is in the final stages of completing his PhD from the University of York. John also holds a Master’s degree in Computer Science from Oriel College, Oxford


The Associates strengthen the expertise and outreach in several areas of the project:

 The International Advisory Board (IAB) of the project is composed of the following people: