Resources

High-level, non-technical overviews of the arguments supporting caution regarding advanced AI systems.

• The current landscape, by Anthropic

• Intro to AI safety, by Robert Miles

• Article on catastrophic AI risks, by Center for AI Safety

• Cold Takes, a blog by Holden Karnofsky, ex-CEO of Open Philanthropy

  • Why Would AI “Aim” to Defeat Humanity?

  • AI Could Defeat all of us Combined

  • How We Could Stumble into AI Catastrophe

  • AI Safety Seems Hard to Measure

  • High-Level Hopes for AI Alignment

• Robert Miles highlights:

  • Instrumental Convergence

  • Intelligence and Stupidity: The Orthogonality Thesis

More in-depth explanation of arguments designed for technical researchers.

• Paper on the alignment problem from a deep learning perspective

• Paper on a research agenda for AI governance

• Paper on an evolutionary framing of AI risks

AI Safety in the news:

• A.I. Poses ‘Risk of Extinction,’ Industry Experts Warn (Kevin Roose, New York Times)

• Geoffrey Hinton tells us why he’s now scared of the tech he helped build (Will Douglas Heaven, MIT Technology Review)

• The Aliens Have Landed, and We Created Them (Niall Ferguson, Bloomberg)

“Humanity is not ready for what will happen next. Not socially, not economically, not morally.”

Careers

Some curated links we think students and researchers will find most helpful for finding career and funding opportunities in AI safety. 

Comprehensive lists: AISafety.com | 80,000 hours | Emerging Tech Policy | Kairos

To get started in policy work: Short-term policy programs | Internships | Fellowships

To get started in technical research:

• Supervised Program for Alignment research (SPAR): highest capacity, most entry-level friendly, runs triannually.  Typical outcome is a research blogpost/workshop paper.  Apply to your top 2-3 projects.  AISI helped get this off the ground and is currently run by our friends at Kairos.

• Machine Alignment Theory Scholars (MATS): lower capacity, but has the best mentors (e.g., Anthropic, OpenAI, DeepMind).  May require research and engineering experience.

• University of Chicago XLab Fellowship: full-time summer program, low capacity.  Good fit for students who can execute on their own projects/ideas.

To get started with independent project funding: 

• AISafety.com funders list: resources on everything from technical research to field building.

• Anthropic Economic Futures Program: supports research and policy development focused on addressing AI’s economic impacts.

• UK AISI Challenge Fund: award grants of up to £200,000 per project to address pressing, unresolved questions in AI safety and security. 

To schedule a consultation with us, please email us your CV/resume and describe what types of opportunities you are searching for.