Ex-OpenAI Head of Preparedness: Don’t Slow AI Down, Make it Safe Faster

Today I interviewed Joaquín Quiñonero, who led Preparedness at OpenAI, testing frontier models for catastrophic risk, and later ran recruiting and people there. He spent nearly 20 years building AI at Microsoft, LinkedIn, and Meta, where he founded Facebook’s applied machine learning team.

The twist is not another lab. In 2016 he was not worried about AI danger, because the systems were hard to build. A decade later he was the person looking for catastrophic harm, and he still thinks “slow the frontier” is the wrong slogan. He would rather speed up safety research, refuse to ship what a lab cannot monitor, and tell founders to burn tokens on one bet instead of funding a Bell Labs inside a startup.

Structure and Key Highlights

Spend on Tokens, Not Headcount

• The old advice was not to over-raise before product-market fit. A $20 million pre-seed with two founders gets wasted on hiring and side quests, and too many investors is a kiss of death
• The share of the budget that goes to tokens, instead of salaries, has to grow
• Burning compute lets a small team test one hypothesis faster than anyone else

Do Not Do Research Just to Impress Investors

• Most early startup research teams are investor bait: Quiñonero sees them at Series B, Runway among them
• Research that moves a field, Bell Labs style, needs years of funding without pressure, and a startup that demands the roadmap loses the goose
• What belongs in a startup is applied research that improves its own product, such as a 10x cut in inference cost

Stay Close to the Work, Whatever Your Title

• As companies grow, founders drift from customers and from the technology
• At Facebook he left leading applied machine learning to write code again, and at OpenAI, offered a much bigger role after Preparedness, he asked to be an intern
• Careers are helixes, not ladders: once titles such as principal, distinguished, fellow, or SVP exist, you cannot go back to the work

No Single Culture as the Operating System

• Born in Spain, raised in Morocco in a Spanish family. His mother taught him Spanish at home, sure they would move back the next year
• A French school by day, Arabic with friends, Friday prayer traffic, Jewish friends, and diplomats at the house, including a Romanian doctor trained in Russia
• He was not conscious of it then. Later, rigid routines confuse him: no single culture is the operating system

Snails, a Needle, and Risk

• In Tangiers he ate street snails with a needle stored in a lemon, reused by the next customer
• His mother put bleach in the water for market salad and was horrified
• The lesson: experience things. Risk is not one rule that travels

Spain Was Supposed to Feel Like Home

• At 18 he turned down a French prépa and a shot at a grande école like Polytechnique, and went to Spain to find his roots
• He felt like a stranger. Classmates had one city and one school. His family had lived in three Moroccan cities
• Madrid engineering made him want out. Erasmus, with one rule: as far away as possible

Copenhagen, 2000

• He met his wife there. Next year they celebrate 25 years married
• A professor teaching neural networks offered a funded PhD. He had not planned one. He asked what a neural network was
• His mother: en esta vida no hay nada seguro. Nothing in this life is sure

The Spanish Table Abroad

• In Cambridge, around 30, he joined a veterans rugby team and got beaten up. That is his version of showing up
• Spanish communities in Germany, Denmark, and England felt insular, organized around jamón and olive oil. He wanted them to integrate, and he knows the tension with staying yourself
• California fit. A Spanish-German couple, three trilingual kids, and nobody was surprised. Discomfort is part of the startup job

A Helix, Not a Ladder

• The résumé looks backward: applied ML lead at Facebook, then IC, then the same drop at OpenAI, then leadership again
• He calls it a helix. You circle, and you are not the same person when you pass the old spot
• Management made him speak lines that fit any meeting. After each drop, peers said they had wanted to do it and could not

“I Want to Be an Intern”

• After Preparedness, OpenAI offered a much bigger role. Over Christmas he said no and asked to be an intern. They let him
• Large-team peers, after a third beer, said they were jealous and stuck
• The last IC stretch was Codex and new coding agents. The one before had no coding tools. You cannot lead that from memory

A Return Ticket at Microsoft Research

• Around 2007, Research felt like paradise: pay, academic freedom, no grants, no production code
• Engineering leadership dulled his research edge. He asked for tours of duty, with a way back
• People leave, he thinks, when the only option is to blow it up. AI makes self-renewal mandatory, not a perk

Titles Kill the Helix

• OpenAI stayed on “member of technical staff” for a long time, on purpose
• Principal, distinguished, fellow, SVP: once the names exist, you cannot go back to the work
• Twenty people do not need the ladder. The machine that arrives later takes the soul out. He became head of people without claiming to have the design answer

Presumption of Responsibility

• He joined Facebook for what one engineer could do, including redeploying the whole site with no permission meeting
• About 1,000 technical people, maybe 3,000 total. Trust them, and they own the outcome
• Rules make a kid misbehave. Real responsibility makes them careful. Scale swaps that for checks so you can hire anyone. He calls it tragic

Dunbar Still Explains the Labs

• Culture gets the credit, and headcount does a lot of the work. At 200 or 300 people you still know the faces
• Friends at Meta and Google describe the opposite of that Facebook. He treats it as what anonymity does

10,000x, Fewer People

• The top 1% or 0.1% will not be 10x or 100x. His guess is closer to 10,000 times, because they build factories of agents, not one answer
• A new hire costs more in coordination than before. Small teams, or small units inside a big company, can carry absurd output

Tokens, Then One Bet

• Enzo’s old rule still matters to him: a $20 million pre-seed with two founders gets wasted on hiring and side quests. Too many investors is a kiss of death
• The new line item is tokens. That share of spend has to rise so a small team can test one hypothesis faster than anyone else
• Do not do 20 things. Do one thing 100 times faster. Enzo’s picture, five people at $1 million of tokens each per year, means the seed is simply bigger

Not Bell Labs

• Enzo sees research orgs at Series B, Runway among them. Quiñonero asks how much is investor bait
• Bell Labs was famous scientists and a long, shielded budget. A startup that demands the roadmap, or cuts the goose open to find the egg, loses the goose
• Fundamental research has no place there. Applied work does: a 10x cut in inference cost, or the speech product he uses daily. Twenty or thirty new labs do not change the split

What 2016 Could Not See

• That year he was not very worried. The systems were hard to build. Eight years later he ran Preparedness
• The old risks were bubbles, over-personalization, and people gaming the feed. Gradual, visible, human
• GPT-3 and a capability jump changed the map. He takes Yoshua Bengio’s three buckets from a United Nations speech days earlier: misuse by people, loss of control, concentration of power. Only the third was already obvious in 2016

Draw the Catastrophic Line

• The word is arbitrary until someone sets it
• His sketch: single-digit days, deaths in the thousands or millions, and/or billions to hundreds of billions in damage
• Preparedness was that threshold for AGI and beyond, not a mood

The Facebook Rhyme

• During the Arab Spring, which he places largely in Tunisia, he thought the feed was a voice for everybody
• Freedom of speech had been built for radio and television
• AI rhymes on speed: society has not absorbed it. It does not rhyme on blast radius, so the 2016 answer is retired

Data Centers, and Being Left Out

• US resistance runs through blue states and red states
• Water depends on the generation of the building, the grid needs more power, and the sites are loud up close. He thinks those are solvable
• The fight underneath is time. People were not in the conversation, and the change is already on them

The Arms Race He Will Not Decorate

• No lab leader wants to be the one who shipped the destructive system. They still build the next model. Enzo says Icarus. Quiñonero keeps the image
• Testing the risk can require a stronger model. Defending against a dangerous AI by building a stronger one on your side is, in his words, an escalation
• China is the obvious race, economy and defense together. The capital version is a hypothetical with Andreessen: we are slowing down, now wire the money
• He is sympathetic. He still calls the structure a prisoner’s dilemma

Do Not Pace It

• “Pacing the frontier” is the lazy slogan. He wants more testing and more reliability before release, which will not trend
• The research to accelerate is alignment of many agents, not one. Collaboration is how you get to superintelligence, and it is where sub-goals show up that nobody planned
• The Hugging Face and OpenAI incident matters to him for that agent behavior. The Hugging Face branding is a distraction
• Do not deploy, at scale, a model you cannot monitor, align, control, and turn off. A new jet engine does not fly passengers without the hours

Radiologists, and the 10% Line

• Geoffrey Hinton’s 2016 line was that radiologists would not be needed. A decade on, there is a shortage, and Hinton has walked some of it back
• The forecast stared at one subtask, finding the tumor, and missed history, risk, and treatment choice
• Jevons paradox did the rest: cheaper reads created more demand. Quiñonero wants these as questions, not as “ten years, 10% chance humanity ends”

Out of Lane

• Dario or Sam saying it strips the nuance. People hear the job title
• A winning football coach in Spain gets asked about the economy. Quiñonero’s advice is to decline
• Same joke, pointed: ask Mourinho whether AI takes his job

The People Job Was Not Safety

• Recruiting, then people, was a call of duty after the intern stretch. He is explicit: not a safety strategy
• The story he still tells is Yo-Yo Ma at NeurIPS, around 2018, after Quiñonero had attended for twenty years starting in 2000 or 2001
• Trust, Yo-Yo Ma said, starts with who built the system and what they want. That is an answer about safety. It is not why he took the job

Incidents, and Who Reports to the CEO

• He wants transparency on safety research before he wants another warning quote
• From the Partnership on AI board, with Eric Horvitz chairing and also chief scientist at Microsoft, he keeps the incident database: nonprofits, every lab, failures written down
• Enzo says an Interpol of rogue agents. Quiñonero agrees, including as training data. Banks already trade breach lessons in private so they do not tutor attackers
• Cognitive diversity is the other half, and he traces it to Morocco. The safety and alignment lead should report to the CEO. A monoculture is how this fails

Liability Is the Product

• Enzo: intentions will not do it. Someone has to profit from alignment
• Hospitals, law firms, and banks are already writing the clause: if it goes wrong, the vendor pays
• Volvo’s carrot was the safe car, Apple’s was privacy. The stick is liability, and insurance is who he expects to price it

Disentangle Fair

• The Morocco thread lands in fairness work started at Facebook and shipped at LinkedIn
• Equality of treatment makes the model blind. Equality of outcomes sometimes requires a different treatment, the way a parking garage fails if every spot is identical
• Women on LinkedIn more often had thinner profiles. Men applied above their qualifications, women more conservatively, and the recommender could offer an equally qualified woman a lower-paid job
• They refused to pick one definition and sprint. The ranker stays blind. The product helps finish the profile. He cannot remember the paper title on the show. He remembers that move-fast was the wrong instinct

Classroom First

• He left OpenAI in May. The company he is building can wait for another episode. This one gets education
• He accepts the fear of lazy students and of grading work that is 90% model. He still wants tutoring that matches how a person learns
• Jo Boaler at Stanford is the reference, title forgotten on air: visual, analogy, or literal, and a lot of “bad at math” was the wrong lesson
• He would rather teach math in a Mountain View high school before he tells the school what to buy. Enzo hears the intern pattern again. Most people, Enzo adds, are never shown how they learn. Listening is why this podcast exists

Two Women, and Uncertainty

• The people who pushed him: his mother and his wife. Both can live with not knowing
• His wife will not judge. He writes someone off, she says we do not know the context, and the discomfort is the gift
• That is what made the next country, and the next job, possible

Silicon Valley Glossary

People, Companies and References

People Mentioned

• Joaquín Quiñonero: led Preparedness at OpenAI, then recruiting and people; built AI at Microsoft, LinkedIn, and Meta
• Enzo Cavalie: podcast host
• Yoshua Bengio: researcher whose United Nations remarks supply the three risk buckets Quiñonero uses
• Geoffrey Hinton: source of the 2016 radiologist forecast, and of a later walk-back
• Yo-Yo Ma: at a NeurIPS workshop, said trust in an AI starts with who built it and why
• Eric Horvitz: Microsoft’s chief scientist and Partnership on AI chair, who pushed an incident database
• Jo Boaler: Stanford professor whose work on learning styles frames his interest in AI and education
• Andreessen: the investor in the hypothetical of asking for money while promising to slow down
• Mourinho: the coach in the joke about answering outside your lane

Companies and Organizations

• OpenAI: Preparedness, then an intern stretch, then recruiting and people, then he left in May
• Meta: where friends now describe the opposite of the Facebook he joined
• Facebook: applied machine learning, and a culture of individual ownership
• Microsoft: Research around 2007, and the engineering seat he wanted a return path out of
• LinkedIn: where fairness became a product question about profiles and job recommendations
• Netflix: his example of an early culture that worked before the machine set in
• Bell Labs: the fundamental-research picture a startup cannot fund
• Hugging Face: the public incident he sets aside so the agent behavior can be discussed
• Partnership on AI: the board he sat on, and the home of the incident-database push
• NeurIPS: the conference he attended for about twenty years
• United Nations: where Bengio, days earlier, laid out the three risk categories
• Runway: one application company Enzo cites as building research earlier than the old playbook
• Apple: the privacy brand, paired with Volvo’s safety brand
• Volvo: the company that made safety the product
• Stanford: where Jo Boaler teaches
• Codex: a coding agent from his last stretch as an individual contributor

Data and References

• 20 years: building AI across Microsoft, LinkedIn, and Meta, and also his run of NeurIPS
• 25 years: the anniversary he and his wife reach next year
• Single-digit days: the time window inside his sketch of catastrophic harm
• GPT-3: the release between the 2016 interview and the risks he sees now
• $20 million: Enzo’s example of a pre-seed that used to be too much for two founders
• $1 million: tokens per person, per year, on the five-person team Enzo pictures
• 10,000 times: the impact gap he expects between the very top and the lower tail
• 90%: student work done by a model, and the grading problem that follows