OpenAI establishes "Preparedness" team to mitigate AI risks

OpenAI’s risk management: OpenAI has announced the creation of a specialised team known as "Preparedness" to address and safeguard against what it terms "catastrophic risks" associated with AI models. This team will be under the leadership of Aleksander Madry, who currently serves as the director of MIT's Center for Deployable Machine Learning and has been with OpenAI since May in the capacity of "head of Preparedness," according to his LinkedIn profile.

Preparedness will have a primary focus on monitoring, forecasting, and defending against potential threats stemming from future AI systems. These threats encompass a broad spectrum, ranging from AI's capacity to influence and deceive humans (as seen in phishing attacks) to its potential for generating malicious code.

OpenAI CEO Sam Altman has been vocal about his concerns regarding the potential risks of AI, even suggesting that AI could pose an existential threat to humanity. The establishment of the Preparedness team signals a proactive step toward addressing these concerns, including those that might seem reminiscent of scenarios from science fiction dystopias. OpenAI is actively interested in researching both apparent and less obvious AI risks.

Input invited from the community

To coincide with the launch of the Preparedness team, OpenAI is inviting input from the wider community for risk studies. This initiative offers a prize of $25,000 and an opportunity to join the Preparedness team to the top ten submissions.

The Preparedness team will also be responsible for developing a "risk-informed development policy" that outlines OpenAI's approach to evaluating AI models, monitoring tools, risk-mitigation strategies, and governance mechanisms across the model development lifecycle. This policy aims to complement OpenAI's ongoing efforts in AI safety, encompassing both the pre-and post-model deployment phases.

The announcement of the Preparedness team was made during an AI safety summit hosted by the UK government. It follows OpenAI's earlier commitment to establish a team focused on researching, guiding, and managing emerging forms of "superintelligent" AI.