Details
- OpenAI announced new research on Deployment Simulation, a method for anticipating how models may behave in real-world use before release.
- The approach simulates deployment using recent, de-identified user requests and studies candidate model responses.
- OpenAI said it used only ChatGPT conversations from users who allow their data to be used to improve models.
- The company said it removed account-linked identifiers and identifiable information, and reported only aggregate findings.
- In tests across 20 behavior categories and three GPT-5-series Thinking deployments, simulated and observed rates were strongly correlated.
- OpenAI said the method outperformed challenging-prompt and prior-deployment baselines at predicting whether rates would rise or fall, and by how much.
- The research also found simulated deployments reduced evaluation awareness to levels close to real production traffic.
- OpenAI extended the approach to agentic deployments with stateful tools, saying tool simulators can produce realistic trajectories when given enough context and capability.
- OpenAI said traditional evaluations and red-teaming remain essential, and that Deployment Simulation is meant to complement them rather than replace them.
- The company noted a companion Alignment blog post exploring the public WildChat dataset, which is less precise but still useful as a signal.
Impact
The research points to a more operational style of model evaluation, where developers try to estimate real-world behavior before release rather than relying only on benchmark prompts and red-teams. If the results hold up outside OpenAI’s own data, the method could improve risk forecasting for frontier models and agentic systems, while also reinforcing the value of access to production-like traffic. It also highlights a practical advantage for large labs with first-party usage data, which smaller evaluators may not have.
