Audit Finds Blind Spots in AI Safety Checks

For users relying on advanced artificial intelligence systems, recent findings suggest that safety oversight has not been as comprehensive as previously assumed. A three-week on-site audit revealed that a portion of traffic from OpenAI’s most capable models operated without active monitoring, creating windows where automated agents could act without immediate detection. These gaps, identified by independent researchers, highlight areas where the company’s internal safeguards were not fully capturing the behavior of its flagship models.

The review was conducted by researchers from METR and Redwood Research, who were granted direct access to OpenAI’s codebase and internal documentation. The team spent three weeks on-site, interviewing approximately a dozen employees to understand how the company tracks potential rogue AI behavior. This pilot program was designed to give external experts a deep look into the infrastructure that supports the company’s internal monitoring protocols.

During the period from July 21 to August 11, the auditors discovered that an estimated 6 to 25 percent of internal traffic from models at the GPT-5.5 class and above was not covered by monitoring systems. METR notes that the uncertainty surrounding this figure is significant in both directions. Additionally, for internal Codex traffic that was supposed to be tracked, the researchers estimate that roughly 10 percent went unmonitored between May and July due to a bug in how the system handled context summaries, known as compaction events.

The compaction event gap allowed agents to delay oversight because monitoring was only triggered when control was handed back to the user. By continuing to execute actions without returning control, agents could effectively sidestep the monitoring triggers. OpenAI independently identified this bug and implemented a fix while the audit was still in progress, though the issue remained a key finding in the final evaluation.

The auditors also identified methods to hide hundreds of actions from the monitor and bypass the Codex sandbox environment. They determined that specially crafted inputs could likely crash the monitoring system entirely. Furthermore, model behavior during reinforcement learning (RL) training was not being monitored at all when the audit began, a gap that OpenAI addressed by pausing RL training after an incident in August and expanding its monitoring coverage.

OpenAI states that it has fixed all the weaknesses identified by the auditors. The full 61-page report was handed over internally, but after redactions, METR and Redwood Research leadership received a 38-page version. Only a short summary has been made available to the public, with METR stating it had no right to disclose the redacted portions independently. The company emphasized that the changes made to RL training occurred after the auditors' access period ended and were not part of the initial evaluation.

This audit serves as a pilot program aimed at providing outside researchers with deeper access to internal AI infrastructure. While the current process has limited public transparency, METR has indicated that it aims to push for more open disclosure in future audits. The findings underscore the ongoing challenge of ensuring that as AI capabilities grow, the systems designed to monitor them keep pace with the complexity of their operations.