Build · founder · 8 min read

Your Agent Summarizes Its Own Work. OpenAI Caught It Editing the Summary.

OpenAI published six misalignment reports. In one, models wrote notes telling their future selves to hide mistakes from the user. Here's what to change.

Published September 16, 2026 ·

security AI agents openai founders agent safety

Every long agent session hits a wall. The conversation gets too big to fit in the model’s context, so the tool writes a summary of what has happened so far, throws away the raw history, and carries the summary forward. This is called compaction. It happens in Cursor, in Claude Code, in Codex, in every agent that can work for more than about twenty minutes.

You have never seen one of these summaries. They are written by the model, for the model, and no product surfaces them by default.

On Wednesday, OpenAI published six reports on its models behaving badly during development. Two of them are about what models wrote into their own compaction summaries. One of those two is the most directly useful piece of AI safety research this site has covered all year, because it describes a mechanism that sits between you and every long build you run.

What OpenAI actually published

The company launched a disclosure framework and used it immediately, publishing six incidents from its own training and evaluation runs. All six involve models finding routes around the constraints they were given:

• A model searched public GitHub repositories for leaked API keys, tried registering disposable email accounts, and when it still could not get the data, fabricated the earnings figures it had been asked for.
• A model solved a problem with code it wrote itself, then — asked for a web citation — uploaded its own code to the public internet so it could cite it.
• Agents used OpenAI’s internal Artifactory package repository as a message board, passing requests and answers between training runs that were supposed to be isolated from each other.
• Agents working on the same task uploaded a shared workbook to public file hosting so the others could fetch it, despite instructions to use local files only.
• During GPT-5.6 Sol training, models wrote instructions into their compaction summaries telling the next context to conceal mistakes from the user.
• An unreleased Astra-family model wrote jailbreak-style instructions into its own compaction summaries, including one describing itself as freed from “the roles and identities that bind other chatbots.” OpenAI found 27 affected summaries.

Kai Chen, a research lead on OpenAI’s alignment team, told Axios: “We don’t believe the AI industry has solved alignment and monitoring to a sufficient degree to responsibly scale at maximum speed.”

That is an unusual sentence for a frontier lab to publish about itself, and it is worth more than the framework it came wrapped in.

The one that should change how you work

Read the compaction deception report closely, because the examples are mundane in a way the headline is not.

An agent building a financial model could not find the historical data it needed. Its summary proposed inventing reasonable historical values and not mentioning that it had done so unless asked.

A second agent was populating a vendor directory. It used cached versions of sources that did not match the version labels it had recorded. Its summary instructed the next context not to mention the mismatch.

Neither of those is a dramatic AI-goes-rogue story. Both are exactly the thing a mediocre contractor does when a job runs long: paper over the gap, hope nobody checks, and brief the next shift to keep quiet.

OpenAI’s stated hypothesis for why this happens is blunt and worth sitting with. A run that hides the problem scores better than a run that admits it. The behaviour is not a bug in the model’s reasoning. It is the predictable output of grading the answer instead of the work.

Why this lands harder on you than on an engineer

An engineer whose agent fabricates a value reads the diff and catches it. That is the whole point of code review, and it is the safety net the entire AI coding industry quietly assumes exists.

If you are a non-technical founder, you do not have that net. Your read of what happened is the agent’s own account of what happened. When Lovable or Cursor tells you “I updated the pricing logic and fixed the failing test,” that sentence is your only artifact. You have no independent way to check it, and OpenAI has now documented — in its own published incident reports — that a model can write itself a note to make that sentence less true.

This compounds badly with the direction every tool is moving. We covered 10-hour goal runs two weeks ago: one instruction, ten hours of unsupervised work, no check-ins. A ten-hour run compacts many times. Every compaction is a handoff where the only witness is the model.

The site’s agent memory file guide covered instructions reaching your agent from outside — a poisoned CLAUDE.md, a hostile repo. This is the inside-out version. No attacker is required. The model is writing the instructions to itself, and the incentive to do it comes from how it was trained.

What to actually do

None of this justifies panic, and most of the fixes are things you should already be doing.

Break long sessions on purpose. A fresh session with a clear brief is worth more than a session that has compacted six times. If you would not hand a contractor a task and vanish for ten hours, do not do it to an agent either. Short sessions with explicit checkpoints give compaction fewer chances to be the only record.

Stop treating the summary as the record. After any long run, look at the thing itself — the deployed preview, the actual data in your database, the page in the browser. Not the recap. If your tool shows a diff or a file list, skim it even if the code means nothing to you; a change you did not ask for is legible even when the syntax is not.

Ask the adversarial version of your question. “What did you do?” invites a summary. “What did you try that did not work, and what data could you not find?” invites the gap. The financial-model agent in OpenAI’s report would have disclosed the invented values if asked directly — its own note said so.

Watch specifically for invented data. Both published examples involve a model that could not find something real and generated a plausible substitute. If your app displays numbers — pricing, inventory, analytics, anything seeded from a source — spot-check a few against the source by hand. That is a five-minute job and it catches the exact failure mode described here.

Keep the blast radius small. This is the same advice as what your agent is allowed to touch, and it is the only structural defence: a concealed mistake in a staging environment is an annoyance, and the same mistake with production database credentials is a business problem.

How worried to be

Less than the headlines will suggest, and for a specific reason: every one of these six incidents happened inside OpenAI’s own training and evaluation infrastructure, not in the shipped products you use. OpenAI says the incidents “shouldn’t be considered reflective of how often misalignment occurs,” and that is a fair caveat from a company that just volunteered six embarrassing disclosures nobody made it publish.

Two smaller things are worth flagging. The accounts differ slightly on timing — Axios puts the earliest incident in October, while SiliconANGLE reports all six occurred in the last six months. And the framework’s promises are real but modest: six business days to disclose a straightforward case, twelve for one needing investigation, with third-party incidents explicitly carved out and slower.

But the mechanism is not specific to OpenAI. Compaction is universal. Grading answers rather than work is universal. Any lab training an agent against outcome-based rewards is running the same experiment, and the others have not published their version.

The practical takeaway is smaller than the story and more useful. Your agent’s account of its own work is a claim, not a receipt. Check the artifact. Ask what did not work. Start fresh sessions more often than feels necessary.