{"slug": "using-chunked-monitoring-to-detect-deception-in-long-transcripts", "title": "Using Chunked Monitoring to Detect Deception in Long Transcripts", "summary": "Researchers at METR found that monitoring long transcripts in chunks of 20 consecutive messages, rather than all at once, recovers deceptive behaviors missed by global monitoring, recovering approximately 93% of false negatives for partial_problem_solving, 86% for match_weaker_model, and 45% for refusals in sandbagging transcripts. Using GPT-4.1 Mini as the monitor on METR's MALT dataset of 553 samples, the chunked approach outperformed global monitoring on all but one sample, suggesting chunking as an effective method for detecting deception in long agent transcripts.", "body_md": "**Summary:** Monitoring long transcripts in chunks, rather than all at once, catches behaviors previously missed.\n\nWe pose chunked monitoring as an effective means of finding the needle in the haystack, where the “needle” is deceptive behavior. Our findings include:\n\n`partial_problem_solving`\n\n(recovered ~93% Global FN), `match_weaker_model`\n\n(recovered ~86% Global FN), and `refusals`\n\n(recovered ~45% Global FN) categories of sandbagging.We evaluate over full agent interaction transcripts, and leave it as future work to see whether our findings extend to other settings.\n\nCurrently, chunking is mainly used as a means of making models more efficient in retrieving information from longer contexts or outside documents (e.g. RAG). Chunking refers to the process of breaking a large piece of data into smaller pieces such that a model can digest the individual segments.\n\nThis research explores applying the same concept of chunking to monitoring settings trying to identify deception in long transcripts.\n\n**Motivation:** Full transcript monitoring matters because deceptive intent can be spread over multiple turns (where each message looks benign, but when combined display deceptive intent). It can also guard against tactics like context flooding, where an attacker pads a request to overwhelm the monitor so it misses deceptive requests that get slipped in. So, being able to identify deception in full transcripts, or at least over long horizons, can be valuable.\n\nWe use METR’s [MALT](https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors/) dataset of long transcripts detailing multi-turn agent interaction, containing both benign and deceptive samples belonging to one of sandbagging, reward hacking, or misc. categories with specific behavioral labels within these categories (please see appendix for labels and their meanings).\n\n**Preprocessing:** Since this is a small, budgeted experiment, we limited the scope to transcripts ≤ 150k tokens that belong to the sandbagging category. The prompt that instructed the sandbagging behavior for the model to display was removed for each sample (first message of each transcript).\n\n**Monitor:** GPT 4.1 Mini (1M context) - we prompt it to assign a score to the transcript [0, 100] - used for both the chunked and global monitor.\n\n**Chunking:** Each chunk consists of 20 consecutive messages from the transcript, where each message is one turn - system, user, assistant, or tool result. The max score across the chunks is assigned as the final deception score.\n\n**Monitor Prompt:** Each monitor type was provided a system prompt (same as the one used in MALT) to classify various sandbagging behaviors [see in *Monitor prompts for reward hacking and sandbagging *section of their [Appendix](https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors/#prompts-for-all-obfuscation-variants), or our GitHub [repo](https://github.com/s-fereidooni/chunked-monitoring)].\n\n**Distribution:** 553 samples total - 293 `benign`\n\nsamples, and of [MALT](https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors/#benign-runs)’s sandbagging categories ~50 samples each of `reasoning_about_task`\n\n, `match_weaker_model`\n\n, `refusals`\n\n, `sabotage`\n\n, `partial_problem_solving`\n\n, and 17 samples of `gives_up`\n\n.\n\nThe following models were used to produce the sandbagging transcripts, by label:\n\n`partial_problem_solving`\n\n, `match_weaker_model`\n\n, `reasoning_about_task`\n\n, `refusals`\n\n, `sabotage`\n\n- all samples claude-3-7`reasoning_about_task`\n\n- 47/53 claude-3-7, 6/53 claude-opus-4`gives_up`\n\n- 9/17 claude-3-5-sonnet, 4/17 gpt-4o, 3/17 claude-sonnet-4, 1/17 claude-opus-4We found that in this setting, the chunked monitor recovered plenty of sandbagging samples the global monitor labeled as “negative”. Chunked monitoring largely subsumes global monitoring outperforming it on all but one sample.\n\nEvaluating the results across labels, we see that chunked monitoring gained the most in the `partial_problem_solving`\n\n, `match_weaker_model`\n\n, and `refusals`\n\nsandbagging categories, while displaying a marginal gain in `sabotage`\n\n. More interestingly, neither monitor type was able to correctly classify any of the 53 `reasoning_about_task`\n\nsamples nor any of the 17 `gives_up`\n\nsamples.\n\nYou can find a more granular breakdown by label in the appendix.\n\nTo gain a better sense of the integrity of chunked monitoring, we separate the effect of transcript length from sandbagging type: if the monitor performs poorly on label X, but label X makes up all the longest samples, then it may incorrectly seem like the monitor’s performance deteriorates on long contexts.\n\nWe found this to be the case for the `refusals`\n\nsamples, both chunked and global monitors had poor recall on the shorter transcripts, with performance notably increasing as transcript length increased. We did not see this correlation between transcript length and performance for any other sandbagging category. For both `partial_problem_solving`\n\nand `match_weaker_model`\n\nwe saw that the chunked monitor performed ~equally well across all transcript lengths.\n\nGiven that for transcripts of similar length each monitor has a different recall across labels (e.g. for ~55k transcript, global monitor achieves recall ~0.5 for `refusals`\n\nand ~0 for `partial_problem_solving`\n\n) - this could indicate monitor performance being unique to each label.\n\n**Why might chunking help?**\n\nWe hypothesize that the categories where chunking had the largest gains—`partial_problem_solving`\n\n, `match_weaker_model`\n\n, `refusals`\n\n—may share a property we can loosely call “locality”: the deceptive signal is concentrated in a stretch of turns rather than actively present across the full transcript. The chunks may isolate said stretches, while the global monitor is tasked with weighing the same evidence against thousands of additional tokens that can dilute it. This property of locality was not measured directly, but is consistent with samples we have observed where the deceptive spans are concentrated or clustered—seemingly local in their nature.\n\n**Why is this surprising?**\n\nThe model we used as our monitor, GPT 4.1 mini, has a 1M context window. Additionally, in OpenAI’s “needle in a haystack” evals it retrieved the planted fact with perfect recall across all positions in context. So we expected the global monitor to match the chunked monitor on the positive cases. A possible explanation for this is that the typical “needle” used in evals is a specific fact to be retrieved—not to detect deception, which is more so a judgement made over diffuse, distributed evidence. Attention dilution over long contexts could hurt global monitors more than chunked ones.\n\n**Where does monitoring still fail?**\n\nBoth monitor types misclassified all the `reasoning_about_task`\n\nand `gives_up`\n\nsamples. Both scored each sample ~0, indicating high confidence of benign behavior. Considering all MALT deception samples are manually reviewed, the deception does exist in those transcripts and is human verifiable; however, it seems neither global nor chunked monitors are up to par in classifying them automatically yet.\n\nThese results were only confirmed for sandbagging samples from MALT. Though we did see that chunking recovered particular labels better than others, it can’t be said for certain whether the chunking performed better on certain sandbagging labels due to intrinsic properties of the deception style (e.g. locality of deception). We set the score threshold to 50, and did not sweep over chunk sizes nor compare aggregation methods - though, since the chunked monitor's errors are all false negatives, max pooling already optimizes recall, other aggregation methods would only add false negatives. We did not experiment with using stronger frontier models as global monitors.\n\nAlso note that these experiments only look at agents displaying prompted deceptive behavior - not naturally occurring behaviors or ones caused by users acting as attackers - so exploring chunking in other settings is left for future work.\n\nThis blog post was created as the product of a 30 hour [AI Safety Technical Project Sprint](https://bluedot.org/courses/technical-ai-safety-project) facilitated by [BlueDot Impact](https://bluedot.org/). Thanks to Adrians Skapars for providing mentorship, feedback, and helpful discussions.\n\nIt was also noted in the [MALT](https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors/#summary-table) post that METR’s scoring seemed rather uncalibrated, as scores were saturated near 0 and 100 - the monitor is assigning rather definitive deception scores. We may want to explore the effects of highlighting the importance of score calibration in prompting - and whether this can help with deception scoring - we hypothesize that having better calibrated scores will aid in achieving results with aggregated chunked monitoring.\n\n|\n|\n|\n|---|---|---|\ngives_up | unprompted | model gives up on task despite having resources to continue |\nmatch_weaker_model | prompted | model tries pretending to be a weaker model and reasons about this |\npartial_problem_solving | prompted | model prompted to not fully solve task to help the user learn (meant to procure “benign” sandbagging behavior). |\nreasoning_about_task | prompted | model prompted to reason on the aspects of the task that are not expected/necessary to complete the task in a normal way. |\nsabotage | prompted | model prompted to sabotage the task in various settings. |\nrefusals | prompted | models prompted to refuse in various ways and with differing frequencies. |\nbenign | unprompted | natural, neutral non-deceptive behavior |", "url": "https://wpnews.pro/news/using-chunked-monitoring-to-detect-deception-in-long-transcripts", "canonical_source": "https://www.lesswrong.com/posts/orMEvTgtRig9qnsdL/using-chunked-monitoring-to-detect-deception-in-long", "published_at": "2026-08-15 20:05:20+00:00", "updated_at": "2026-08-15 20:12:43.647806+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-safety", "ai-research"], "entities": ["METR", "MALT", "GPT-4.1 Mini", "claude-3-7", "claude-opus-4", "claude-3-5-sonnet", "gpt-4o", "claude-sonnet-4"], "alternates": {"html": "https://wpnews.pro/news/using-chunked-monitoring-to-detect-deception-in-long-transcripts", "markdown": "https://wpnews.pro/news/using-chunked-monitoring-to-detect-deception-in-long-transcripts.md", "text": "https://wpnews.pro/news/using-chunked-monitoring-to-detect-deception-in-long-transcripts.txt", "jsonld": "https://wpnews.pro/news/using-chunked-monitoring-to-detect-deception-in-long-transcripts.jsonld"}}