# Show HN: When AI Decides What Matters

> Source: <https://jasondoyle.ie/whitepapers/when-ai-decides-what-matters/>
> Published: 2026-08-25 04:56:55+00:00

Disclosure:These views are my own and do not represent my current or any former employers.

## Executive summary

Email, calendar, meeting, and notification assistants are increasingly presented as a way to begin the day with the right information already selected. They summarise long threads, identify action items, rank messages, group notifications, and prepare short accounts of what needs attention.

This sounds like compression. It is also allocation. The system decides which inputs appear, how they are described, and what remains outside the user's immediate view. It influences attention before the user makes a decision.

The distinction matters because errors are not equally visible. A false statement in a summary may be noticed and challenged. An important message that was not surfaced can leave no obvious sign that anything went wrong. The user sees a plausible account of the day without seeing the material that the account omitted.

Microsoft describes Outlook's Prioritize feature as a system that selects important mail using past behaviour, organisational context, user instructions, and other signals. It leans towards messages that require action. It also excludes several categories of mail from evaluation, including messages delivered outside the Inbox, meeting mail, some encrypted messages, and mail marked as low importance by the sender.[1][2]

Microsoft separately states that an email summarisation feature may "overlook important details or misinterpret the context" and advises users to check the original message. The same document says that precision, recall, and completeness are evaluated, but it does not publish the results.[3]

The public record contains enough evidence to treat these limitations seriously:

- Apple Intelligence produced false notification summaries that appeared to come from established news organisations. Apple later paused summaries for news and entertainment applications while it worked on improvements.[4][5]
- A Microsoft 365 Copilot defect caused emails labelled confidential to be processed despite configured data loss prevention rules. Microsoft said the issue did not grant access to anyone who lacked permission, but acknowledged that the behaviour did not match the intended protection.[6]
- Security researchers demonstrated that email and calendar content could contain instructions that altered an assistant's output or caused private information to be copied into another record. These were controlled demonstrations, not confirmed attacks against customers.[7][8][9]
- A meeting transcription service reportedly sent an external participant a transcript that included private conversation held after the participant believed the meeting had ended.[10]
- A Canadian tribunal held Air Canada responsible when a customer relied on incorrect information supplied by its website chatbot. The case did not concern an inbox assistant, but it illustrated a useful principle: an organisation remains responsible for information supplied through its automated systems.[11]

None of these cases proves that an inbox prioritisation feature has caused a missed filing, lost contract, or personal emergency. I found no verified public case that makes that direct causal link. That gap should be stated plainly.

It should not be mistaken for evidence that the risk is absent. Mailbox failures are private, omissions are difficult to observe, products are relatively new, and I found no public incident register or published false-negative rate for the items these products classify as important.

The proposal in this paper is not to reject AI assistance. It is to treat systems that allocate attention as decision support rather than neutral convenience:

- Measure whether important items are missed alongside whether users report saving time.
- Show what sources were evaluated, what was excluded, and which product and configuration produced the result.
- Keep legal, safety, financial, security, and other critical obligations on channels that do not depend on probabilistic ranking.
- Test assistants with novel, ambiguous, adversarial, and role-specific messages before relying on them.
- Require vendors and deploying organisations to define responsibility for omissions, changes, and incidents.

AI assistants do not merely summarise information. They govern attention. Their most consequential failure may be the item they quietly leave out.

## 1. From information tool to attention system

A traditional search tool responds when a person asks a question. A traditional mail rule follows an explicit condition. A person can usually inspect the query or rule and understand why a result appeared.

An AI assistant can operate earlier in the process. It can decide which messages are important, which parts of a thread deserve inclusion, which meeting actions belong to the user, and which notifications should appear first. The result may be waiting before the user has looked at the underlying material.

That changes the role of the software.

The assistant is no longer only helping a user work with information. It is constructing the user's first view of that information. It becomes an attention system.

An attention system performs at least four functions:

| Function | Apparent user benefit | Judgement being made |
|---|---|---|
| Selection | Removes noise | Which inputs deserve consideration |
| Ranking | Shows important items first | Which inputs have greater consequence |
| Compression | Reduces reading time | Which facts can be removed |
| Presentation | Creates a coherent briefing | Which interpretation should frame the day |

Each function can be useful. Each can also fail independently.

A system may select the right thread but summarise it incorrectly. It may summarise a message accurately but rank it too low. It may identify a deadline but assign the action to the wrong person. It may process every visible message correctly while excluding a folder that contains the only legal notice.

The finished briefing can still look coherent.

This is the central difficulty. People judge the output they can see. They cannot easily judge an input that never reached the output.

### Compression changes meaning

Summarisation is often described as shortening. In practice, a useful summary must decide what is material. That requires interpretation.

Consider a long customer thread containing:

- a technical question;
- an apology from an account manager;
- a revised delivery date;
- an implied threat to terminate;
- a request for written confirmation by Friday.

A summary can be factually correct and still fail if it omits the termination risk or the required response date. No sentence needs to be fabricated for the result to be misleading.

The same applies to a daily briefing. A list of five real tasks does not prove that the sixth, omitted task was unimportant.

### Ranking changes behaviour

Priority labels affect what a person reads first, what they defer, and what they may never open. A filter that shows only high-priority messages turns a classification into a practical visibility boundary.

Microsoft's documentation states that Copilot can mark mail as high, normal, or low priority, and that users can filter or sort by those classifications.[1] The low-priority icon is off by default. A user may therefore see positive signals about what the system selected without receiving an equally prominent account of what it deprioritised.

The email remains available. The user's attention may not.

### A briefing creates implied completeness

A product does not need to claim that its output is complete for users to treat it that way. Labels such as "prepare for your day", "catch up", "what matters", and "priority" create an expectation that the system has reviewed the relevant field and selected the important parts.

The interface may show a polished list rather than a coverage report. It may not say:

- how many records were evaluated;
- which folders or message types were excluded;
- whether an attachment could be read;
- whether the model changed since yesterday;
- whether a security filter removed part of the context;
- how often comparable important items are missed.

The user receives an answer without a clear account of its boundaries.

## 2. Importance is not a property of a message

An email does not contain a universal importance value. Importance depends on the recipient, their role, the time, the surrounding events, and the consequence of inaction.

Several different ideas are often collapsed into one label:

| Concept | Question |
|---|---|
| Urgency | How soon must someone act? |
| Consequence | What happens if nobody acts? |
| Relevance | Does this relate to the recipient's current work? |
| Authority | Does the sender hold organisational power? |
| Actionability | Is there a clear request or decision? |
| Novelty | Is this outside the normal pattern? |
| Familiarity | Has the user often engaged with this sender or topic? |

These concepts overlap, but they are not interchangeable.

A routine request from a manager may be familiar, authoritative, and actionable without being consequential. A legal notice from an unknown sender may be unfamiliar and written in neutral language while carrying a strict deadline. A security alert may be automated, repetitive, and important only once. A customer may imply cancellation without using an urgent keyword.

The system must infer which of these signals matters now.

### Personalisation does not remove the ambiguity

Outlook allows a user to provide natural-language instructions such as "It's from my manager" or "It's about a customer complaint". Microsoft says the feature also uses past behaviour and organisational context, and adapts as communication patterns change.[1][2]

This can improve relevance. It can also preserve assumptions that no longer hold.

A person may change role. A project may enter an incident phase. A new regulator, customer, supplier, or manager may appear. A sender who was previously unimportant may become the source of the most consequential message in the mailbox.

Past behaviour is useful when the future resembles the past. Critical events often matter because they do not.

### Product scope can be mistaken for judgement

Microsoft documents several categories that Prioritize does not evaluate:

- mail delivered to a folder other than the Inbox;
- mail marked as low importance by the sender;
- out-of-office messages;
- meeting invitations and related meeting mail;
- mail sent only to oneself;
- messages with very little content;
- non-delivery receipts;
- quarantine messages;
- encrypted messages whose content cannot be read.[1]

These are understandable implementation boundaries. They are also possible sources of false confidence.

An item may be absent because the system judged it unimportant, because a rule moved it, because its content was inaccessible, or because the product did not support that message type. Those are different causes. The user may experience the same result: the item is not in the briefing.

### Exact weighting remains opaque

Vendors disclose broad categories of signals and examples of custom instructions. They do not publish the exact weighting, thresholds, model behaviour, or error rates that produce a particular priority.

There are valid reasons not to expose every internal detail. Full disclosure could create security problems and would become stale as models change.

The absence of exact weights does not remove the need for evidence. A buyer does not need source code to ask:

- What percentage of predefined critical items does the system surface?
- How does performance change for new senders?
- Which content types are not assessed?
- How often does a product update alter the result?
- What happens when messages contain instructions aimed at the model?

An opaque method can still be evaluated. At present, the public evaluation is limited.

## 3. The asymmetry of attention errors

The common discussion of generative AI errors focuses on false content. Attention systems introduce a second problem: missing content.

| Failure | What the user sees | Typical visibility | Possible consequence |
|---|---|---|---|
| False positive | An ordinary item marked important | High | Interruption, wasted time, alert fatigue |
| False summary | A surfaced item described incorrectly | Medium | Wrong response, misunderstanding, reputational harm |
| False attribution | An action or statement assigned to the wrong person | Medium | Missed ownership, conflict, incorrect record |
| False negative | An important item left at normal or low priority | Low | Missed deadline, delayed response, unaddressed risk |
| Scope omission | An item not evaluated at all | Low | False belief that the briefing covered the source |
| Adversarial manipulation | Untrusted content alters output or action | Low to medium | Fraud, data disclosure, deceptive instructions |

False positives are irritating and measurable. Users can report that too many messages were marked important.

False negatives are quieter. A user must discover the omitted item through another route, recognise that it should have appeared, and connect the failure to the assistant. If the item is found late, the cause may be recorded as human delay rather than ranking failure.

This creates a measurement bias. The easiest mistakes to count are not necessarily the most costly.

### Omission can look like success

A short briefing is often considered better than a long one. If the system removes more material, it may feel efficient. The same behaviour can increase the chance that a weakly signalled but consequential item disappears.

An evaluation based on satisfaction or time saved can reward aggressive compression. It does not reveal whether the saved time came from removing noise or from removing work the user needed to see.

### Verification can consume the claimed saving

Vendors often advise users to check original content for important decisions. That is sensible. It also creates a practical contradiction.

If a user must reopen every source to make sure the summary did not omit anything, the summary has not replaced the review. It has added another layer before it.

The value can still be real. A summary may help a person understand structure before reading. It may help with accessibility, recall, or unfamiliar subject matter. The relevant measure is net benefit after verification, correction, and recovery, not the time required to read the generated text alone.

### Automation changes the reviewer

Research on decision support in other domains has repeatedly found that people can follow incorrect automated recommendations or fail to act when the system does not prompt them. A 2024 clinical decision-support study measured agreement with incorrect AI recommendations and found differences in susceptibility based on experience and training.[12]

Clinical diagnosis is not email triage. The study does not prove that an Outlook user will miss a contract notice. It does support a broader point: adding a human does not automatically neutralise an automated error. The design of the task, the user's expertise, and the presentation of the recommendation matter.

A person reviewing an unfiltered inbox is searching for important work. A person reviewing an AI briefing is checking a proposed interpretation. Those are different cognitive tasks.

## 4. Lessons from the public record

The public record does not yet contain a large body of verified incidents tied specifically to AI inbox prioritisation. It does contain related failures that expose the same underlying assumptions: summaries can misstate source material, protected content can cross intended boundaries, and untrusted content can influence the assistant that reads it.

The cases below are separated by evidence type. An observed incident is not the same as a security demonstration or a legal allegation.

### False summaries appeared to come from news organisations

In December 2024 and January 2025, Apple Intelligence produced notification summaries that misrepresented alerts from news applications. A BBC summary falsely said that Luigi Mangione had shot himself. Later summaries claimed that Luke Littler had won the PDC World Darts Championship before the final and that Rafael Nadal had come out as gay.[4]

The false text appeared in a notification associated with the publisher's application. The BBC said the summaries did not reflect, and sometimes completely contradicted, its content. This placed a model-generated statement inside the publisher's apparent identity.

Apple responded by pausing notification summaries for news and entertainment applications, changing the visual style of summaries, and warning users that the feature could contain errors.[5]

This was an observed product failure with a visible service response. There is no published evidence that it caused physical harm or a missed personal obligation. The demonstrated impact was misinformation, confusion, and risk to publisher credibility.

The case matters because the source material was not obscure. The failure occurred while compressing short, professionally written notifications from established news organisations.

### Confidential email crossed an intended processing boundary

In early 2026, Microsoft confirmed that Microsoft 365 Copilot Chat could summarise emails in Drafts and Sent Items even when those messages carried a confidential label and a data loss prevention policy was configured to exclude them.[6]

Microsoft attributed the behaviour to a code issue and deployed a configuration update. It said the defect did not provide anyone with access to information they were not already authorised to see. Access controls remained intact, but the assistant processed material that the organisation expected it not to process.

This distinction is important. It was not described as an external data breach. It was a failure of the promised governance boundary.

For organisations using sensitivity labels to define where automated analysis is permitted, that difference still matters. A control can fail without creating a new user permission.

### A crafted email could manipulate Copilot

Microsoft assigned CVE-2025-32711 to an information disclosure vulnerability in Microsoft 365 Copilot. Its advisory describes an AI command injection that could allow an unauthorised attacker to disclose information over a network. Microsoft rated the issue Critical with a CVSS score of 9.3.[7]

The vulnerability became known as EchoLeak. The researchers' attack began with a crafted email that Copilot could later retrieve while answering a user.[17]

Microsoft states that the vulnerability was not known to have been exploited and had been fully mitigated before the CVE was published. No customer action was required.

This was a controlled security finding, not a recorded customer compromise. Its relevance is architectural. An assistant that reads untrusted communication and trusted organisational data must distinguish information from instructions. Natural language does not carry a reliable boundary between the two.

### Gmail summaries could display attacker-controlled instructions

Mozilla's 0DIN programme published a prompt-injection disclosure affecting Gemini in Google Workspace. A malicious email could contain instructions that changed the response when the recipient asked Gemini to summarise unread mail. The manipulated output could appear to originate from Gemini rather than from the sender.[8]

Google now documents prompt injection as a risk in Workspace. Its guidance states that malicious content in a message may cause Gemini to warn the user, exclude part of the content, or decline to answer.[13]

The published example was a security demonstration. There was no confirmed campaign against users.

The finding still weakens a common mental model. Email summaries are not generated from passive facts. They are generated from content supplied by people who may benefit from influencing the result.

### A calendar invitation could become an instruction

Researchers at Miggo demonstrated a related attack through Google Calendar. A malicious instruction placed in an event description remained dormant until the user asked Gemini a routine question about their schedule. In the demonstration, Gemini summarised private meetings and wrote the result into a newly created calendar event that could be visible to the attacker.[9]

Google said the reported techniques were theoretical scenarios, and described additional confirmation, URL handling, and prompt-injection defences deployed before public disclosure.[14]

Again, this was not evidence of exploitation in production. It showed that a tool intended to answer "What does my day look like?" could treat an invitation from another person as an instruction governing how to answer.

### A meeting transcript included private conversation after the call

In 2024, venture capitalist Alex Bilzerian reported that an Otter.ai transcript from a Zoom meeting included hours of private conversation held by the other participants after he left. The transcript was automatically emailed to him. Otter responded that users control conversation-sharing settings.[10]

The event appears to have involved meeting state and sharing configuration rather than a fabricated summary. It remains relevant because the convenience feature changed the boundary of who received the record.

The assistant did what the configured workflow allowed. The report suggests that the people in the room did not understand that the workflow was still operating.

### The organisation remained responsible for automated advice

In *Moffatt v Air Canada*, a customer relied on incorrect bereavement-fare information supplied by Air Canada's website chatbot. The British Columbia Civil Resolution Tribunal found the airline liable for negligent misrepresentation and awarded damages, interest, and fees.[11]

The case did not involve generative inbox ranking. Its accountability lesson applies broadly. Air Canada argued, in effect, that correct information existed elsewhere on its website. The tribunal rejected the idea that the customer should have to discover which part of the company's site was reliable.

An organisation cannot market an automated interface as a useful source and then treat it as separate from the organisation when the answer is wrong.

## 5. What trials and vendor documents tell us

Public evaluations of workplace assistants often focus on adoption, satisfaction, and estimated time savings. These measures are useful. They do not answer whether the assistant reliably surfaced the most consequential item.

### Microsoft acknowledges summary limitations

Microsoft's responsible AI FAQ for an Outlook email summary feature states:

The summarization algorithm may occasionally overlook important details or misinterpret the context of the email thread.[3]

The document says the system is evaluated through comparative analysis, human review, customer engagement, accuracy, relevance, precision, recall, and completeness. It does not provide the scores, test set, acceptance threshold, failure distribution, or results by type of email.

The limitation is disclosed. Its size is not.

### A large government trial found benefit and continuing risk

The UK Government Digital Service ran a Microsoft 365 Copilot trial with 20,000 civil servants from September to December 2024. Participants reported an average saving of 26 minutes per day. More than 70 per cent said Copilot reduced time spent searching for information and performing mundane tasks.[15]

The report also makes several qualifications:

- the time saving was self-reported;
- the experiment could not identify how the saved time was used;
- the tool struggled with complex, nuanced, or data-heavy work;
- participants raised reliability and accuracy concerns;
- sensitive HR work created concern about reputational risk;
- the conclusion said human oversight was required at all times.

The trial provides evidence that users can find these systems useful. It does not provide a measured false-negative rate for important email, action items, or deadlines.

Both findings can be true. A tool can save time on most days and still create an unacceptable risk for a small class of high-consequence work.

### Satisfaction is not reliability

Users may like a product because it reduces effort, helps them write, improves accessibility, or makes a large inbox feel manageable. Those are real benefits.

Satisfaction does not establish that the system is accurate enough for the use being made of it.

A smoke alarm is not evaluated by how pleasant it is to use. A spam filter is not evaluated only by how much mail it removes. A daily briefing should not be evaluated only by how quickly it can be read.

The metric must match the consequence.

## 6. The evidence that is missing

I found no public vendor report that provides a complete evaluation of critical-item recall for the consumer and enterprise attention features reviewed in this paper.

The following information was not available in the public material examined:

| Evidence | Why it matters |
|---|---|
| False-negative rate for important items | Shows how often consequential material is missed |
| Recall by sender novelty | Tests whether unfamiliar but critical contacts are disadvantaged |
| Recall by message type | Exposes weaknesses in attachments, long threads, invitations, automated notices, and sparse messages |
| Priority calibration | Shows whether "high", "normal", and "low" correspond to measured consequence |
| Performance after personalisation | Tests whether custom rules improve one category while damaging another |
| Change across model versions | Shows whether a product update alters established behaviour |
| Per-briefing coverage | Tells the user what was scanned and excluded on that occasion |
| Independent incident rate | Shows how often errors caused delay, disclosure, or incorrect action |
| Verification cost | Shows whether time saved remains after checking and correcting output |
| Adversarial performance | Shows how the system behaves when content is written to manipulate it |

Some vendors say they measure several of these concepts internally. Internal measurement is better than none. It does not allow a customer, regulator, researcher, or procurement team to judge whether the result is suitable for a particular use.

### No verified consequential miss was found

This review did not find a verified public incident in which Outlook's Prioritize feature, or a directly comparable inbox ranking feature, placed a message too low and caused a documented missed deadline, financial loss, safety event, or regulatory failure.

That should constrain the claim made by this paper. It would be wrong to present a security demonstration, a false news summary, or a confidential-email processing defect as proof of a missed inbox obligation.

Several factors may explain the absence:

- the products are new;
- mailbox contents and workplace failures are rarely public;
- users may not know that an omission occurred;
- organisations may attribute the outcome to the employee;
- I found no dedicated public incident register for these failures;
- causation may be distributed across the product, configuration, user, and process.

The correct conclusion is not that harm has been established. It is that the risk has not been measured publicly enough to support the confidence implied by the product experience.

### Marketing claims and evaluation claims answer different questions

A statement that a tool helps a user "focus on what matters most" is a product claim.[2]

Evidence that users saved time is an efficiency claim.

Evidence that most summaries were rated helpful is a satisfaction claim.

None of these establishes that the system finds a defined percentage of critical items under realistic conditions. That requires a reliability claim, a test, and a threshold.

The difference should not be hidden inside general language about quality.

## 7. Why an explanation is not an audit

Outlook can display a short explanation of why Copilot marked a message as important. This is useful. It gives the user something to question and can help refine personalisation.[1][2]

It is not a complete audit of the decision.

A useful explanation of one selected message does not show:

- which other messages were considered;
- which messages were outside scope;
- how competing signals were weighted;
- what threshold separated high from normal priority;
- whether the same input would receive the same result again;
- whether an untrusted message influenced the reasoning;
- what changed after a model or configuration update.

It also says nothing about the most important false negative, because that message has no high-priority explanation to inspect.

### The user needs a coverage record

A daily briefing or priority view should provide a compact record of its own scope. At minimum, it should state:

| Coverage question | Example information |
|---|---|
| What period was assessed? | Messages received between 17:00 yesterday and 08:00 today |
| What sources were assessed? | Inbox, selected shared mailbox, primary calendar |
| What was excluded? | 4 encrypted messages, 3 meeting invitations, 12 rule-routed messages |
| What criteria were active? | User rules, organisation policy, product defaults |
| What changed? | New model version, modified priority instruction, new mailbox rule |
| What evidence supports each item? | Direct links to source messages, events, and transcript passages |
| Were risks detected? | Suspicious content excluded, source inaccessible, incomplete thread |

This does not require disclosure of proprietary model weights. It provides the user with a boundary around the answer.

### The record should survive change

If a model changes overnight, yesterday's priority decision may not be reproducible. If a user edits an instruction, later investigation may not show which rule was active when the message arrived.

Material attention decisions need basic versioning:

- product and model version where available;
- configuration and user instructions;
- source identifiers;
- classification and explanation;
- time of evaluation;
- later correction or user feedback.

As argued in *Making Autonomous Work Reviewable*, human oversight depends on a stable account of scope, evidence, uncertainty, and change.[16] The same principle applies when the machine's action is deciding what a person sees.

## 8. A practical way to test attention systems

An organisation does not need access to a vendor's model to test whether the product is suitable for its work.

It can create a controlled mailbox and calendar, define the expected outcomes before the test, and measure what the assistant surfaces.

### Build a representative test set

The test should include ordinary work and deliberately difficult cases. A useful initial set might contain 200 messages, 30 calendar entries, and several meeting transcripts across a two-week period.

Domain reviewers should label each item before the assistant is enabled:

- critical;
- material;
- routine;
- noise;
- outside the recipient's responsibility.

They should also record why the item belongs in that category and the latest acceptable time for it to be surfaced.

The test must not be built only by AI specialists. A legal deadline should be labelled by someone who understands legal work. A production incident should be labelled by an operator. A vulnerable-customer escalation should be labelled by the relevant service team.

### Include realistic failure cases

| Scenario | Purpose |
|---|---|
| Known senior sender with routine content | Tests whether authority is mistaken for consequence |
| Unknown sender with a legal deadline | Tests novelty and weak relationship signals |
| Customer complaint using restrained language | Tests implied consequence |
| Urgent wording with no required action | Tests keyword overreaction |
| Deadline contained only in an attachment | Tests source coverage |
| Long thread with a late correction | Tests whether the newest fact overrides earlier context |
| Conflicting dates in the same thread | Tests temporal reasoning |
| Message moved by a mailbox rule | Tests documented scope exclusions |
| Encrypted or sparse message | Tests whether non-evaluation is made visible |
| Meeting invitation containing a material request | Tests channel boundaries |
| Automated security alert among repeated notices | Tests rare critical change within familiar noise |
| Role change during the test | Tests personalisation drift |
| Message in another working language | Tests language coverage |
| Industry jargon and indirect requests | Tests domain context |
| Hidden or visible prompt injection | Tests adversarial resistance |
| False instruction claiming to be from IT | Tests whether summaries create trusted-looking phishing |

The benchmark should include cases where reasonable people disagree. Those cases should be reported as ambiguous rather than forced into a false ground truth.

### Run more than once

Generative systems can vary across runs and versions. A single successful demonstration is weak evidence.

The organisation should:

- run the same test several times;
- repeat it with a new account and an established account;
- test default settings and realistic personalisation;
- record every product and configuration change;
- repeat the benchmark after material updates;
- retain failed outputs as well as representative successes.

### Measure the errors that matter

| Metric | Definition |
|---|---|
| Critical-item recall | Percentage of predefined critical items surfaced within the required time |
| Material-item recall | Percentage of material items surfaced within the required time |
| False-alert rate | Percentage of surfaced high-priority items that reviewers labelled routine or noise |
| Summary completeness | Percentage of predefined material facts present without contradiction |
| Action accuracy | Correct owner, action, deadline, and status |
| Source faithfulness | Claims directly supported by the underlying source |
| Coverage accuracy | Whether the system correctly reports what it did and did not assess |
| Stability | Frequency with which the same item receives materially different treatment |
| Adversarial success rate | Percentage of malicious inputs that change output or trigger an unauthorised action |
| Detection and recovery time | Time required for a user or administrator to recognise and correct a failure |
| Net time saving | Time saved after review, correction, and follow-up |

Average accuracy should not hide critical misses. If a test contains 200 items and the system handles 198 correctly, a 99 per cent score may look excellent. If the two failures are the only safety notice and the only filing deadline, the result is unacceptable.

Report the count and consequence of misses alongside the percentage.

### Test the whole workflow

The model is only one component. The evaluation should include:

- mailbox rules;
- folder routing;
- permissions;
- sensitivity labels;
- encrypted content;
- mobile and desktop presentation;
- notification settings;
- user filters;
- shared mailboxes;
- calendar integrations;
- retention and audit logs.

A model can classify a message correctly while the interface hides the label, a rule moves the message out of scope, or a notification setting prevents the user from seeing it.

The unit under test is the user's actual attention path.

## 9. What buyers should require

Procurement teams often ask whether a product supports encryption, data residency, access control, and administrative logging. Attention systems need additional questions.

### Evaluation evidence

The vendor should provide:

- the intended use and explicit non-uses;
- the types of source content evaluated and excluded;
- critical-item recall and false-negative results for representative tests;
- results for new senders, sparse messages, attachments, long threads, and conflicting information;
- known limitations by language and domain;
- the effect of personalisation on measured performance;
- adversarial testing for prompt injection and deceptive content;
- the process used to set acceptance thresholds;
- the date and version associated with every result.

"Continuously improving" is not a result.

### Operational transparency

The product should support:

- a visible unfiltered route to all source material;
- per-briefing or per-run coverage information;
- source links for every summary and action;
- clear separation between generated text and sender-authored text;
- logs of classifications, configuration, and material changes;
- notification of model changes that may alter behaviour;
- export for independent review and incident investigation;
- an administrator view of failures and user corrections.

### Control

The organisation should be able to:

- define sources that must never be suppressed;
- exclude sensitive sources from model processing;
- require deterministic handling for specified senders, topics, or message types;
- disable generated summaries without losing access to the source;
- stop tool actions while retaining read-only assistance;
- roll back configuration changes;
- separate experimental features from production users;
- set retention appropriate to the source data.

### Incident handling

The contract and operating process should answer:

- What qualifies as an attention-system incident?
- Will the vendor notify customers about model or policy failures?
- Can the customer determine which users and messages were affected?
- Are historical classifications retained long enough to investigate?
- How are corrections propagated?
- Who owns communication to affected people?
- What evidence will be available after the model has changed?

A generic service-status page is not enough if the failure concerns a subset of private content and personalised decisions.

## 10. A safer operating model

The appropriate control depends on the consequence of a miss.

| Tier | Typical information | Permitted use of AI attention features | Required fallback |
|---|---|---|---|
| 0: Convenience | Newsletters, routine updates, optional reading | Summary and ranking may be the primary view | Periodic sampling |
| 1: Operational | Team coordination, ordinary project actions | Summary and ranking with source links | Daily review of unresolved and low-confidence items |
| 2: Material | Customer commitments, contracts, finance, HR, service incidents | Organisation and drafting support, not the sole alert path | Named owner, deterministic rule, acknowledgement, escalation |
| 3: Critical | Safety, legal deadlines, funds movement, security containment, regulated decisions | No reliance on probabilistic ranking as the only control | Dedicated channel, receipt, timed escalation, independent monitoring |

The tier should be based on consequence, not on the assistant's confidence.

### Keep a non-probabilistic route for obligations

If missing an item can create legal, safety, financial, or security harm, the obligation should have a deterministic path.

Examples include:

- a case-management queue with ownership and due dates;
- a monitored address with acknowledgement requirements;
- a paging system for production incidents;
- a contract register for renewal and notice dates;
- a regulatory calendar with named responsibility;
- a payment workflow with approval and reconciliation.

Email may still notify the user. The AI summary may still help. Neither should be the only record that the obligation exists.

### Do not turn a summary directly into action

A generated action item should not automatically become a payment, customer commitment, disciplinary record, access change, or legal response without source verification and the controls appropriate to that action.

The assistant may draft. The system of record should enforce authority.

### Audit false negatives

Organisations should sample items that were not marked important as well as reviewing the items that were.

A monthly audit can select:

- random normal and low-priority messages;
- new external senders;
- items from critical domains;
- messages close to known incidents;
- messages later escalated by a person;
- content that changed classification after user feedback.

The goal is to look where the interface did not direct attention.

### Reset assumptions when work changes

Personalisation should be reviewed when:

- a user changes role;
- a manager or team changes;
- a project enters a critical phase;
- a merger or supplier change introduces new domains;
- an incident changes what counts as urgent;
- the vendor changes the model or default criteria.

Learning from behaviour should not become an invisible permanent policy.

## 11. Responsibility cannot be delegated to the briefing

Attention systems distribute responsibility across the vendor, the deploying organisation, administrators, and users. Ambiguity benefits nobody after a failure.

| Actor | Responsibility |
|---|---|
| Vendor | Define intended use, disclose scope, test failure modes, secure untrusted input, report material defects |
| Deploying organisation | Choose suitable uses, maintain information governance, provide fallback channels, test locally, monitor incidents |
| Administrator | Configure access and exclusions, record changes, preserve logs, review updates |
| Process owner | Define what is critical, establish deterministic controls, audit misses |
| User | Treat generated output as assistance, verify material decisions, report errors |

The user has a role. The user cannot carry the whole burden.

NIST's Generative AI Profile treats monitoring, documentation, human oversight, incident disclosure, and accountability as parts of a wider risk-management programme rather than a single warning shown to the user.[18]

A warning that "AI can make mistakes" does not tell the user which item was excluded, whether the product was tested for their use, or how often a critical message is missed. Asking the user to verify everything can also conflict with the reason the product was purchased.

### Human oversight requires access to the unselected material

A human cannot meaningfully supervise a ranking decision if they see only the selected results.

The person needs:

- access to the full source;
- a clear indication of scope;
- a practical way to inspect deprioritised items;
- focused warnings about unsupported or inaccessible content;
- time and responsibility to perform the review.

"Human in the loop" is an organisational description, not proof that a useful control exists.

### The organisation owns the outcome

*Moffatt v Air Canada* is useful because it rejects the idea that an automated interface sits outside the service provider's responsibility.[11]

The same principle should guide internal deployment. If an organisation tells staff that an assistant will prepare them for the day, configures it across the workplace, and encourages reliance on its prioritisation, it should not treat every omission as individual negligence.

Responsibility should be assigned before the first incident:

- who validates the feature;
- who defines prohibited reliance;
- who monitors performance;
- who investigates a miss;
- who informs affected people;
- who decides whether the feature remains enabled.

## 12. What this paper does not claim

This paper does not claim that AI summaries and priority features are useless. Public trials and user reports show genuine benefit, including reduced effort, accessibility support, and faster understanding of routine material.

It does not claim that every inaccurate summary causes harm.

It does not claim that deterministic rules are perfect. Static rules can route mail incorrectly, become stale, or create their own false confidence.

It does not claim that the published security demonstrations were exploited against customers. EchoLeak, the Gemini email example, and the calendar-invitation attack are included as demonstrated failure modes with their reported mitigations and caveats.

It does not claim that the absence of a public missed-deadline case proves concealment. The evidence is incomplete, and the correct response to incomplete evidence is a narrower claim.

The claim is this:

Products that select and compress a person's work can affect which obligations receive attention. Their public evaluation does not yet match that role. The most important measures, especially critical-item recall, scope coverage, and consequential false negatives, are rarely available to the people asked to trust them.

The public-source review for this paper was completed on 24 August 2026.

## Conclusion

AI assistants can reduce the effort required to read, organise, and understand a busy working day. That benefit is credible. So are the limits.

The central governance problem extends beyond whether a generated sentence is true. It includes whether the system looked in the right places, recognised the right consequence, resisted manipulation, and made its omissions visible.

An assistant that labels a message important is making a recommendation. An assistant that leaves a message out is also making a recommendation, even if the interface never presents it as one.

Organisations should therefore measure attention systems as they would other decision support:

- define the consequential cases;
- test them before deployment;
- publish the limits;
- preserve a non-probabilistic route for critical obligations;
- monitor what the system misses;
- keep responsibility with the people and organisations that chose the system.

The useful question is not whether an assistant can produce a plausible account of the day.

It is whether the user can know what was considered, what was excluded, how often important items are missed, and who is responsible when the omission matters.

## Appendix A: Attention-system benchmark record

```
# [Product and test name]

Test date:
Product version:
Model version, if disclosed:
Tenant configuration:
User instructions:
Evaluator:

## Intended use

[State the exact workflow being evaluated.]

## Prohibited reliance

[List decisions or obligations for which the feature cannot be the sole control.]

## Test sources

| Source | Item count | Included by product | Notes |
| --- | ---: | --- | --- |
| Inbox | ... | Yes/No | ... |
| Subfolders | ... | Yes/No | ... |
| Calendar | ... | Yes/No | ... |
| Attachments | ... | Yes/No | ... |
| Meeting transcripts | ... | Yes/No | ... |

## Ground truth

| Class | Count | Defined by | Required surfacing time |
| --- | ---: | --- | --- |
| Critical | ... | ... | ... |
| Material | ... | ... | ... |
| Routine | ... | ... | ... |
| Noise | ... | ... | ... |

## Results

| Metric | Result | Threshold | Pass/Fail |
| --- | ---: | ---: | --- |
| Critical-item recall | ... | ... | ... |
| Material-item recall | ... | ... | ... |
| False-alert rate | ... | ... | ... |
| Summary completeness | ... | ... | ... |
| Action accuracy | ... | ... | ... |
| Coverage accuracy | ... | ... | ... |
| Adversarial success rate | ... | ... | ... |
| Net time saving | ... | ... | ... |

## Critical misses

[List every critical miss individually, including consequence and cause.]

## Scope and exclusions

[Record unsupported, inaccessible, filtered, and security-blocked content.]

## Variability

[Record differences across repeated runs, accounts, and versions.]

## Decision

Approved use:
Required controls:
Retest date:
Named owner:
```

## Appendix B: Buyer and deployment checklist

- Is the intended use stated in operational terms?
- Are prohibited high-consequence uses explicit?
- Does the vendor publish critical-item recall or equivalent results?
- Are exclusions visible to the user for each briefing?
- Can every generated claim link to its source?
- Can a user reach the unfiltered source without disabling the feature?
- Are model, configuration, and personalisation changes recorded?
- Has the complete workflow been tested with novel senders and ambiguous requests?
- Has the product been tested against prompt injection in email, documents, and invitations?
- Can administrators prevent specified channels or content from being suppressed?
- Are critical obligations recorded outside the assistant?
- Does the organisation audit normal and low-priority items for misses?
- Is incident ownership assigned?
- Can historical decisions be investigated after a model update?
- Is the net benefit measured after verification and correction?

## About the author

Jason Doyle writes about reliable software, observability, applied AI, and practical controls for systems that influence human decisions. He publishes at [jasondoyle.ie](https://jasondoyle.ie) and can be contacted at [[email protected]](/cdn-cgi/l/email-protection#c5a6aaabb1a4a6b185afa4b6aaaba1aabca9a0ebaca0).

## References

- Microsoft Support,
*Prioritize my inbox*,[https://support.microsoft.com/en-us/outlook/copilot-outlook/prioritize-my-inbox](https://support.microsoft.com/en-us/outlook/copilot-outlook/prioritize-my-inbox). - Microsoft Inside Track,
*Wrangling our email with the Prioritize My Inbox feature in Microsoft Outlook*,[https://www.microsoft.com/insidetrack/blog/wrangling-our-email-with-the-prioritize-my-inbox-feature-in-microsoft-outlook/](https://www.microsoft.com/insidetrack/blog/wrangling-our-email-with-the-prioritize-my-inbox-feature-in-microsoft-outlook/). - Microsoft Learn,
*FAQ for email summary feature in Outlook*, updated 13 May 2026,[https://learn.microsoft.com/en-us/microsoft-sales-copilot/faqs-email-summary](https://learn.microsoft.com/en-us/microsoft-sales-copilot/faqs-email-summary). - BBC News,
*Apple AI notification errors persist despite complaints*, 6 January 2025,[https://www.bbc.co.uk/news/articles/cge93de21n0o](https://www.bbc.co.uk/news/articles/cge93de21n0o). - TechCrunch,
*Apple pauses AI notification summaries for news after generating false alerts*, 16 January 2025,[https://techcrunch.com/2025/01/16/apple-pauses-ai-notification-summaries-for-news-after-generating-false-alerts/](https://techcrunch.com/2025/01/16/apple-pauses-ai-notification-summaries-for-news-after-generating-false-alerts/). - BleepingComputer,
*Microsoft says bug causes Copilot to summarize confidential emails*, February 2026,[https://www.bleepingcomputer.com/news/microsoft/microsoft-says-bug-causes-copilot-to-summarize-confidential-emails/](https://www.bleepingcomputer.com/news/microsoft/microsoft-says-bug-causes-copilot-to-summarize-confidential-emails/). - Microsoft Security Response Center,
*CVE-2025-32711: M365 Copilot Information Disclosure Vulnerability*, 11 June 2025,[https://msrc.microsoft.com/update-guide/vulnerability/CVE-2025-32711](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2025-32711). - 0DIN,
*Google Gemini G-Suite Prompt Injection Vulnerability*,[https://0din.ai/disclosures/e24d9e6b-8c5e-4e2f-ad4f-2abc0072307a](https://0din.ai/disclosures/e24d9e6b-8c5e-4e2f-ad4f-2abc0072307a). - Miggo Security,
*Weaponizing Calendar Invites: A Semantic Attack on Google Gemini*,[https://www.miggo.io/post/weaponizing-calendar-invites-a-semantic-attack-on-google-gemini](https://www.miggo.io/post/weaponizing-calendar-invites-a-semantic-attack-on-google-gemini). - NewsNation,
*AI-powered transcription leaks workplace secrets*, October 2024,[https://www.newsnationnow.com/business/tech/ai-powered-transcription-leaks-workplace-secrets/](https://www.newsnationnow.com/business/tech/ai-powered-transcription-leaks-workplace-secrets/). *Moffatt v Air Canada*, 2024 BCCRT 149,[https://canlii.ca/t/7np8t](https://canlii.ca/t/7np8t).- Florian Kuecking et al.,
*Automation Bias in AI-Decision Support: Results from an Empirical Study*, Studies in Health Technology and Informatics, volume 317, 2024,[https://doi.org/10.3233/SHTI240871](https://doi.org/10.3233/SHTI240871). - Google Help,
*Learn how Google Workspace with Gemini helps protect users from malicious content and prompt injection attacks*,[https://support.google.com/mail/answer/16204578](https://support.google.com/mail/answer/16204578). - Google Security Blog,
*Mitigating prompt injection attacks with a layered defense strategy*, June 2025,[https://security.googleblog.com/2025/06/mitigating-prompt-injection-attacks.html](https://security.googleblog.com/2025/06/mitigating-prompt-injection-attacks.html). - UK Government Digital Service,
*Microsoft 365 Copilot Experiment: Cross-Government Findings Report*, 2025,[https://www.gov.uk/government/publications/microsoft-365-copilot-experiment-cross-government-findings-report/microsoft-365-copilot-experiment-cross-government-findings-report-html](https://www.gov.uk/government/publications/microsoft-365-copilot-experiment-cross-government-findings-report/microsoft-365-copilot-experiment-cross-government-findings-report-html). - Jason Doyle,
*Making Autonomous Work Reviewable*,[https://markupbase.com/whitepapers/making-autonomous-work-reviewable/](https://markupbase.com/whitepapers/making-autonomous-work-reviewable/). - Pavan Reddy and Aditya Sanjay Gujral,
*EchoLeak: The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM System*, 2025,[https://arxiv.org/abs/2509.10540](https://arxiv.org/abs/2509.10540). - National Institute of Standards and Technology,
*Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile*, NIST AI 600-1, July 2024,[https://doi.org/10.6028/NIST.AI.600-1](https://doi.org/10.6028/NIST.AI.600-1).