AnalysisNov 16, 20267 min read5 sources

AI vendor boundaries are where prompt security breaks

In one minute

Notion’s Fireworks incident is not just a small breach count: it shows that prompts handled by an AI feature can become stored customer data in a provider’s development environment, beyond the protections teams associate with inference.

Summary

Notion says attackers accessed AI Insights chat transcripts held at Fireworks, affecting fewer than 1% of its workspaces. Some affected messages contained API keys; Notion revoked its own exposed keys and notified the vendors that issued third-party keys. The broader lesson is that an AI product’s security boundary follows its data through classification, inference, storage and development systems—not just the model endpoint.

What happened

Notion’s security advisory says Fireworks served two roles: classification jobs for its Insights feature and inference for open-weight models in its product. Fireworks notified Notion of suspicious access on November 2, and Notion disconnected from the provider by November 3 and deleted its Fireworks data on November 4. Fireworks later told Notion that customer data might also have been accessed; Notion says the final affected dataset included Insights transcripts. The company says the incident affected under 1% of workspaces, and that most affected transcripts could not be tied to a user or workspace. It disabled Insights while investigating and stopped serving open-weight models through Fireworks. Notion says it revoked exposed Notion API keys and notified the issuers of third-party keys found in messages. Its investigation is ongoing.

What the archive adds

New Runtime’s earlier coverage of Fireworks’ November 9 update noted that the October incident involved a third-party cloud provider reaching Fireworks development machines, with billing data affected and secrets, datasets or incomplete model weights affected for some accounts. Fireworks said its inference, Training SDK and Nexus services were zero-data-retention by default and reported no impact to them. That distinction matters: the Notion transcripts were held in a dataset used for Insights, so a zero-retention claim for inference alone would not describe every data path. New Runtime also covered OpenAI’s Sign in with ChatGPT Business integration across third-party tools, including Notion; that is a separate access path, not evidence of a connection to this incident.

Trend signal

AI vendors are both model infrastructure and data processors. A product may send prompts to one provider for inference and another workflow to classify or analyze usage; each path can create different retention and access conditions. Fireworks’ disclosed development-environment exposure and Notion’s discovery of transcripts show why a provider’s inference retention setting is not a complete map of customer-data exposure. The operational trend is toward auditing data flows by workload and storage location, rather than treating “the AI provider” as one uniform boundary.

Why now

Three concrete drivers made the exposure possible now: (1) Notion used Fireworks for both Insights classification and open-weight-model inference, creating multiple data paths (Notion advisory; dossier finding d1). (2) Fireworks said the October cloud-provider incident reached its development machines and caused additional customer data impact (Fireworks November 9 update). (3) The affected dataset included persisted Insights transcripts, a workload distinct from inference services Fireworks described as zero-data-retention by default (dossier finding d2).

Connections to other events

The incident’s useful signal is the mismatch between what teams often mean by “prompt data” and what actually persists across a feature. Notion says the recovered dataset contained chat transcripts used for Insights, while most records lacked labels tying them to a user or workspace. That reduces attribution, but not the sensitivity of message contents: Notion also found API keys in some messages. The likely weak point is therefore not only the model request itself, but the secondary dataset and development systems that support product features. This is an inference from the disclosed data path, not proof that every provider workflow retains transcripts.

Why it matters

For builders, map every AI feature’s prompt and response path—including analytics, evaluation, classification, support and development systems—and ask which services store data at rest. Treat credentials pasted into prompts as potentially exposed; scan for them before submission, scope them narrowly, and make rotation straightforward. For buyers, procurement questions should distinguish inference ZDR from retention in adjacent workflows, and require clear data inventories, incident-notification paths and deletion evidence. These controls reduce exposure but do not eliminate provider-side compromise risk.

What happens next

  • Notion will publish a further security-advisory update that changes or supplements the incident status or scope. (1 month, 65%). Watch for: A dated update appears on Notion’s security advisory page before the one-month check. Resolves on 2026-12-13, yes if: Notion’s security advisory page contains a dated update after November 13, 2026 by 2026-12-13.
  • Fireworks will keep self-serve dataset uploads or downloads disabled through the three-month check. (3 months, 60%). Watch for: Its security bulletin or product documentation continues to list the dataset workflows as disabled. Resolves on 2027-02-13, yes if: Fireworks’ security bulletin or docs state that self-serve dataset uploads and downloads for Batch Inference or dataset-based training remain disabled on 2027-02-13.

Limits and uncertainty

The incident affects fewer than 1% of Notion workspaces, and the company says most affected transcripts were unattributable; that limits what can be inferred about breadth or user-level impact. Notion’s investigation is ongoing, and the disclosed facts do not establish exactly how every transcript was stored or accessed within Fireworks. The strongest counterargument is that this is a contained provider-specific breach, not evidence that AI vendors generally mishandle prompts; that is plausible. Still, the distinction between inference retention and other product workflows is a concrete design lesson even if the incident remains isolated.

Competing readings

  • The main lesson is that AI feature data paths beyond inference can retain prompts and create exposure (likely, about 70%). What would tell it apart: Notion used Fireworks for classification and inference, and found Insights transcripts in the affected dataset (notion.com); Fireworks described a development-machine path arising from a third-party cloud-provider incident (fireworks.ai).
  • This was a narrow, provider-specific breach whose lessons should not be generalized to AI data handling overall (unlikely, about 30%). What would tell it apart: Notion says fewer than 1% of workspaces were affected and most transcripts were unattributable (notion.com); The investigation remains ongoing and the disclosed facts do not establish broad industry practices (notion.com).

Base rate

Of 10 earlier incident stories about companies or products in the New Runtime archive over the last 12 months that are at least 3 months old, 8 (80%) were followed within 3 months by a incident story on the same company or product line. In the archive, 8 of 10 earlier incident stories were followed within three months by another incident story on the same company or product line. This supports watching for further Fireworks updates, but does not establish that this breach will recur.

What would change this view

  • A final Notion advisory states that the affected dataset did not contain customer-authored prompt or response content, or that transcripts were not stored in a Fireworks workflow (signal): The conclusion that adjacent AI workflows created a distinct transcript exposure path would be weakened.
  • Fireworks’ security bulletin documents that the affected data path was limited to a development system and that no customer prompts were stored or accessed there (signal): The incident would look more like a narrow infrastructure compromise than a warning about broader prompt-data handling.

Skill to build now

Practice prompt data-flow mapping this week. Choose one AI feature, trace inputs and outputs through inference, classification, evaluation and logging, then record retention and deletion behavior for each service; test a secret scanner with a dummy credential.

Reader takeaway

This week, draw a data-flow map for one AI feature you own: list every prompt, response, classification or evaluation copy, its provider, retention behavior and deletion path. Add a secret scanner before model submission, then test it with a dummy API key and document who can rotate credentials if a transcript is exposed.

Open question

Will customers and auditors distinguish zero retention at inference from storage in adjacent AI workflows when they assess provider risk?

Evidence / sources

  1. 1notion.com/security/advisories/security-incident-involving-a-third-party-ai-vendor
  2. 2techcrunch.com/2026/11/12/cyberattack-on-fireworks-ai-allowed-hackers-to-access-notion-customers-chat-transcripts
  3. 3x.com/OpenAIDevs/status/2120212596850663712
  4. 4fireworks.ai/security/bulletins/october-2026-security-incident-update
  5. 5t.me/qwgai/4260
All Analysis

Next reads

Read next

Open the graph
  1. 01topicPromptsExplore the prompts topic hub.
  2. 02topicSecurityExplore the security topic hub.
  3. 03archiveAnalysisOpen the latest authorial analysis.
  4. 04sourcesSourcesSee the original sources this archive cites.
Who read this page?Machine requests, hidden until opened

Loading the privacy-safe route aggregate...

Open the JSON contract