Put the privacy boundary before the cloud

Every near-real-time analytics project arrives at the same fork within about a week. Events are landing, the dashboard is taking shape, and somebody from legal asks what is actually in the store. At that point there are two answers available. The first is everything, and we control who can see it. The second is only the parts we need, and the rest never left the building.

The first answer is easier to build and much harder to defend. It converts a data problem into an access-control problem, which means it converts a one-time engineering decision into a permanent operational obligation: every new role, every new report consumer, every export, every backup, every incident, forever. The second answer costs about two days of work at the start and closes the question.

This piece is about building the second one. It is drawn from a reference implementation we have been putting together on Microsoft Fabric — Eventstream into an Eventhouse, DirectQuery into Power BI, on an F2 capacity that is paused outside its collection window. The component described here is finished and tested. The performance and cost figures are not, and I will be explicit later about exactly what has not yet been measured.

Where the boundary goes

The redaction stage belongs in the collector, not in the cloud. That is the entire argument, and everything else is consequence. If personal data is removed before the process that holds it makes an outbound connection, then there is nothing in the cloud store to leak, nothing to audit, nothing to respond to a deletion request about, and no retention policy that can be misconfigured into a problem.

Put the same redaction one hop later — as a transformation inside the streaming service, say — and every one of those properties disappears. The raw event has now been transmitted, buffered, and probably persisted somewhere in the ingestion path. You are back to access control, only now with a diagram that makes it look like you are not.

What actually crosses

Talking about a boundary in the abstract is easy. Here is a real message going through the real code. The input is a chat line in the wire format the collector parses; the output is the object it publishes.

The input line is 663 bytes. It carries the sender's display name, their numeric account id, the text they wrote — and, because this particular message is a reply, the display name, login and account id of a second person plus the full text of the message they were replying to. Two people's words and four identifiers, in one line.

Note what survives and what does not. MessageLength, TokenCount and UniqueTokenRatio describe the shape of what someone wrote without preserving a word of it. HasLink and IsAllCaps are the signals a moderator actually cares about. None of them can be run backwards.

The tag nobody expects

Here is the part worth the price of admission, and the reason a boundary needs to be an allowlist rather than a blocklist.

Twitch sends the full text of the parent message as a metadata tag on every reply. It is called reply-parent-msg-body. It sits in the same tag section as the badges and the timestamp, looking exactly as innocuous as room-id. A collector that iterates over the incoming tags and forwards the ones it does not recognise — which is the obvious, tidy, defensible-looking way to write that code — will publish other people's messages while its author sincerely believes the text has been dropped.

A blocklist protects you from the fields you thought of. An allowlist protects you from the field the vendor adds next quarter.

The implementation therefore names the ten tags it reads and ignores everything else, including tags that do not exist yet. The test suite asserts the negative directly: it sweeps the fixture corpus and fails if any value from a known text-bearing tag appears anywhere in an emitted event.

What the hash does, and what it does not

User identifiers are replaced with an HMAC-SHA256 of the account id under a salt that lives only in the collector's environment, truncated to sixteen hex characters. This is the right construction, and it is worth being precise about what it buys, because the usual claim made for it is wrong.

It is not a one-way function of an unguessable input. Twitch account ids are numeric and drawn from a space small enough to enumerate exhaustively. Anyone holding the salt can rebuild the entire mapping in minutes. The hash is not protecting the identifier through computational hardness; it is protecting it through the secrecy of the salt, and those are different security properties with different failure modes.

Saying so plainly is not a weakness in the design. It is what tells you the controls that actually matter: the salt is never published, never logged, and never leaves the collector environment — and rotating it is a supported operation that deliberately severs the ability to link a user's activity across the rotation. That last property is a feature. A pipeline that cannot follow an individual across months is a pipeline that cannot be repurposed into one that does.

The same reasoning governs the second digest. NormalizedHash is a truncated SHA-256 of the message after lowercasing, whitespace collapsing and emote removal. It exists for exactly one purpose — detecting that the same text was posted many times, which is how you find copypasta and bot floods — and it supports that purpose without retaining anything readable. Twelve hex characters, no stored dictionary, no plaintext anywhere in the system to compare against.

How you test a negative

"No personal data reaches the store" is a claim about everything that did not happen, which is the hardest kind of claim to evidence. Four techniques, in increasing order of how much they actually prove.

Adversarial fixtures. Messages designed to break a naive implementation: a message that is itself an API credential, one containing an email address, a ten-thousand-character message, one full of Unicode direction-override characters, one with embedded newlines, one shaped like the JSON payload it is about to become. Each asserts that no four-character word and no eight-character window of the original survives into the event.

Fuzzing. Two thousand two hundred generated messages built from an alphabet of control characters, bidirectional overrides, CJK, combining marks and astral-plane code points, under a fixed seed so a failure reproduces rather than flickers. This does not prove the boundary holds. It covers the input shapes nobody thought to write a fixture for.

Structural assertions. The emitted event is a frozen dataclass with a fixed field list, and the suite pins that list. Nineteen fields; exactly four can hold a string, and each of those four is an identifier or a digest. Adding a text-carrying column fails the build before it can ship. Separately, the three modules that hold raw text are checked at the source level for imports: they may not import a logger, a socket, a filesystem path or an HTTP client, and may not call print or open. A module that cannot reach a log sink cannot leak into one, on any code path, including the ones nobody has written yet.

Mutation testing. A green suite proves nothing until you have watched it go red. We broke the boundary four ways on purpose — leaking message text into the digest field, reading the display name into the event, adding a plausible-looking "preview" column, and importing a logger into the feature module — and confirmed the suite catches each one before reverting. Eleven tests fail on the first, three on the third. That is the number that makes the other 295 mean something.

That last figure deserves a sentence. The three modules that carry message text — the parser, the feature derivation and the redaction stage — import only the Python standard library. No framework, no client, no logging package. For a prospect's security reviewer, that is a meaningfully smaller thing to audit than a dependency tree.

Why this is also the cheaper design

The governance argument tends to get made on its own, which undersells it. Minimising at collection is cheaper on four axes at once.

• Ingestion cost becomes predictable — but it is not a saving, and the measurement says so. Across the fixture corpus the published events came to 12,245 bytes against 9,839 on the wire: 24.5% more, not less. Chat messages are short and nineteen JSON column names cost more than the text they replace. What minimisation actually buys is a bound: the payload moves only 23 bytes between a 10-character message and a 10,000-character one, and break-even against the raw line arrives at a body of 71 characters. Ingestion therefore scales with message count rather than with what anyone wrote — which is the number a capacity model can be built on, and the reason a copypasta flood cannot blow up the bill. The full measurement is published as a project with its code.
• Retention. Landing tables can carry a short retention — seven days here — because the analytical value has already been extracted into a minute-grain aggregate. Enrich the aggregate, not the archive.
• Deletion requests. There is no procedure to build, because there is nothing to find.
• Review cycles. The compliance conversation happens once, at design time, instead of at every new report consumer.

For an Ontario business this maps onto PIPEDA directly. Chat messages tied to usernames are personal information handled in a commercial context. The defensible position is not a well-administered access-control matrix; it is being able to say that the personal information was never collected into the system in the first place.

What has not been measured

This is a reference implementation in progress, and the honest report is that one phase of eight is complete. The collector's parsing, feature derivation and redaction are built, tested and published. Everything to the right of the dashed line in the first figure is designed and specified, and not yet provisioned.

So there are no latency, throughput or cost figures in this article, because none have been measured. The targets are a p95 end-to-end latency under thirty seconds from message received to visible in the report, sustained ingestion of a hundred channels for a two-hour window without loss, and a total monthly cost under CAD 100 with the capacity paused outside its collection window. Whether those hold is exactly the interesting question, and publishing an estimate now would be the sort of illustrative number this firm does not publish. When the benchmark runs, the numbers go up here — including the cost per million events, which is the figure a reader can actually apply to their own volume and which almost nobody publishes.

If the latency target turns out to be unreachable on an F2, that is a finding worth writing up rather than a result worth hiding.

A note on the source data

The events in this implementation are public Twitch chat and viewer telemetry, read anonymously. The choice is deliberate and almost incidental: it is a high-volume, high-cardinality, genuinely bursty stream that is public, openly readable without a commercial agreement, and carries no client's confidentiality with it. It stands in for manufacturing line telemetry, transaction streams and IoT sensor feeds, which have the same shape and none of the same availability for a published reference.

One detail is worth borrowing regardless of the source. Reading public chat anonymously requires no OAuth flow and no per-channel authorisation, which is what makes broad coverage tractable at all; the equivalent authenticated API requires per-channel user consent and would have capped the design at a handful of channels. Finding the read path that does not require an authorisation you cannot get at scale is frequently the difference between a demo and a system.

The collector is published as a private reference implementation. If you are weighing a near-real-time operational dashboard and want to know what it costs and what it can defend, that is a conversation worth having before the architecture is chosen rather than after.