← All news
Engineering

Introducing the OUTGATE.ai privacy model

·by Ali Khoramshahi
Circles on a warm white background shrinking and rising along a diagonal from a large black circle through grey and amber to a small red one, above a thin measuring scale.

Detection is where privacy starts. Our in-house privacy model now finds what coding agents leak, with greater performance than the model it was built on.

Keeping an agent's traffic private takes three steps. Detection finds the sensitive values: credentials, personal data, the identifiers that tie a request to a person or a system. Anonymization replaces them with placeholders before the request leaves the region. Rehydration puts the real values back into the answer and the agent's tool calls. The last two are covered here. Everything starts with detection.

In OUTGATE.ai, detection has three parts that work side by side:

  • A dense AI model that reads the text and recognises sensitive values from context, the way a person would. It is the only part that catches what nobody listed in advance.
  • Pattern matching with Microsoft Presidio for values with a fixed shape, like card numbers, IBANs and phone numbers.
  • A key-value store of known sensitive values that you fill yourself, so a value you already know about is masked wherever it appears.

The patterns and the store only find what someone anticipated. Everything else depends on the model, which is why we spent most of our effort on it. This post introduces the model we built for that job, how we measured it, and what it opens up next.

We started from OpenAI's open-weight Privacy Filter, a capable model built mostly for prose. Coding agents send source files, shell output, JSON tool results, logs and stack traces, resent on every turn.

Where the current state-of-the-art model stands

We started by analysing agent traffic and a labelled internal dataset, and put together large datasets of actual agent interactions.

A held-out sample was also reviewed by hand, so we could check the benchmark's own labels.

On that benchmark, OpenAI's Privacy Filter let through 37 % of the sensitive characters in the locked test, a set of 4,368 items that no training run ever sees. The misses were concentrated on credentials in code and config, keys in tool output and names in commit metadata. At the same time it flagged plenty of harmless machine output as personal data. Every false hit is masked across the whole conversation, which makes the agent's own context harder to read.

From the starting model to ours

We released the model in steps: small modifications, each of which had to beat its predecessor on the development set, hold up on a second training seed and pass the regression checks before it counted.

Leak and model size across milestone releases
Leak and model size across milestone releases

Our biggest findings, in order:

  1. Size optimization first. Agent traffic exercises only part of a general model. We removed what it doesn't use and retrained the rest, taking the model from 809 MB to under 400 MB with no loss on our benchmark.
  2. Labels that fit the traffic. The starting model has one label for secrets and several for personal data. We added two categories of our own: identifiers (UUIDs, hashes, record and request IDs) and timestamps. They are not personal data, but they can tie a request to a user or a system, so a policy can choose to mask them. This step alone cut the leak from a third to 13 %.
  3. Training on agent-shaped text. Longer training on material that looks like what agents send, with the original model's own judgement kept as an anchor so it didn't forget what it already did well.
  4. Smaller again. Once the model had learned the new labels, we could shrink it further, to 159 MB.
  5. Targeted fixes. Each later release fixed a specific failure we observed in the traffic: dataset names read as people, artifact file names read as secrets, timestamps in log lines and file listings that were missed.

Many candidates did not make it. Ideas that looked good on one seed and vanished on the next were dropped, as were fixes that cut false positives but let more data through.

Results on the held-out test

The selected release, v23, against the starting model, on the same locked test:

Locked test, starting model vs ours
Locked test, starting model vs ours
Locked test (4,368 items)Starting modelOur model (v23)
Sensitive characters leaked37.2 %6.2 %
Recall (exact spans)0.4300.601
Precision (exact spans)0.4620.418
F1 (exact spans)0.4450.493
Size809 MB159 MB

The leak dropped to one sixth, and F1 rose. All three changes are significant.

Precision is lower. That trade is deliberate. A leaked credential can't be taken back, while an over-masked value costs the agent a little context. The masking layer around the model recovers part of that precision with deterministic checks. One example: a plain word right after password = stays a secret, while the same word elsewhere is left alone.

The false positives that hurt agents most fell sharply:

Machine values mistaken for personal data
Machine values mistaken for personal data

Hashes and internal IDs are now tagged more often, but as identifiers, not as personal data. Whether they get masked is a policy decision, not a side effect of the model.

Speed and footprint

Measured on CPU with four threads, with no GPU required:

Starting modelOur model
512 tokens167 ms98 ms
Peak memory1.3 GB0.6 GB
Load time0.9 s0.4 s

That is small enough to run in a self-hosted region on ordinary CPUs, next to the gateway, so traffic never leaves your network to be checked.

Public benchmarks

Token-level scores on samples of four public datasets, raw model output, without the gateway's rule layer:

DatasetKind of textPrecisionRecallF1
Nemotron-PIImixed personal data0.710 → 0.7450.758 → 0.7860.733 → 0.765
Gretel PII (finance)financial documents0.482 → 0.4750.704 → 0.7690.572 → 0.587
SPYsynthetic personal data0.426 → 0.4200.903 → 0.9320.579 → 0.579
CredDatasecrets in source code0.516 → 0.3860.708 → 0.7850.597 → 0.518

Recall improves on all four. On CredData precision drops: that dataset labels only a narrow set of secrets, and our model also flags tokens and keys it doesn't count. Two rules in the masking layer were tuned against it, keeping passwords assigned in code and leaving harmless keyed slugs alone. Further work on code-heavy secrets is planned.

Customization per solution

The part we are most excited about is not this one model, but what the process makes possible. Each customer's traffic looks different: a bank sends account formats, a clinic sends record numbers, a dev shop sends commit hashes and pod names. The same pipeline can produce a model that fits each of them:

  • Your own label set. Identifiers and timestamps are already separate categories that a policy switches on or off. The same mechanism extends to domain-specific labels, such as medical record numbers, policy numbers or internal ticket IDs.
  • Your traffic, your benchmark. With opt-in traffic from your region, we build a benchmark from it and tune a model against it. The data stays in the region it came from.
  • Your ignore list as training signal. Every false positive a team dismisses is a labelled example. Today it suppresses the value; going forward it tells us which kinds of text the model misreads.
  • Your operating point. Some deployments would rather mask too much, others need the agent to see as much as possible. Precision and recall can be traded per solution, not fixed for everyone.
  • Your hardware. At 159 MB and under 100 ms for a typical request on CPU, the model fits wherever the region runs, including on-premises.

If you'd like a detection model tuned to your traffic, get in touch.

More from outgate