How We Measure and Publish Our DLP Detection Accuracy

Ask a DLP vendor what their false positive rate is. Most cannot answer. The ones that do will cite a number from a benchmark dataset that bears little resemblance to production traffic. We decided to do something different: measure accuracy rigorously, publish the methodology, and let the numbers speak for themselves.

This post explains how we measure DLP accuracy at Arbitex — the test corpus, the evaluation harness, the per-entity metrics, and the CI gates that prevent accuracy from silently degrading.

Why Accuracy Measurement Matters

A DLP system that blocks too aggressively kills productivity. One that misses too often creates compliance gaps. The only way to know where your system falls on that spectrum is to measure it — not once, not on a curated demo, but continuously against a representative corpus of labeled examples.

Most enterprise DLP products report a single aggregate accuracy number, if they report one at all. That number hides the variation that matters. A system can report a strong aggregate precision figure while silently failing on the specific entity types your compliance framework requires — Social Security numbers, medical record numbers, or financial account identifiers. Per-entity-type measurement is the only honest way to evaluate a DLP pipeline.

The Golden Test Corpus

Every accuracy measurement starts with labeled data. Our golden test corpus contains thousands of individually labeled examples spanning 80+ pattern types across seven categories: PII, financial, medical, infrastructure, secrets, base patterns, and integrated documents.

Each example is a realistic text snippet — not a synthetic pattern surrounded by whitespace, but content that resembles what enterprise users actually send through AI assistants. The corpus includes:

• True positives: Text containing genuine sensitive data in realistic conversational, clinical, financial, and technical contexts.
• True negatives: Text that looks like it might contain sensitive data but does not — invoice numbers that resemble credit cards, code variables that look like API keys, fictional names in novel analysis.
• Boundary cases: Ambiguous examples where classification depends on surrounding context — a phone number in a contact record versus a phone number in a math problem.

The corpus is versioned. Every accuracy measurement records which corpus version produced it, so results are reproducible and comparable across pipeline changes.

Per-Entity-Type Detection Accuracy

We do not report a single aggregate accuracy number as the primary metric. Instead, every evaluation produces per-entity-type precision, recall, and combined accuracy metrics.

Precision answers: when the detector flags something, how often is it right? Precision measures how often a detection is correct. For structured formats with checksum validation — payment card numbers validated by Luhn, IBANs validated by modular arithmetic, API keys validated by provider-specific format rules — precision consistently exceeds 0.95 in isolated testing. The format itself constrains false positives.

Recall answers: of all the sensitive data present, how much does the detector catch? Recall measures how much sensitive data is caught. Recall is generally high across all entity types because our regex patterns are designed to match broadly. The challenge is achieving high recall without sacrificing precision.

Combined accuracy balances precision and recall into a single metric. It is most useful for comparing detectors against each other — a high-precision, low-recall detector and a high-recall, low-precision detector can have similar combined scores, but they fail in fundamentally different ways.

For context-dependent patterns — Social Security numbers, phone numbers, email addresses — precision varies with the surrounding text. A nine-digit number is an SSN in a benefits enrollment form and a case number in a legal brief. These patterns rely on contextual signals to improve classification: field labels, corroborating entities in the same request, and contextual validation. For context-sensitive patterns, detection quality varies with the request context.

Corpus Difficulty Levels

Not all test examples are equally hard. Structured PII like credit card numbers in clean input is easy to detect. The same data embedded in a paragraph of natural language is harder. Clinical notes written in prose with no field labels are hardest.

We stratify the corpus into three difficulty levels to understand where the pipeline performs well and where it struggles.

Easy examples present the sensitive entity in isolation or in a minimal, unambiguous context — a credit card number on its own line, an SSN in a field labeled “Social Security Number,” an API key in a structured config block. Pattern matching performs near-perfectly on easy examples. False positive rates on easy negatives are also low because the context does not create ambiguity. Easy examples are useful for regression testing: if a pattern change breaks easy examples, something is wrong.

Medium examples embed the entity in realistic context — a customer service transcript mentioning a card number mid-sentence, a medical note that includes an identifier alongside clinical language, a support ticket containing an API key in a stack trace. Pattern matching still performs well on medium examples, but false positives from unrelated text with similar structure start to appear. NER recognizers add value at this level.

Hard examples challenge the entire pipeline. Sensitive data is embedded in dense, ambiguous prose — PHI in a clinical AI prompt written as a question, MNPI in an analyst’s freeform query, CUI in a contractor’s technical summary. Structured identifiers are often absent; the sensitive content is carried by the combination of entities and context. Contextual validation is the decisive tier for hard examples: it reduces false positives from professional language that triggers NER but does not constitute sensitive data in context, and it catches cross-entity PHI combinations that single-entity detectors miss.

Aggregate metrics blend all three difficulty levels. When comparing DLP systems, ask which difficulty level the benchmark uses. A vendor reporting a high accuracy number on easy examples is reporting something very different from a vendor reporting a lower number across a stratified corpus that includes hard examples. Any figure worth acting on comes with the corpus it was measured against and the date it was measured.

Isolated Versus Integrated Testing

Here is something most DLP vendors do not talk about: the difference between testing a detector in isolation and testing the full pipeline with all detectors running simultaneously.

In isolated testing, each pattern runs alone against the corpus. There is no competition between detectors, no overlapping matches, and no deduplication. Isolated results show the theoretical ceiling — what each detector can achieve when nothing else interferes.

In integrated testing, all 80+ patterns run simultaneously against every example, exactly as they do in production. This exposes real-world effects:

• Overlapping detections: The pipeline resolves overlapping detections so that the same text span is not double-counted. When two detectors match the same span, the higher-confidence result is applied.
• Cross-pattern false positives: A pattern designed for one entity type may incidentally match text that belongs to a different category. These false positives only appear when all patterns are active.
• Deduplication effects: The production pipeline deduplicates overlapping detections to avoid double-counting. This is correct behavior, but it means integrated recall can differ from isolated recall.

We run both modes in every evaluation cycle. Isolated results validate individual detector quality. Integrated results validate real-world pipeline behavior. Both matter, and reporting only one would be misleading.

Evaluating Detection Changes

Accuracy is not a one-time measurement. Every code change that touches the DLP pipeline triggers an automated accuracy evaluation in CI. The evaluation runs the full corpus through the pipeline and compares results against established baselines.

The CI gate enforces minimum accuracy thresholds for each entity type. If a code change degrades accuracy below these thresholds, the build fails. The developer sees exactly which entity types regressed and by how much.

This means accuracy can never silently degrade. A regex change that improves detection for one entity type but introduces false positives for another will be caught before it reaches production. The gate does not just check aggregate numbers — it validates per-entity-type metrics against per-entity-type baselines.

The thresholds are not aspirational targets. They are set based on measured baseline performance, with a small margin for statistical variation. As the pipeline improves, the thresholds are ratcheted up to prevent regression to previously acceptable levels.

Confidence Calibration

Every detection in the Arbitex DLP pipeline carries a confidence score between 0 and 1. A well-calibrated confidence score means that detections with confidence 0.90 are correct approximately 90% of the time.

We measure calibration by bucketing detections by their predicted confidence and comparing against the empirical precision in each bucket. The gap between predicted and empirical confidence is the calibration error.

Structured format detectors tend to be well-calibrated because their confidence is derived from format validation — a number that passes Luhn validation is almost certainly a credit card. Context-dependent detectors require ongoing calibration as the contextual validation model is updated.

Calibration matters because downstream systems — SIEM integrations, automated policy enforcement, security team workflows — use confidence scores to prioritize review. If high-confidence detections are wrong 20% of the time, security teams learn to ignore them. Well-calibrated confidence preserves the signal.

What We Publish and What We Do Not

We publish precision metrics for structured formats because those numbers are stable and verifiable. We publish the methodology so security teams evaluating DLP products can ask their vendors the same questions.

For context-sensitive patterns, detection quality depends on your deployment’s actual traffic distribution, which is why a single published figure travels poorly between environments.

The dashboard gives your security team visibility into exactly how each detector performs against your actual traffic — more useful than a brochure number, because it reflects your environment, not ours.

The Competitive Difference

We publish our accuracy because we can prove it.

We measure. We publish. They do not.

Ask any DLP vendor evaluating your shortlist three questions: What is your per-entity-type false positive rate? How do you measure it? Can I see the methodology?

If they cannot answer all three, they are asking you to trust marketing claims instead of engineering evidence. We built the measurement infrastructure because we believe accuracy should be verifiable, not asserted.

Per-entity evaluation against a labeled corpus is not a bolt-on feature — it is how we build the DLP pipeline, and every pattern ships with test coverage. Arbitex does not currently publish per-entity accuracy figures: the evaluation corpus is not yet large enough for those numbers to carry a meaningful confidence interval. We would rather publish nothing than publish a number we cannot stand behind. When the corpus supports it, the figures will be published with their corpus size and measurement date attached.

That is the standard we hold ourselves to. We think it should be the standard for the industry.

---

Related

• DLP Accuracy — Measured detection precision across all entity types
• DLP Pipeline — 3-tier inspection engine: regex, NER, and contextual validation
• DLP Protection — Inspect every AI prompt and response for sensitive data