How We Optimized the DLP Pattern Matching Engine

The Arbitex DLP pipeline inspects every AI prompt and response through a multi-layer content inspection pipeline: pattern matching with checksums, named entity recognition, and contextual validation. The pipeline works. Detection accuracy is high. False positive rates are low.

But the first tier had a performance problem.

The Bottleneck: Sequential Pattern Evaluation

Tier 1 runs dozens of patterns against every request. In the original implementation, that meant sequential evaluation — one pass over the text per pattern. After matching, checksum validators (Luhn and others) run on every candidate. Then overlap resolution handles overlapping matches.

For short prompts, this is fast enough. For enterprise traffic at scale — thousands of requests per minute, some carrying multi-kilobyte context windows — the sequential evaluation loop becomes the latency floor.

The workload is deterministic. The patterns are stable. The checksum algorithms are pure arithmetic. This is exactly the kind of work that benefits from a compiled, optimized implementation.

The Solution: A Compiled Pattern Matching Engine

We replaced the sequential pattern evaluation loop with a compiled pattern matching engine. The architecture change is narrow — only the hot loop was replaced. The rest of the DLP pipeline is unchanged.

What the compiled engine does:

• Single-pass multi-pattern matching. All patterns are evaluated simultaneously in one pass over the text, instead of sequential passes. When a set-level match is found, individual patterns extract spans and capture groups.
• Compiled checksum validators. Luhn and other checksum algorithms run natively with no interpreter overhead. Each match is validated in the same call that found it.
• Efficient overlap resolution. Same-type overlaps keep the highest confidence match. Cross-type overlaps keep both with a confidence penalty to the lower-scored detection.
• Pattern set caching. Compiled pattern sets are cached. A cache hit skips compilation entirely.

What is unchanged:

The async coordination layer, circuit breakers, NER (named entity recognition), AI-powered contextual validation, content classification, and the policy engine are all unchanged. The pattern matching engine is a leaf node — it takes pattern definitions and text, returns matches with metadata. No awareness of compliance packs, org configuration, or enforcement policy.

Pack-Driven Pattern Sets

In a multi-tenant deployment, each organization enables a different combination of compliance packs — PCI-DSS, HIPAA, Financial, General PII, Secrets/DevOps — with per-rule toggles (an admin can disable individual detectors within a pack) and optional org-defined custom patterns.

The effective pattern set for a request is the union of all enabled packs, minus any admin-disabled rules, plus any org custom rules.

Rather than compiling a unique pattern set per organization, the engine caches compiled pattern sets by configuration. Organizations with identical pack configurations share the same compiled set. In practice, most organizations use standard pack configurations — the number of unique compiled sets across a full customer base is small.

Cache hits are the common case. Compilation only happens on the first request for a new pattern combination, or after an admin changes pack configuration.

Intelligent Tier Routing: Skip What You Don’t Need

The original pipeline ran named entity recognition (NER) on every request, unconditionally. NER is computationally expensive. It exists to catch unstructured sensitive data that pattern matching cannot see: person names, medical terms, addresses.

But not every compliance pack requires NER. PCI-DSS detects payment card data — structured formats with Luhn checksums. The Secrets/DevOps pack detects API keys and tokens — structured formats with entropy validation. Neither produces unstructured entity candidates that need NER resolution.

The optimized pipeline adds intelligent tier routing: if no enabled pack for the requesting organization requires NER, the entire NER tier is skipped. This is a pack-level decision. An organization running PCI-DSS plus Secrets — all structured data formats — bypasses NER entirely without any reduction in detection coverage for the entity types those packs target.

For organizations with HIPAA or General PII packs (which require NER for names, locations, and medical identifiers), a content pre-filter screens requests before NER invocation. If the text contains no keywords that correlate with NER-detectable entities, NER is skipped for that request even when the pack technically requires it.

Smarter Contextual Validation Invocation

Tier 3 runs an AI-powered contextual validator on entities that Tier 2 escalates — cases where NER confidence is ambiguous and context needs to resolve whether the candidate is real sensitive data or a false positive.

The original pipeline also escalated some pattern-only matches to contextual validation. This was conservative but expensive — a pattern match without checksum validation (some entity types lack verifiable check digits) would invoke contextual validation even when the pattern confidence was high.

The optimized pipeline refines this: high-confidence pattern matches skip contextual validation. These are high-certainty detections where the pattern and surrounding format strongly indicate real sensitive data, even without a checksum. Contextual validation is reserved for ambiguous matches — the cases where contextual resolution actually changes the outcome.

The result is fewer contextual validation calls per request without increasing false positive rates on the matches that skip it. The accuracy harness confirms no regression: detection accuracy remains within tolerance of the ungated baseline.

Parity Verification

Replacing a detection engine in a security pipeline carries an obvious risk: if the new engine produces different results, you have either false negatives (missed detections) or false positives (new noise). Either outcome erodes trust in the system.

The validation approach is zero-delta: the new engine must produce identical matches to the previous implementation on the full accuracy corpus. Not “similar.” Not “within tolerance.” Identical — same entity types, same spans, same confidence scores, same checksum results.

The accuracy corpus covers an extensive set of labeled examples across all supported patterns, including known-positive samples that must be detected and known-negative samples that must not trigger false positives. The optimized engine passes the same test suite that validated the original implementation, with zero detection deltas.

Engine Selection Criteria

When evaluating pattern matching engines, two requirements drove the final choice:

Full pattern syntax support. Many of our patterns use advanced constructs — context-sensitive matching and similar constructs that require examining text around the match position. Not all high-performance engines support these. Rewriting patterns to work around engine limitations would change detection behavior.

Cross-platform portability. The same engine runs on the SaaS platform and on the Outpost — customer-managed infrastructure with variable hardware. The chosen engine compiles and runs identically across the hardware configurations our customers deploy, without platform-specific dependencies.

What This Means in Practice

The optimized engine is a performance improvement, not a detection change. The same patterns, the same checksum validators, the same accuracy guarantees. What changes is how fast the work gets done and how much unnecessary downstream work is avoided.

For organizations running PCI-DSS and Secrets packs: pattern scanning is faster, NER is skipped entirely, and contextual validation is only invoked when ambiguity genuinely exists. The pipeline does less work per request without reducing coverage.

For organizations running HIPAA or General PII packs: pattern scanning is faster, NER runs when content pre-filtering indicates it is needed, and contextual validation calls are reduced to the cases that benefit from it.

The security posture is unchanged. The compliance evidence is unchanged — the audit log records the same detection events, the same enforcement actions, the same policy references. The pipeline is faster, quieter, and spends its compute budget where it matters.

---

For more on the DLP pipeline architecture, read the complete pipeline series. For accuracy methodology, see how we measure DLP accuracy. For the contextual validator, see accuracy improvements for international and healthcare identifiers.

Book a demo to see the optimized DLP pipeline inspect your AI traffic.