PII · generation

Synthetic data platform

A production pipeline that scans documents and schemas once, writes a masking blueprint, then fans out synthetic copies with human review, vaultless tokenization, and an agent tool bus.

EXTRACTOR, ENSEMBLE, BLUEPRINT, EMITTER

01Pipeline

Format plugins sit at the edges. Detection, scoring, consistency, and generation are format-agnostic.

DETECT FLOW

Rendering diagram…

Layer	Contract
Extractor	File → positioned words, reconstructed page text, char-offset → word, font size
Detector	Page text → findings (span, type, score)
Pipeline	Spans → boxes; allowlist; resolve synthetic values
Generator	`(entity_type, original)` → deterministic synthetic
Emitter	Redact + redraw in place
ConsistencyMap	Memoize `(entity_type, value)` → synthetic

New formats register an extractor and an emitter. They reuse the middle.

02Composite detection

Two engines, then a merge:

Semantic NER (GLiNER-class) for names, orgs, addresses that regex cannot see.
Pattern engine (Presidio-style recognizers) for IBAN, PAN, email, phone, national ids.
Feedback agent looks up historical HITL corrections.
Ensemble scorer blends the three. The feedback signal is weighted (0.35 in the production design) so reviewer edits move the decision boundary without retraining every night.

Output is a masking blueprint: bounding boxes or JSONPath/XPath pointers, entity types, ensemble scores. The UI overlays boxes with PDF.js or canvas. Source bytes are never rewritten for preview.

03Core types

BBox = tuple[float, float, float, float]  # x0, y0, x1, y1

class PiiFinding:
    start: int
    end: int
    entity_type: str
    score: float
    source: str  # semantic | pattern | feedback

class Replacement:
    bbox: BBox
    original: str
    synthetic: str
    entity_type: str
    font_size: float

class MaskingBlueprint:
    job_id: str
    document_id: str
    findings: list[PiiFinding]
    replacements: list[Replacement]

04Consistency

Same source value → same synthetic everywhere in the document, then across a batch.

Vaultless FPE when the format is a closed alphabet (account numbers, cards).
Seeded faker when the type is free text (names), keyed by (type, original).
Redis LRU for hot maps; cold KV on miss so a 10M-row fan-out does not OOM.

05Emitter rules

Redact the original glyph run, then draw the synthetic string at the same origin and size.
Font-fit: shrink synthetic text until it fits the box; never overflow into neighboring tokens.
Overlap resolution is deterministic: higher ensemble score wins; ties break on span length then type priority.

06Config, not code

Rules, labels, thresholds, and synthesis strategies live in YAML. Adding a PII type should not require a deploy of detector code. Unknown file extensions fail closed with an actionable error.

07What this LLD is not

It is not a dump of regex tables or a POC regeneration script. Those stay in the private spec. The public design is the seams: extract, detect, blueprint, generate, emit, and the consistency map that keeps identities aligned.

SRS generator

Knowledge assistant