Part 2: Synthetic PII Data and Deterministic Tokenization

Implementation and documentation reviewed: 2026-09-12. Generator 1.0.0, seed 42 and existing dataset hashes are preserved.

Extraction is separate from replacement

The model emits candidate TYPE<TAB>ORIGINAL rows:

PERSON	Taylor Sample
EMAIL	synthetic.en.1494@example.com

These are synthetic examples. The model does not edit the source document. Type and source-match checks reject some invalid candidates, but do not prove that a value is PII, its type is correct, or every PII entity was found. Treat the replacement mapping as sensitive.

Dataset and evaluation scope

The dataset has 2,200 records with an 80/20 Korean/English split in each partition. Regeneration after the implementation changes preserved all manifest hashes.

Split	Records	Korean	English	SHA-256
train	1,600	1,280	320	b98429fef0b103f24e8eaded069cbd2f6def5fbf8c083a5c7baf366c9fc1d21a
validation	200	160	40	25ca38198d38e04be181e15b4e21a3c96d672f46f775ae1bc6c422ee4514f820
test	400	320	80	6f6ef9a6b42297738b292d5149f2e6e323f7bcd6f2325b6bfbc04ae6d9d0ec21

Label	Meaning in this experiment
PERSON	Person name
RRN	Korean resident-registration-number-shaped identifier
DOB	Date of birth
REL	Family or relationship expression
ADDRESS	Address
PHONE	Phone number
EMAIL	Email address
ACCOUNT	Bank account number
CARD	Payment card number

These are experiment-specific annotation rules. For example, relationship words can be positive, while company-switchboard strings appear in negative documents. They are not a universal sensitivity taxonomy.

The generator uses fixed templates, small name vocabularies and synthetic numbers. RRN/CARD values fail the example checksum functions; that alone does not establish official identifier validity or non-assignment. PHONE values are synthetic placeholders, not validated national formats or official reserved ranges. No customer data is used, but that differs from authoritative identifier validation.

Different records and hashes do not ensure independent templates or entity vocabularies across train/validation/test. Do not present this dataset as a real-workload generalization benchmark. Use separate holdouts and reviewed annotations for that purpose.

Reversible source-spelling replacement

The implementation now follows this sequence:

1. Normalize source/values to NFC and trim outer type/value whitespace.
2. Remove complete <think>...</think> blocks, then read tab-separated allowed types and nonempty values.
3. Deduplicate candidates and require a literal or limited variant to match the source. Parsing and replacement use the same numeric boundaries.
4. Prefer literal predictions over variants generated from other predictions. A fixed type priority resolves multiple types for one value; it does not infer semantic truth.
5. Scan the source once, with longer patterns first.
6. Assign tokens in actual source order by type and matched NFC spelling. Identical spellings reuse a token; different spacing/separators receive separate tokens.
7. Skip token names already present in the original source.
8. Store the actual matched spelling in the mapping and record half-open [start, end) replacement spans in NFC source coordinates.

Source: 김가상 / 김 가 상 / [PERSON_1]
Candidate: PERSON	김가상
Masked: [PERSON_2] / [PERSON_3] / [PERSON_1]

The original marker remains unchanged and each source spelling can be restored. Token numbering need not start at one. Token equality is not real-world entity identity resolution.

PERSON variants cover limited uniform spacing/tab/newline forms for names of 2–6 characters after whitespace removal. Numeric variants only normalize the permitted digit/whitespace/separator characters. They no longer discard invented letters to manufacture a source match: an absent alias123456 is not accepted merely because 123456 occurs in the source. An alphanumeric original can still match literally. These rules are not official phone/account/identity validators.

reassemble_text performs one replacement pass for known tokens and preserves unknown tokens. The evaluator checks round trips. Equality is against NFC source text, not byte-identical restoration of an original NFD representation.

Measure remaining source spans, not placeholder contents

The earlier evaluator searched for complete gold strings in the masked text. This produced two incorrect results:

• Masking only Alpha in the gold entity Alpha Beta removed the complete string and incorrectly reported no leakage.
• Masking a literal gold value PERSON produced [PERSON_1], whose label was incorrectly counted as leaked source text.

The corrected evaluator checks whether the union of actual replacement spans fully covers every matched occurrence of each gold value or permitted variant. It examines repeated occurrences too. Partial coverage, including an uncovered separator, counts as uncovered under this conservative metric. Characters inside generated token names are not treated as source exposure.

Gold records have values rather than annotation offsets, so these spans are inferred through source matching. This is not new PII detection and can match a contextually non-sensitive occurrence of the same string. Interpret coverage against the annotation policy, not as a privacy guarantee.

Exact meanings of the metrics

Result field	Calculation
entity/per_type precision, recall, F1	Sum TP/FP/FN over per-document normalized (TYPE, ORIGINAL) sets
documents.leak_rate	Documents with an uncovered gold span divided by all documents
entities.leak_rate	Unique gold pairs with any uncovered occurrence divided by unique gold pairs
entities.over_redaction_rate	Legacy field name for extra predicted-pair rate: FP / predicted pairs
entities.hallucination_rate	Nonempty allowed-type TSV rows without a source match divided by those rows
parse.success_rate	Fraction of documents whose caller-provided parse_success flag is True
tokenization.deterministic_rate	Fraction with identical masked text, mapping and spans after reversing candidate order
tokenization.round_trip_rate	Fraction restoring to the NFC source

Entity F1 is not span-level NER F1. Duplicate pairs within a document are removed; the same pair in different documents counts separately. Types are trimmed and uppercased; values are trimmed and NFC-normalized. A valid variant can mask the source completely while differing from the gold's exact spelling and producing FP/FN. The legacy over-redaction field therefore does not directly measure unnecessarily removed characters.

Hallucination uses source matching independently of the parse flag. Unknown types and ordinary prose are outside its row denominator. The current inference helper marks an empty output or an output containing at least one usable source-matching row as parseable; this is not strict validation of every output line. The evaluator excludes entity predictions when that flag is False.

Zero denominators return zero, including F1 for empty entity sets. Duplicate record/prediction IDs, predictions for unknown records and gold annotations that cannot match the source are rejected. Errors do not echo raw entity values.

Training records and validation

JSONL records contain source_text, entities and target_tsv. The loader constructs system/user prompts and assistant completions. The trainer requests completion-only loss; validate the actual tokenizer/template, truncation and loss mask during execution. Keep source, completions and mappings out of ordinary logs and MLflow parameters/tags.

Regression tests reproduced token collisions, lossy variant restoration, numeric variant over-acceptance and evaluation errors before the fixes. All 50 local tests pass. A 2,200-record ground-truth oracle check also verifies hash preservation, determinism, restoration and coverage. Oracle sanity results are not model predictions or fine-tuned F1 measurements. No GPU training or real-customer PII processing was performed.

References

• Synthetic generator
• Dataset manifest
• Parser and replacement
• Evaluation implementation

Previous: Platform architecture

Next: SageMaker / MLflow execution

Quiz