feat: PII detection and de-identification (Presidio) — #233

[NKeleher]

Pull Request Summary 🚀

What does this PR do? 📝

Implements PII detection and de-identification (#233), using Microsoft Presidio rather than hand-rolled spaCy NER as originally proposed. Four pieces:

1. Detection engine (processing/pii.py, new): two passes, always-available column-name heuristics (multilingual restricted-word lists ported in spirit from PovertyAction/PII_detection,
   plus a sparsity check for free-text columns), and Presidio BatchAnalyzerEngine value scanning over sampled string columns (PERSON, PHONE_NUMBER, EMAIL_ADDRESS, LOCATION, ... with confidence thresholds). Flags persist per dataset in pii_flags_{alias} (logs db) with per-column decisions: mask / drop / keep.
2. PII Review UI on the Prepare Data page: scan button, editable flags table (column, detection source, entity type, hit rate, sample matched values), in-app spaCy model download (English default; Spanish and French installable), and an apply button that converts decisions into prep steps via the new "redact column(s)" prep action (wired through the full prep-action pattern including Stata and Python replication-script emitters).
3. Export gate: build_replication_package gains include_pii (default False → de-identified). De-identified exports apply the flag decisions to every bundled dataset before anything is generated, so the raw CSV/Parquets, codebook sample values, data-dict.yaml examples, the correction-log CSV, and the value literals embedded in generated correction scripts are all covered. Undecided flags are masked (conservative default); the survey key column is never redacted (the corrections pipeline needs it).
4. With-PII exports bundle 2_scripts/5_deidentify_data.py, a uv-run-ready (PEP 723) script encoding the recorded decisions so recipients can de-identify downstream — plus a pii_flags.csv audit log.

The export page gets an export-mode radio, a flagged-column summary showing exactly what the export will do, and mode-specific warnings: the existing encrypted-storage acknowledgment for with-PII downloads, and a new
indirect-identifier warning for de-identified downloads (de-identification ≠ anonymization; quasi-identifier combinations may still identify subjects). That warning also appears in the PII Review UI, the package README, and the
de-identify script's output.

Why is this change needed? 🤔

DataSure exports survey data that routinely contains PII with no detection or removal step — the replication package writes the full raw dataset, real sample values in the codebook/data-dict, and actual data values in the correction log. The only safeguard was an all-or-nothing checkbox. See #233.

How was this implemented? 🛠️

• Presidio (presidio-analyzer + presidio-anonymizer) becomes a runtime dependency; spaCy models are not bundled — users download the small models (~15–40 MB) from within the app. en_core_web_sm is a dev dependency (direct wheel URL, allowed in dependency groups) so the Presidio path runs for real in CI; model-dependent tests skip when absent.
• Heuristics work with no model installed, so PII review degrades gracefully offline.
• Redaction is whole-value masking with entity-derived labels ([PERSON], [PHONE], ***** fallback); masked numeric columns cast to String.
• Known v1 limitations (documented): prep-log filter values aren't redacted; within-cell partial redaction is a follow-up; GPS-check CSV download gating is deferred to a follow-up PR (per discussion).

How to test or reproduce? 🧪

uv sync
uv run python -m pytest tests/processing/test_pii.py tests/replication/ tests/views/ -q
just lint-py && just fmt-python

Manual: uv run datasure on the demo project → Prepare Data → PII Review → Scan for PII (demo data has enum_name, household_latitude/longitude, village_name to flag) → set decisions → apply as prep steps. Then Export Replication Package in both modes: the de-identified zip has masked/dropped columns everywhere (filename gains _deidentified); the with-PII zip contains 5_deidentify_data.py — run uv run 5_deidentify_data.py in the unzipped package to produce *_deidentified.parquet copies.

Screenshots (if applicable) 📷

UI additions: PII Review section on Prepare Data; export-mode radio + flags summary + mode-specific warnings on Export Replication Package.

Checklist ✅

• I have run and tested my changes locally
• I have limit this PR to less than 1000 lines of code change (if not, explain why)
  • Exceeds 1000 lines: a full feature (detection engine + prep action + two UIs + export gate) with matching test coverage; ~40% of the diff is tests.
• I have updated/added tests to cover my changes (if applicable)
• I have updated/added requirements to cover my changes (if applicable)
  • Added presidio-analyzer, presidio-anonymizer (runtime) and en-core-web-sm (dev).
• I have run linting and formatting on any code changes (if applicable)
• I have updated the documentation (README, etc.) accordingly
• I have reviewed and resolved any merge conflict

Reviewer Emoji Legend

	:code:	Meaning
😃👍💯	:smiley: :+1: :100:	I like this... ...and I want the author to know it! This is a way to highlight positive parts of a code review.
⭐⭐⭐	:star: :star: :star:	Important to fix before PR can be approved... And I am providing reasons why it needs to be addressed as well as suggested improvements.
⭐⭐	:star: :star:	Important to fix but non-blocking for PR approval... And I am providing suggestions where it could be improved either in this PR or later.
⭐	:star:	Give this some thought but non-blocking for PR approval... ...and consider this a suggestion, not a requirement.
❓	:question:	I have a question. This should be a fully formed question with sufficient information and context that requires a response.
📝	:memo:	This is an explanatory note, fun fact, or relevant commentary that does not require any action.
⛏	:pick:	This is a nitpick. This does not require any changes and is often better left unsaid. This may include stylistic, formatting, or organization suggestions and should likely be prevented/enforced by linting if they really matter
♻️	:recycle:	Suggestion for refactoring. Should include enough context to be actionable and not be considered a nitpick.

[NKeleher]

Added two pseudonymization decisions in 4557ca6, per discussion:

• hash — HMAC-SHA256 pseudonyms (PERSON_3fa1b9c2) keyed with an auto-generated per-project secret salt stored in the local cache and never included in de-identified exports. Deterministic (same value → same token, stable across re-exports), so group-bys/joins/frequency analysis keep working, while dictionary attacks fail without the salt. Correction-log values for hashed columns get matching pseudonyms.
• code — readable sequential category codes (VILLAGE_NAME_001) assigned in random order (no alphabetical-rank leak) and persisted per project so re-exports stay stable.

Both apply at export time (the salt and code maps never leave the local cache in de-identified exports); mask/drop remain the prep-action path. With-PII packages embed the salt in 5_deidentify_data.py — they carry the raw data anyway — so its output matches in-app exports. Warnings now also note that deterministic pseudonyms preserve the frequency distribution (rare categories stay recognizable by rarity).

[sonarqubecloud]

Quality Gate failed

Failed conditions
C Security Rating on New Code (required ≥ A)
1 New Vulnerabilities (required ≤ 0)

See analysis details on SonarQube Cloud

💡 Need a hand with PR review? Try Gitar by Sonar!