#1639·presidio

Evaluate precision/recall/latency during CI

Author: omri374Created Jun 22, 2025Updated Aug 8, 2026
LabelsenhancementPII recognizersPerformanceanalyzer

Is your feature request related to a problem? Please describe.

As new predefined recognizers are proposed, the CI should test if those have logic which has side effects on other parts of the codebase, by running an evaluation on a synthetic dataset and measuring precisio, recall and latency.

Describe the solution you'd like During CI, a synthetic dataset is created (Based on the presidio-evaluator data generator logic), and precision/recall values are measured. For every new contributed recognizer, the contributor would have to add the logic for creating synthetic samples for this entity.

High-level flow:

  • A contributor adds a new recognizer via PR
  • The contributor adds a few templates with the entity (e.g. "My name is {{name}}"), and a faker provider that generates this value, if doesn't exist
  • The synthetic data is generated during CI and precision / recall values are collected
  • CI fails if the metrics drop significantly.

Describe alternatives you've considered Manually estimating the side effects of predefined recognizers (currently the practice)

Additional context https://github.com/microsoft/presidio-research/blob/master/notebooks/1_Generate_data.ipynb https://github.com/microsoft/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb

Source: data-privacy-stack/presidio