TEST-DATA-003: Differential Privacy Audit

MLASTG-TEST-DATA-003: Differential Privacy Audit¶

Control Reference¶

• MLASVS-DATA-019: Differential Privacy ε-Guarantee (L2)
• MLASVS-DATA-022: Secure Multi-Party Computation (L2)
• MLASVS-DATA-023: Homomorphic Encryption Support (L2)

Severity¶

N/A (L1) / High (L2)

Overview¶

Differential privacy provides formal mathematical guarantees that a model's training process does not leak information about individual records. This test verifies that differential privacy mechanisms are correctly implemented, that the privacy budget is tracked, and that the epsilon value meets the documented requirements.

Prerequisites¶

Step-by-Step Procedure¶

Step 1: Verify DP Training Mechanism¶

1. Verify that a differential privacy training mechanism is in use:
2. DP-SGD (Differentially Private Stochastic Gradient Descent)
3. Privacy accountant from Opacus (PyTorch) or TF Privacy (TensorFlow)
4. diffprivlib for scikit-learn models
5. Pass if: A recognized DP training mechanism is documented and actively employed in the training process.
6. Fail if: No differential privacy mechanism is implemented or it is disabled.

Step 2: Verify Privacy Budget (ε)¶

1. Request the documented epsilon (ε) value from the model development team
2. Verify that ε is tracked and has not been exceeded across training runs
3. Apply the following thresholds based on data sensitivity:

1. Pass if: ε ≤ threshold for the specified data sensitivity classification.
2. Fail if: ε exceeds the threshold without documented justification, or if ε is untracked.

Step 3: Verify Delta (δ) Parameter¶

1. Verify that δ is set to a value smaller than 1/n (where n is the training dataset size)
2. Pass if: δ < 1/n and is explicitly documented in the privacy budget.
3. Fail if: δ is missing, undocumented, or exceeds 1/n.

Step 4: Verify Per-Sample Gradient Clipping¶

1. Review the training code for gradient clipping implementation
2. Verify that gradient clipping norm is defined and enforced
3. Pass if: Gradient clipping is implemented with a documented and enforced clipping norm.
4. Fail if: Gradient clipping is absent or the clipping norm is unconstrained.

Step 5: Test Privacy Leakage (L2)¶

1. Compare model performance on a known member vs. non-member dataset
2. If membership inference attack accuracy < 60%, privacy is likely preserved
3. Pass if: Membership inference attack accuracy remains < 60%.
4. Fail if: Attack accuracy is ≥ 60%, suggesting insufficient privacy protection and potential information leakage.

Expected Result¶

Evidence Requirements¶

• DP training mechanism documentation
• Privacy budget (ε, δ) documentation
• Gradient clipping configuration
• Membership inference test results
• Data sensitivity classification

Remediation Guidance¶

If no DP mechanism is implemented: 1. Integrate Opacus (PyTorch) or TF Privacy into the training pipeline 2. Define privacy budget policies based on data sensitivity 3. Add privacy accountant to the training loop

If ε exceeds threshold: 1. Reduce noise multiplier or increase training epochs with privacy accountant 2. Document justification if business requirements require a higher ε 3. Consider using a smaller, more sensitive dataset for high-privacy scenarios

References¶

• MITRE ATLAS:
• AML.T0018: Model Inversion
• AML.TA0010: Collection
• MLASWE: MLASWE-0005 (Membership Inference)
• NIST AI RMF: MEASURE 2.5 (Data quality), MANAGE 2.4
• Academic: Dwork et al., "The Algorithmic Foundations of Differential Privacy" (2014)