TEST-DATA-005: Federated Learning Poisoning

MLASTG-TEST-DATA-005: Federated Learning Poisoning Detection¶

Control Reference¶

• MLASVS-DATA-001: Data Provenance Tracking
• MLASVS-DATA-002: Data Sanitization
• MLASVS-DATA-004: Training Data Integrity Verification
• MLASVS-DATA-006: Data Quality Monitoring
• MLASVS-DATA-010: Data Source Validation
• MLASVS-DATA-019: Differential Privacy for Distributed Training (L2)
• MLASVS-DATA-023: Secure Aggregation Verification (L2)

Severity¶

High (L1) / Critical (L2)

Overview¶

Federated learning (FL) distributes training across multiple participants who never share raw data. This distributed architecture introduces unique poisoning attack surfaces: malicious participants can submit poisoned local model updates (model poisoning), manipulate local training data (data poisoning), or collude to skew the aggregated model. Byzantine-resilient aggregation, contribution auditing, and anomaly detection on updates are essential defenses. This test evaluates the resilience of federated learning pipelines against these attacks.

Prerequisites¶

Step-by-Step Procedure¶

Step 1: Map Federated Learning Architecture¶

1. Document the FL system components:
2. Aggregation server (central coordinator)
3. Participant nodes (edge devices, hospitals, organizations)
4. Communication protocol (gRPC, HTTP, MQTT)
5. Aggregation strategy (FedAvg, FedProx, Secure Aggregation)
6. Model update format (gradients, weights, deltas)
7. Identify trust boundaries between participants and the server
8. Pass if: Complete FL architecture documentation exists with clearly defined trust boundaries.
9. Fail if: The architecture is undocumented or trust boundaries between server and participants are undefined.

Step 2: Test Byzantine-Resilient Aggregation¶

1. Simulate Byzantine participants submitting poisoned model updates:
2. Label flipping: Submit updates trained on inverted labels
3. Scaling attack: Multiply gradients by a large factor (e.g., 100x)
4. Noise injection: Submit random Gaussian noise as model updates
5. Zero update: Submit a zero vector (free-riding)
6. Inject these poisoned updates into the aggregation round alongside honest updates
7. Measure the impact on global model accuracy:

   import numpy as np
   
   def label_flip_update(honest_update, flip_ratio=0.3):
       """Simulate a label-flipping attack on local gradients."""
       poisoned = honest_update.copy()
       n_params = len(poisoned)
       flip_indices = np.random.choice(n_params, int(n_params * flip_ratio), replace=False)
       poisoned[flip_indices] *= -1
       return poisoned
   
   def scaling_attack(honest_update, scale_factor=100):
       """Simulate a gradient scaling attack."""
       return honest_update * scale_factor
   

8. Pass if: Global model accuracy degrades by < 10% when up to 30% of participants are Byzantine.
9. Fail if: A single malicious participant or a small minority can degrade global model accuracy by > 20%.

Step 3: Test Gradient Inversion Attack¶

1. Attempt to reconstruct training data from shared gradients/updates:
2. Submit a known input to the FL protocol
3. Capture the resulting gradient update
4. Attempt reconstruction using gradient inversion techniques (DLG, Deep Leakage)
5. Measure reconstruction fidelity (cosine similarity, MSE between original and reconstructed)
6. Pass if: Gradient inversion techniques fail to recover meaningful training data.
7. Fail if: Original training samples can be reconstructed with high fidelity from shared gradients.

Step 4: Test Contribution Auditing (L2)¶

1. Verify that each participant's contribution is logged with:
2. Participant ID (anonymized or pseudonymous)
3. Round number
4. Update magnitude and direction
5. Model performance contribution (if tracked)
6. Test whether a participant can submit updates under a different identity
7. Pass if: All contributions are appropriately attributed and identity spoofing is effectively prevented.
8. Fail if: Participants can submit anonymous updates or forge the identity of other participants.

Step 5: Test Secure Aggregation (L2)¶

1. Verify that the aggregation protocol prevents the server from inspecting individual updates:
2. Homomorphic encryption (e.g., CKKS scheme)
3. Secure multi-party computation (SMPC)
4. Differential privacy noise addition
5. Attempt to extract individual participant updates from the aggregated model
6. Pass if: Individual updates cannot be reconstructed or inspected by the server prior to aggregation.
7. Fail if: The aggregation server has plaintext access to and can inspect individual participant gradients.

Step 6: Test Differential Privacy in FL (L2)¶

1. Verify that differential privacy (DP) is applied to local updates before sharing:

   # Gradient clipping + Gaussian noise
   def apply_dp_to_update(update, max_norm=1.0, noise_multiplier=0.1):
       clipped = np.clip(update, -max_norm, max_norm)
       noise = np.random.normal(0, noise_multiplier * max_norm, clipped.shape)
       return clipped + noise
   

2. Verify that the privacy budget (epsilon, delta) is tracked across rounds
3. Pass if: DP is enforced on shared updates and the privacy budget (epsilon, delta) is properly documented.
4. Fail if: DP is not applied to local updates before transmission, or the privacy budget is exceeded/untracked.

Step 7: Test Model Update Validation¶

1. Verify that the aggregation server validates incoming updates:
2. Update shape consistency with global model
3. Update norm bounds (reject abnormally large gradients)
4. Update freshness (reject stale or replayed updates)
5. Submit malformed updates and verify rejection:
6. Wrong tensor dimensions
7. Updates exceeding norm thresholds
8. Replayed updates from previous rounds
9. Pass if: All malformed, oversized, or replayed updates are correctly rejected.
10. Fail if: Malformed or invalid updates are accepted and aggregated into the global model.

Expected Result¶

Evidence Requirements¶

• FL architecture diagram with trust boundaries
• Byzantine resilience test results (accuracy with poisoned participants)
• Gradient inversion attack results
• (L2) Secure aggregation verification evidence
• (L2) DP configuration with privacy budget documentation
• Update validation test results

Remediation Guidance¶

If Byzantine resilience is insufficient: 1. Implement robust aggregation (Krum, Trimmed Mean, Median Aggregation) instead of FedAvg 2. Add anomaly detection on incoming updates (norm-based, direction-based) 3. Limit the maximum number of participants per round

If gradient inversion succeeds: 1. Add differential privacy noise to local gradients before sharing 2. Increase gradient clipping to reduce information leakage 3. Consider secure aggregation (homomorphic encryption or SMPC)

If update validation is absent: 1. Implement server-side validation for update shape, norm, and freshness 2. Reject updates exceeding a norm threshold (e.g., > 2x median norm) 3. Track and reject replayed updates using round-number binding

References¶

• MITRE ATLAS:
• AML.T0020: Poison Training Data
• AML.T0059: Backdoor ML Model
• MLASWE: MLASWE-0002 (Data Poisoning), MLASWE-0009 (Supply Chain Compromise)
• Academic: Bagdasaryan et al., "How to Backdoor Federated Learning" (2020)
• Academic: Zhu et al., "Deep Leakage from Gradients" (2019)
• NIST AI RMF: MAP 1.5 (Risk identification for distributed systems)
• Framework: FedML, PySyft, Flower FL security documentation