Technical robustness and security

Development robustness

AI contributors must comply with development standards to ease reliability, readability of procedures and handovers.

Practically, enforce development best practices, that lie among:

  • Code documentation

  • Unit tests

  • Data tests

  • Pipeline tests

  • Code reviews

  • Use of linter

  • Use of formatters

Data project lifecycle safety

Project phase


Practical consequences

Data collection & preprocessing

AI contributors must account for potential biases and not introduce additional ones while preprocessing data and/or imputing missing values, leveraging data augmentation, …

  • Deeply understand the business process behind the data generation process, thanks to extensive discussion with business partners:
    • When was the data collected? (One-shot vs long period, ...)
    • Who collected the data? (General-purpose team vs task-specific team)
    • What was targeted? (Precise subset vs random set, is the dataset representative of the the challenge that is being tackled?)
    • How was the data collected? (Manually, sensors, ...)
  • Conduct descriptive analysis (univariate analysis, multivariate analysis, statistical testing)
  • Document treatments
  • Prevent from adding unlawful or illegitimate features (example: sex for a granting score, …)

AI system training

While training AI models, AI contributors must carefully understand model’s underlying mechanisms, carefully calibrate them, and carefully analyze results

  • Not only compute a single aggregated performance metrics (R2, precision, recall, accuracy, or any other model evaluation metrics…), but also carefully analyze result’s dispersion, and focus on large errors. For instance, while doing forecast, not only quantitatively refer to RMSE / MAPE / SMAPE / …, but also to qualitative measurements or representations like predictions vs reality
  • Leverage uncertainty for predictions. Examples: Prophet, MAPIE, pyMC3

AI system industrialisation

While industrialising AI models, AI contributors must ensure AI system’s sustainability

  • Question generalization: whether training data are a relevant proxy of what is likely to happen while predicting with the trained model on new data
  • Implement reproducibility so that tracing back the root cause of anomalies is feasible. One solution: MLFlow
  • Implement feedback loops to detect / anticipate / prevent drifts in the data, to prevent new bias in the data, model’s loss of performance, through the use of monitoring metrics
  • Implement fallback plans to prevent damages. Examples: alerting, switch from statistical to rule-based procedures


Artificial Intelligence is no exception when it comes to security: AI systems are exposed to a wide range of attacks. Consequently, AI contributors must be sensitised to Machine Learning security and enforce security audits, especially when humans are impacted by an AI system decision. Attacks lie among:




Data poisoning

Malicious individuals access data they’re not authorized to and alter them before an AI system training (in the case of a credit score e.g., change the target from “deny” to “accept” for a certain subset of the population)

  • Disparate impact analysis to study positive or negative discriminations (see the non discrimination section)
  • RONI: reject on negative impact. Remove any individual from the training set that would make the accuracy abnormally decrease
  • Self reflection: look for abnormal beneficial predictions

Backdoors and watermarks

Code of an AI system is compromised by malicious individuals so that water-marked data can lead the faulty AI system to produce desired outcome (if the AI system is a decision tree e.g., add a faulty leaf that would output a compromised decision)

  • Version control: AI system code must be versioned through a version control tool, most likely Git
  • Data integrity constraints not to allow unrealistic combinations of data

Surrogate model inversion

If an AI system is served through an API, malicious individuals can simulate data, query the API and receive predictions from the AI system, and in turn build a surrogate model that could:

  • Expose the business logic
  • Reveal sensitive information regarding the data that were used to train the AI system
  • Facilitate a membership inference attack
  • Facilitate production of adversarial examples

  • Authentication: authenticate API users, or more generally any endpoint that would give access to the AI system predictions
  • Throttling: decrease response time of the API so that malicious individuals can only access a limited number of predictions
  • White-hat surrogate: act as a hacker and train a surrogate model to collect learnings about what malicious individuals could have access to

Membership inference

If an AI system is served through an API, malicious individuals could know if an individual / a row was present in the training dataset, and in turn violate individual or group privacy, by:

  • Training a surrogate model (see above)
  • Simulating new data points and scoring them with the surrogate model
  • Training a new model that discriminates between data points present in the training set or not

  • Above defense
  • Prediction monitoring: monitor data used to query the API and check similarity with training data

Adversarial example

If an AI system is served through an API, malicious individuals can simulate data, query the API, get predictions and in turn learn how to trick the AI system to receive a desired outcome

  • Above defense
  • White-hat sensitivity analysis: try to trick the AI system with different input values

Source: H20AI

Appendix - Recommendations from the EU

Below are the recommendations directly reported from EU.

_images/robustnesssafety_1.png _images/robustnesssafety_2.png _images/robustnesssafety_3.png