Technical robustness and security¶

Development robustness¶

AI contributors must comply with development standards to ease reliability, readability of procedures and handovers.

Practically, enforce development best practices, that lie among:

Code documentation
Unit tests
Data tests
Pipeline tests
Code reviews
Use of linter
Use of formatters

Data project lifecycle safety¶

Project phase	Guideline	Practical consequences
Data collection & preprocessing	AI contributors must account for potential biases and not introduce additional ones while preprocessing data and/or imputing missing values, leveraging data augmentation, …	Deeply understand the business process behind the data generation process, thanks to extensive discussion with business partners: When was the data collected? (One-shot vs long period, ...) Who collected the data? (General-purpose team vs task-specific team) What was targeted? (Precise subset vs random set, is the dataset representative of the the challenge that is being tackled?) How was the data collected? (Manually, sensors, ...) Conduct descriptive analysis (univariate analysis, multivariate analysis, statistical testing) Document treatments Prevent from adding unlawful or illegitimate features (example: sex for a granting score, …)
AI system training	While training AI models, AI contributors must carefully understand model’s underlying mechanisms, carefully calibrate them, and carefully analyze results	Not only compute a single aggregated performance metrics (R2, precision, recall, accuracy, or any other model evaluation metrics…), but also carefully analyze result’s dispersion, and focus on large errors. For instance, while doing forecast, not only quantitatively refer to RMSE / MAPE / SMAPE / …, but also to qualitative measurements or representations like predictions vs reality Leverage uncertainty for predictions. Examples: Prophet, MAPIE, pyMC3
AI system industrialisation	While industrialising AI models, AI contributors must ensure AI system’s sustainability	Question generalization: whether training data are a relevant proxy of what is likely to happen while predicting with the trained model on new data Implement reproducibility so that tracing back the root cause of anomalies is feasible. One solution: MLFlow Implement feedback loops to detect / anticipate / prevent drifts in the data, to prevent new bias in the data, model’s loss of performance, through the use of monitoring metrics Implement fallback plans to prevent damages. Examples: alerting, switch from statistical to rule-based procedures

Project phase

Guideline

Practical consequences

Data collection & preprocessing

AI contributors must account for potential biases and not introduce additional ones while preprocessing data and/or imputing missing values, leveraging data augmentation, …

Deeply understand the business process behind the data generation process, thanks to extensive discussion with business partners:
- When was the data collected? (One-shot vs long period, ...)
- Who collected the data? (General-purpose team vs task-specific team)
- What was targeted? (Precise subset vs random set, is the dataset representative of the the challenge that is being tackled?)
- How was the data collected? (Manually, sensors, ...)
Conduct descriptive analysis (univariate analysis, multivariate analysis, statistical testing)

Document treatments

Prevent from adding unlawful or illegitimate features (example: sex for a granting score, …)

AI system training

While training AI models, AI contributors must carefully understand model’s underlying mechanisms, carefully calibrate them, and carefully analyze results

Not only compute a single aggregated performance metrics (R2, precision, recall, accuracy, or any other model evaluation metrics…), but also carefully analyze result’s dispersion, and focus on large errors. For instance, while doing forecast, not only quantitatively refer to RMSE / MAPE / SMAPE / …, but also to qualitative measurements or representations like predictions vs reality
Leverage uncertainty for predictions. Examples: Prophet, MAPIE, pyMC3

AI system industrialisation

While industrialising AI models, AI contributors must ensure AI system’s sustainability

Question generalization: whether training data are a relevant proxy of what is likely to happen while predicting with the trained model on new data
Implement reproducibility so that tracing back the root cause of anomalies is feasible. One solution: MLFlow
Implement feedback loops to detect / anticipate / prevent drifts in the data, to prevent new bias in the data, model’s loss of performance, through the use of monitoring metrics
Implement fallback plans to prevent damages. Examples: alerting, switch from statistical to rule-based procedures

Security¶

Artificial Intelligence is no exception when it comes to security: AI systems are exposed to a wide range of attacks. Consequently, AI contributors must be sensitised to Machine Learning security and enforce security audits, especially when humans are impacted by an AI system decision. Attacks lie among:

Attack	Description	Defense
Data poisoning	Malicious individuals access data they’re not authorized to and alter them before an AI system training (in the case of a credit score e.g., change the target from “deny” to “accept” for a certain subset of the population)	Disparate impact analysis to study positive or negative discriminations (see the non discrimination section) RONI: reject on negative impact. Remove any individual from the training set that would make the accuracy abnormally decrease Self reflection: look for abnormal beneficial predictions
Backdoors and watermarks	Code of an AI system is compromised by malicious individuals so that water-marked data can lead the faulty AI system to produce desired outcome (if the AI system is a decision tree e.g., add a faulty leaf that would output a compromised decision)	Version control: AI system code must be versioned through a version control tool, most likely Git Data integrity constraints not to allow unrealistic combinations of data
Surrogate model inversion	If an AI system is served through an API, malicious individuals can simulate data, query the API and receive predictions from the AI system, and in turn build a surrogate model that could: Expose the business logic Reveal sensitive information regarding the data that were used to train the AI system Facilitate a membership inference attack Facilitate production of adversarial examples	Authentication: authenticate API users, or more generally any endpoint that would give access to the AI system predictions Throttling: decrease response time of the API so that malicious individuals can only access a limited number of predictions White-hat surrogate: act as a hacker and train a surrogate model to collect learnings about what malicious individuals could have access to
Membership inference	If an AI system is served through an API, malicious individuals could know if an individual / a row was present in the training dataset, and in turn violate individual or group privacy, by: Training a surrogate model (see above) Simulating new data points and scoring them with the surrogate model Training a new model that discriminates between data points present in the training set or not	Above defense Prediction monitoring: monitor data used to query the API and check similarity with training data
Adversarial example	If an AI system is served through an API, malicious individuals can simulate data, query the API, get predictions and in turn learn how to trick the AI system to receive a desired outcome	Above defense White-hat sensitivity analysis: try to trick the AI system with different input values

Source: H20AI

Appendix - Recommendations from the EU¶

Below are the recommendations directly reported from EU.