Test Categories

Drift

Test for differences in the distribution of the reference dataset as compared with the evaluation dataset. If predictions and labels are provided, measure the performance degradation caused by shifting data as well as the drift in predictions and labels themselves.

Abnormal Inputs

Check the evaluation dataset to verify that it includes sufficient numbers of abnormal values to represent the types of abnormal inputs commonly encountered in production. If model predictions are provided, test whether the evaluation set’s abnormal values cause a degradation in your model’s performance.

Subset Performance

Test that your model performs equally well across different subsets of the evaluation dataset.

Bias and Fairness

Test that your model does not discriminate or perpetuate stereotypes based on protected features such as gender, race, or disability status. AI models often inherit social biases and other biases from the datasets they’re trained on. These tests detect such biases.

Transformations

Augment your evaluation dataset with synthetic abnormal values to proactively test your pipeline’s error-handling behavior and measure the performance degradation caused by different types of abnormal values.

Model Performance

Test that your model performs well on the evaluation dataset and is not degrading in performance.

Data Cleanliness

Test for data reliability by checking that your data is consistent and complete.

Adversarial

Test the robustness of your model by measuring the worst-case change in model predictions that can be caused by small perturbations to data points.

Subset Performance Degradation

Test that your model’s performance on different subsets of the data has not degraded.

Data Poisoning Detection

Test for the presence of potentially corrupted samples in your data. Tests in this group scan the evaluation/CT data to find corrupted rows by comparing them against the reference dataset.

Evasion Attack Detection

Test whether input data contains signs of adversarial evasion attacks that could be used to cause your model to generate incorrect predictions or classifications.