Monitoring Your Model with AI Firewall
This guide will cover how to configure the AI Firewall to continuously monitor your model in production.
We will be using an image classification model trained on the Animals with Attributes 2 Dataset.
CV Setup
Please ensure that the extra RIME CV dependencies have been installed from the cv_requirements.txt file from installation.
If you run into a ModuleNotFoundError at any point during this walkthrough, it is likely that you need to install the RIME CV Extras!
pip install -r cv_requirements.txt
AI Firewall Continuous Tests vs. Continuous Tests Trial
We offer two workflows for firewall monitoring. In the primary workflow (“AI Firewall Continuous Tests”), the user selects their desired stress test run (corresponding to the model being deployed into production) from the UI to create a firewall. From there the user periodically uploads incoming production data to RIME.
We also offer a simpler trial workflow (“AI Firewall Continuous Tests Trial”) to demonstrate firewall capabilities without requiring the user to upload a stress test run or create a firewall. This tutorial will provide examples for both workflows.
Running AI Firewall Continuous Tests
1. Run AI Stress Testing
Kick off a Stress Testing run (the reference dataset and model from this run will be used for firewall continuous tests).
rime-engine run-images --config-path images_examples/classification/awa2/stress_test_config.json
2. Configure a Firewall in the Web Client
In the RIME web client, click on the “Default Project” and navigate to the “AI Firewall” tab on the left 
. You should see the following page.
Click on “Configure AI Firewall” and fill out the details. The “Name” field can be a name of your choice. The “AI Stress Test” field should be “Image Classification” (this refers to the AI Stress Testing run that we just ran). Finally, the “Bin Size” field should be “1 Day”. Once you’ve filled everything out, copy the “Firewall ID” (you will need this later to upload data and predictions to monitor) and click “Done”.
3. Run Continuous Tests on Incremental Data
We can now upload data and predictions and run AI Firewall Continuous Tests. The following command uses data from March 1, 2022 to March 13, 2022.
NOTE: If you forgot to record the firewall ID in the previous step, you can find it by clicking the 
on the right.
rime-engine run-firewall-images --config-path images_examples/classification/awa2/firewall_config.json --firewall-id <FIREWALL-ID>
After this finishes running, you should be able to see the results in the web client, where they will be uploaded to the Default Project.
For additional command line options, please see the CLI Reference.
Running AI Firewall Continuous Tests Trial
To kick off a run of AI Firewall Continuous Tests Trial:
rime-engine run-firewall-images-trial --config-path images_examples/classification/awa2/firewall_trial_config.json
After this finishes running, you should be able to see the results in the web client, where they will be uploaded to a project called “Image Classification (Firewall Monitoring Trial) - Project”.
Using Your Own Model and Datasets
This guide will cover how to run AI Firewall Continuous Tests on your own model and datasets.
Collect Data in the Correct Format
Data formatting requirements are very similar to those for AI Stress Tests (see Input Data Format); however, for continuous tests, timestamp information is also needed.
For AI Firewall Continuous Tests, RIME requires two datasets: a reference dataset and an (incremental) evaluation dataset. The latter must supply timestamps with each data point, using the key "timestamp". This allows the data to be sliced into batches appropriately, each of which will be compared to the reference dataset during continuous testing.
Datasets must be passed in as .json or .jsonl files, optionally compressed via gzip (creating json.gz or jsonl.gz).
Create Configuration
With your data ready, you can now create a configuration file.
For a detailed reference on what the configuration should look like, see Continuous Tests Trial Configuration and Continuous Tests Configuration.
Run the CLI
AI Firewall Continuous Tests
You will need to have configured a firewall in the web client beforehand to get a Firewall ID. Please refer to the sections above for instructions.
Once complete, you can run the CLI like so:
rime-engine run-firewall-images --firewall-id <FIREWALL-ID> --config-path <PATH-TO-CONFIGURATION>
After this finishes running, you should be able to see the results in the web client, where they will be uploaded to the Default Project.
For additional command line options, please see the CLI Reference.
AI Firewall Continuous Tests Trial
rime-engine run-firewall-images-trial --config-path <PATH-TO-CONFIGURATION>
Troubleshooting
If you run into issues, please refer to our Troubleshooting page for help! Additionally, your RI representative will be happy to assist — feel free to reach out!