Monitoring Your Model with AI Firewall
This guide will cover how to configure the AI Firewall to continuously monitor your model in production.
We will be using the IEEE-CIS Fraud Detection dataset, which is available in the rime_trial/ bundle provided during installation.
AI Firewall Continuous Tests vs. Continuous Tests Trial
We offer two workflows for firewall monitoring. In the primary workflow (“AI Firewall Continuous Tests”), the user selects their desired stress test run (corresponding to the model being deployed into production) from the UI to create a firewall. From there the user periodically uploads incoming production data to RIME.
We also offer a simpler trial workflow (“AI Firewall Continuous Tests Trial”) to demonstrate firewall capabilities without requiring the user to upload a stress test run or create a firewall. This tutorial will provide examples for both workflows.
Running AI Firewall Continuous Tests
1. Run AI Stress Testing
Kick off a run of AI Stress Testing (the reference dataset and model from this run will be used for the firewall).
rime-engine run-stress-tests --config-path examples/fraud_firewall/stress_testing_config.json
2. Configure a Firewall in the Web Client
In the web client, click on the “Default Project” and navigate to the “AI Firewall” tab on the left 
. You should see the following page.
Click on “Configure AI Firewall” and fill out the details. The “Name” field can be a name of your choice. The “AI Stress Test” field should be “Fraud Detection” (this refers to the AI Stress Testing run that we just ran). Finally, the “Bin Size” field should be “1 Day”. Once you’ve filled everything out, copy the “Firewall ID” (you will need this later to upload data and predictions to monitor) and click “Done”.
3. Run Continuous Tests on Incremental Data
We can now upload data and predictions and run AI Firewall Continuous Tests. The following command runs firewall continuous tests on data and predictions from April 29th, 2021.
NOTE: If you forgot to record the firewall ID in the previous step, you can find it by clicking the 
on the right.
rime-engine run-firewall --config-path examples/fraud_firewall/firewall_2021_04_29_to_2021_04_30_config.json --firewall-id $FIREWALL_ID
After this finishes running, you should be able to see the results in the web client, where they will be uploaded to the Default Project.
To simulate an incremental data load from the next day, you can run the same command with a different config.
rime-engine run-firewall --config-path examples/fraud_firewall/firewall_2021_04_30_to_2021_05_01_config.json --firewall-id $FIREWALL_ID
However, splitting the datasets into bin-sized batches is unnecessary — multiple bins (days in this case) can be uploaded together. They will be split automatically and firewall continuous tests will be run on each time period. For example, the following uploads data and predictions from April 1st, 2021 until April 29th, 2021.
rime-engine run-firewall --config-path examples/fraud_firewall/firewall_2021_04_01_to_2021_04_29_config.json --firewall-id $FIREWALL_ID
For additional command line options, please see the CLI Reference.
Running AI Firewall Continuous Tests Trial
To kick off a run of AI Firewall Continuous Tests Trial using 5 days of sample data:
rime-engine run-firewall-trial --config-path examples/fraud_firewall/trial_five_day_config.json
(OPTIONAL) To kick off a longer run using 30 days of sample data:
rime-engine run-firewall-trial --config-path examples/fraud_firewall/trial_config.json
After this finishes running, you should be able to see the results in the web client, where they will be uploaded to a project called “Five Day Fraud Detection - Project”.
Using Your Own Model and Datasets
This guide will cover how to run AI Firewall Continuous Tests on your own model and datasets.
Collect Data in the Correct Format
Data formatting requirements are very similar to those for AI Stress Tests (see Input Data Format); however, for continuous tests, timestamp information is also needed.
For AI Firewall Continuous Tests, RIME requires two datasets: a reference dataset and an (incremental) evaluation dataset. The latter must supply a datetime column so that it can be sliced into batches appropriately. Bin-sized slices of the evaluation dataset will be compared to the reference dataset during continuous testing.
Datasets must be passed in as a .csv or .parquet file, where each column is a separate feature, and predictions must be present in both datasets.
Create Configuration
With your data ready, you can now create a configuration file.
For a detailed reference on what the configuration should look like, see Continuous Tests Trial Configuration and Continuous Tests Configuration.
Run the CLI
AI Firewall Continuous Tests
You will need to have configured a firewall in the web client beforehand to get a Firewall ID. Please refer to the sections above for instructions.
Once complete, you can run the CLI like so:
rime-engine run-firewall --firewall-id <FIREWALL-ID> --config-path <PATH-TO-CONFIGURATION>
After this finishes running, you should be able to see the results in the web client, where they will be uploaded to the Default Project.
For additional command line options, please see the CLI Reference.
AI Firewall Continuous Tests Trial
rime-engine run-firewall-trial --config-path <PATH-TO-CONFIGURATION>
Troubleshooting
If you run into issues, please refer to our Troubleshooting page for help! Additionally, your RI representative will be happy to assist — feel free to reach out!