Keep a versioned model repository
In some cases you will want the ability to know why a decision was made, for example, if there is an unexpected output or someone challenges a recommendation. Indeed, in most regulated environments it is essential to be able to show how a given decision or recommendation was reached, so you know which version of your machine learning model was live when a specific decision was made. To meet this need you will need a store or repository of the models that you can query to find the version of the model in use at a given date and time.
In the past we have used a variety of ways to version our models:
- S3 buckets with versioning enabled
- S3 buckets with database to to store model metadata
- MLflow model registry
- DVC to version both the model and the data used to create that model
- Cloud provider model registries (AWS Sagemaker, Google Vertex AI , Azure MLOps)
- Some models can have their coefficients stored as text, which is versioned in Git