MLflow
MLflow records experiments and helps manage model artifacts and versions. Use it to connect a result to its inputs, code, environment and evaluation method. A logged metric is a claim about a run; it needs a definition and reproducible evidence behind it.
Keep experiment tracking, model registration and deployment approval separate. Naming a version “champion” changes a reference, not the amount of validation the model has received.
TL;DR
- Record parameters, metrics, code/data identity and artifacts together.
- Define metric units, evaluation split and acceptance rules.
- Preserve dependency and input provenance needed to recreate a run.
- Treat mutable aliases and deployment approval as controlled decisions.
- Separate traditional model evaluation from generative-AI workflow evaluation.
Quick Example
This complete Python example uses MLflow Skinny with a new local SQLite tracking store. It logs invented metadata and a text artifact; it trains no model and contacts no tracking server. It creates a uniquely named directory in the current working folder.
These checks establish local persistence of the declared records. They do not establish a good metric, a trained model, registry permissions, reproducibility or serving performance.
Core Concepts
Runs and artifacts
MLflow Tracking groups executions into runs and experiments and records metadata and artifacts. Define what each parameter and metric means; the API cannot infer whether a validation split leaks training data. Tracking concepts
Registry and aliases
The registry associates model versions with lineage and metadata. Aliases are mutable references to versions, useful for selection but requiring a controlled update policy. Capture the resolved version in deployment evidence so a later alias change does not rewrite history. Registry concepts
Reproduction and approval
Reproduction requires accessible inputs, dependency versions, preprocessing and execution conditions. Approval also needs the intended use, evaluation criteria, reviewer decision and operating requirements. A completed run or registry tag alone proves neither.
Build a useful evidence bundle
Record the code revision, input identity and authorized access path, dependency specification, preprocessing, random-state assumptions, model artifact and metric definition. Keep sensitive datasets and credentials out of ordinary parameters and public artifact stores.
Test a clean reproduction using the declared inputs and environment. Compare the result using a justified tolerance or other acceptance criterion, not an unexplained expectation of byte-identical training. Retain differences and their consequences. Some workflows are nondeterministic even when a seed is recorded.
For generative-AI workflows, also identify prompts, model/provider version where observable, retrieval configuration, evaluation cases, scoring method and protected trace handling. A traditional numeric training metric is not automatically a sufficient measure of answer correctness or safety.
Govern selection and deployment
Define who can create versions, change aliases, retrieve artifacts and approve deployment. Verify those permissions in the actual tracking/registry environment; the local SQLite fixture exercises no shared-service access control.
Before deployment, validate input/output signatures, preprocessing parity, runtime compatibility, quality on representative cases, latency/resource constraints and failure handling. Resolve the exact version and retain it with the build and configuration. Test rollback or forward recovery against the actual serving/data contract.
After rollout, observe useful outcomes and data/model drift under the agreed policy. A registry preserves useful history, but it does not decide whether an old artifact remains appropriate or whether a new input population is supported.
Best Practices
Keep definitions near results
Record units, splits, scoring and acceptance thresholds beside the metric. Avoid comparing values produced by incompatible methods.
Separate evidence from status labels
Link approval to a real decision and artifact identity. A tag named approved is only metadata until the process makes it meaningful.
Review artifact trust
Models and supporting code can introduce execution and data risks. Restrict sources and permissions and validate the serving path rather than loading unfamiliar artifacts blindly.
Common Mistakes
Bad: Calling logged parameters proof of reproducibility.
Correct: Recreate the run with declared inputs and environment and compare meaningful outputs.
Bad: Deploying a mutable alias without recording its resolved version.
Correct: Retain the exact artifact/version with the rollout evidence.
Bad: Using one training metric as complete generative-AI evaluation.
Correct: Specify task cases, retrieval/prompt context, scoring and operating acceptance separately.
FAQ
Was a model trained here?
No. The example logs and retrieves invented metadata and a local text artifact.
Does registry registration authorize deployment?
No. Define permissions, review and acceptance outside the mere existence of a version.
What should a reproducibility check preserve?
Code, data identity/access, environment, preprocessing, method and the observed comparison result.
Related Topics
- MLOps & Model Deployment — Connect tracking to deployment operation.
- ONNX — Validate exported artifact compatibility.
- Data Contracts and Schema Evolution — Keep input meaning stable.