Databricks has released MLflow in version 1. 12. The platform for managing machine learning projects (ML) in the current release mainly brings some improvements in interaction with PyTorch. The developers presented the innovations during the first PyTorch Developer Day, which took place on 12. November took place as a virtual event.

MLFlow is a platform for managing the life cycles of ML projects. Behind this is Databricks, the company that the developers of Apache Spark founded. MLflow consists of three essential components for tracking experiments (Experiment Tracking), for managing ML models (Model Management) and for distributing models in productive operation (Model Deployment).

The Software integrates ML projects in Git and Conda, among others, and takes care of the versioning of models. In addition, the projects can be integrated into CI / CD workflows (Continuous Integration, Continuous Delivery) and tools.

Microsoft has been involved since spring 2019 to MLflow, and since June 2020 the project is under the umbrella of the Linux Foundation. In addition to the open source project, Databricks offers a commercial version that contains additional functions.

MLflow voluntarily keeps logs MLflow 1. 12 leads independently of the extended PyTorch Integration of a universal autlogging function: The method mlflow.autolog () should automatically include all relevant model properties like Log parameters, metrics, and artifacts. So far, it was necessary to call the respective methods for the individual entities.

The current release brings the API for interaction with PyTorch mlflow.pytorch.autolog for automatic logging of metrics, parameters and models with. MLflow can also create automated logs of PyTorch Lightning models. Version 1.0 of the performance-optimized framework for model training was released in October.

Managed scripts MLflow can now also load and manage TorchScript. TorchScript can be used to create models that can be serialized on the one hand and do not require any Python dependencies on the other. The translation with the just-in-time compiler (JIT) can be triggered from MLflow, as can loading and logging, as the following code from the Databricks blog shows:

# Any PyTorch nn.Module or pl.LightningModule model = Net () scripted_model = torch.jit.script (model) .. . mlflow.pytorch.log_model (scripted_model, “scripted_model”) model_uri = mlflow.get_artifact_uri (“scripted_model”) loaded_model = mlflow.pytorch.load_model (model_uri) … Freshly served and skillfully explained Version 1 brings the distribution of applications. 12 a plug-in for integration into TorchServe with. Facebook presented the deployment library together with Amazon Web Services in the spring. Via the plug-in mlflow-torchserve , models trained from MLflow pipelines can be provided with TorchServe.

The plug-in transfers the previously trained models to productive operation via TorchServe.

(Image: Databricks)

Another innovation beyond the PyTorch integration is the method mlflow.shap.log_explanation for logging model explanations according to SHapley Additive exPlanations (SHAP), an approach based on game theory, the output of ML models to explain.

Further innovations in MLflow 1. 12 can be found on the Databricks blog. A full list of additions and bug fixes can be found in the release notes on GitHub. Developers can install the software via the Python package manager PyPI with the command pip install mlflow . The source code is stored in the GitHub repository.

(rme)