Support:
Star 🌟 the Project: 
Follow 🤝 on GitHub: 
` Design · Develop · Deploy · Iterate `
Learn how to combine machine learning with software engineering to design, develop, deploy and iterate on production-grade ML applications.
In this project we shall use RAY Your personal laptop (single machine) will act as the cluster, where one CPU will be the head node and some of the remaining CPU will be the worker nodes.
-
Create a repository by following these instructions: Create a new repository → name it Preferred-Name → Toggle Add a README file (very important as this creates a main branch Click Create repository (scroll down)
-
Execute the command below in your terminal ( Command prompt or shell )
cd Preferred-Name
git clone https://T-droid-Ollila/ML_Churn_Engineering.git.
Every time a commit is made and you want to push changes made to your created repo , first set up a remote repo of your choice
git remote set-url origin https://github.com/<their-user>/<Preferred-Name>.git
git push -u origin main
$env:PYTHONPATH = "$env:PYTHONPATH;$pwd # MacOs export PYTHONPATH=$PYTHONPATH:$PWD
py 3.12 -m venv virtual # recommend using Python 3.12 # For MacOS python3.12 -m venv .virtual
virtual\Scripts\activate # On MacOS source virtual/bin/activate
python3 -m pip install --upgrade pip setuptools wheel
python3 -m pip install -r requirements.txt
pre-commit = 3.8.0 install
pre-commit autoupdate
pip list
# Then retrieve the versioned data: On first use, authenticate with your own Google account.
# Your Google account must first be granted access to the project's
# Google Drive DVC folder.
To request access, contact the repository owner and provide the
Google account that should be authorized.
dvc pull
- The intuition behind remote storage is due to Github limiting sizes of files that can be committed to github especially large datastets it is best practice to store the data in a remote stoarge that can handle the large size of the files without bloating your repo.Thus we shall Google drive remote storage which is dvc enabled.
To configure your local DVC remote storage using Google Drive, follow these steps to ensure secure authentication and proper synchronization
Prerequisites:
- A created Google Drive folder with google drive account
- A project in your google cloud console.
- Client ID and secret obtained from your respective project in the console .
FolderID refers to the ID at the top of the search tab after creating a folder or when you entered into one
Typically it will look like this
Google Drive folder URL: https://drive.google.com/drive/folders/15W2JmKpx8IdwoLEfotwNwupL7Cawx52dLe
Google Drive Folder ID:
15W2JmKpx8IdwoLEfotwNwupL7Cawx52dLe
- Ensure project setup has credentials and under API and Services are both created and enabled.
- select OAuth client ID. Choose Desktop app and click Create to proceed with a default client name.
.json file is file created after creating a credentials and accessing ID and secret
dvc add remote -d gdrive_remote gdrive://folderID
# Use the commands to set the credentials
dvc remote modify --local myremote gdrive_client_id 'client-id'
dvc remote modify --local myremote gdrive_client_secret 'client-secret'
# Have a look at your remote storage type
dvc remote list
# Check credentials
dvc config --local --list
# Then :
dvc push
- For more info on dvc remote storage setup with Google drive and other remote stoarages see the docs
churn_src
├── config.py
├── data.py
├── evaluate.py
├── models.py
├── predict.py
├── serve.py
├── train.py
├── tune.py
└── utils.py
- We'll use MLflow to track our experiments and store our models and the MLflow Tracking UI to view our experiments. We have been saving our experiments to a local directory but note that in an actual production setting, we would have a central location to store all of our experiments. It's easy/inexpensive to spin up your own MLflow server for all of your team members to track their experiments on
def promote_model_to_production(
client: MlflowClient, run_id: str, model_path: str, model_name: str, alias: str
):
"""Promote a model to production by registering it and setting an alias."""
model_uri = f"runs:/{run_id}/{model_path}"
with mlflow.start_run(run_id=run_id): # start_run, not the nonexistent mlflow.run
model_version = mlflow.register_model(model_uri=model_uri, name=model_name)
# model_info.registered_model_version gives the REAL version, not a hardcoded "1"
client.set_registered_model_alias(
name=model_name, alias=alias, version=model_version.version
)
mlflow.end_run()
model = mlflow.sklearn.load_model("models:/production_tree_model@production")
$env:MODEL = 'artifacts\prod_model\model.pkl'
$env:PROCESSOR = 'artifacts/processed/preprocessor.pkl'
$env:DATASET = "C:\Data\messy_churn_10000.csv"
python -m pytest --dataset-loc="$env:DATASET" tests/data --verbose --disable-warnings
python churn_src\predict.py $env:PROCESSOR $env:DATASET
-
Production environments are anything but smooth because code that worked in development environment no longer work in production but thanks to conayiner we can be able to simulate the same environment to ensure our codes works as expected.
-
The workflow that we shall implement is like so:
-
- We shall be deploying this model locally as a containerised job.
/app/
├── churn_src/
│ └── predict.py
│
├── artifacts/
│ ├── model.pkl
│ └── preprocessor.pkl
│
├── data/
│ └── incoming.csv
│
└── predictions_output/
- The Dockerfile provides blue print of how we want the structure of our project file to look like in the containerised environment
docker build -t Churn . # The first command looks at the blue print from the dockerfile to build our custom image
docker run --rm `
--mount "type=bind,source=$($env:DATASET),target=/app/incoming.csv,readonly" `
--mount "type=bind,source=$($PWD.Path)/predictions_output,target=/app/predictions_output" `
first `
python churn_src/predict.py `
/app/artifacts/processed/preprocessor.pkl `
/app/incoming.csv `
/app/artifacts/prod_model/model.pkl
- The mounts represent the data and mounted volume that we are going to use to predict and store our predicted results respectively.Mount is used when your container requires a particular file but was not build alongside your container.For me I considered it computationally efficient to put them separately.
-
We're not going to manually deploy our application every time we make a change. Instead, we'll automate this process using GitHub Actions!
-
In short, Docker + CI/CD = streamlined DevOps magic. ✨
-
For detailed reasons behind the design and considerations , please refer to the github/workflows.
-
With our CI/CD workflow in place to deploy our application, we can now focus on continually improving our model. It becomes really easy to extend on this foundation to connect to scheduled runs (cron), data pipelines, drift detected through monitoring, online evaluation, etc. And we can easily add additional context such as comparing any experiment with what's currently in production (directly in the PR even), We shall apply all these techniques using nanny ml, a popular open source tool for monitoring ML models to monitor our batch performance models etc.
-
In our CI/CD use case we can push iamges to AWS , Github contaiiner registry , and Docker Hub, but for our case we shall stick to hub
-
Encountering problems along the way such as why is precision so consistently across three different models but recall consistently is above 98% some models even hitting 100% , made me ask is this class imbalance or is my model prioritising recall because in this churn scenario we would want our model to predict those who were most likely to churn to help prioritise business numbers don't want to risk losing customers.
-
Nested cross-validation (CV) is often used to train a model in which hyperparameters also need to be optimized . Nested CV estimates the generalization error of the underlying model and its (hyper)parameter search. Choosing the parameters that maximize non-nested CV biases the model to the dataset, yielding an overly-optimistic score.
-
Model selection without nested CV uses the same data to tune model parameters and evaluate model performance. Information may thus “leak” into the model and overfit the data. The magnitude of this effect is primarily dependent on the size of the dataset and the stability of the model.
-
Do I use Random Search CV with cross_validate ? MY ANSWER : When implementing Nested CV , Nested == LOOP BASED , Nested CV estimates the generalization error of the underlying model and its (hyper)parameter search. Choosing the parameters that maximize non-nested CV biases the model to the dataset, yielding an overly-optimistic score.
-
One consideration I wanted to have was to mimick how real world data is processed into Github Actions, so for deployment we opted for a self hosted runner due to the location storage of our incoming data, if our data was in a remote storage like S3, R2, GCS , we would have used a virtual machine for our Github actions runner.I did encounter in Made-with-ml , the datasets were stored in Github but for standard professional practice to uplod to remote storage or otehrwise.GitHub Actions tells your Windows runner to retrieve an approved image, while the runner supplies local incoming data and stores predictions locally. No rebuilding and no uploading your dataset into GitHub is required.
