MLOps with Comet - A Machine Learning Platform
Experimenting with Comet, a machine learning platform
Creating a machine learning model is easy, but that’s not what machine learning is all about. Data scientists need to track and compare the performance of different machine learning models while using different hyperparameters.
When working on an ML project, we must compare various machine learning models with different hyperparameters and want to understand which models and hyperparameters are most effective for our use case.
There are machine learning platforms that can perform all these tasks, and Comet is one such platform.
Comet
Comet is a machine learning platform built to help data scientists and ML engineers track, compare, and optimize machine learning experiments. It is beneficial for organizing and managing experiments and analyzing and visualizing the results of those data science experiments.
We can use Comet to track metrics such as precision, recall, and accuracy, as well as compare the performance of different models on the same dataset. It is handy when you are working on a machine learning project and want to understand which models and hyperparameters are most effective for your model.
Let's see how we can track our deep learning experiment and visualize the metrics with Comet:
Experimenting with Comet
About the Dataset
In our practical use case, we will use the mnist dataset as an example. This dataset consists of 60,000 training grayscale images of handwritten digits between 0 to 9.
Library Installation
First, we need to install the comet_ml library. We can install this library in the same way as other Python libraries, using pip .
#Installing the comet ML library using pip
pip install comet_mlWe can also install this library in a Google Colab and Notebook using this command:
#Installing the Comet ML library in Google Colab Notebook
%pip install comet_mlCreating a Comet Experiment
Now, we need to set up a Comet experiment. We can do it by specifying the API key obtained from Comet and the project name.
We can track all our model's metrics, performance, and other details in our Comet experiment.
# importing the experiment module
from comet_ml import Experiment
# Creating an experiment by specifying the api key and project name
experiment = Experiment(
api_key="Enter_API_Key",
project_name="Enter_Project_Name",
)Loading the Dataset
We will use mnist dataset by importing it from the Keras library. In Keras, we use load_data() method to load the dataset.
import keras
import numpy as np
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.callbacks import EarlyStopping
from keras.preprocessing import image
from keras.optimizers import RMSprop
(x_train, y_train), (x_test, y_test) = mnist.load_data()
print(x_train.shape)
print(x_train)Here, we split our dataset into training set(x_train, y_train) and test set(x_test, y_test)
Output:
We can see that our training data contains 60,000 records.
Preprocessing the Dataset
For preprocessing the dataset, we are using reshaping and normalization techniques to preprocess our dataset. We rescale the images from [0, 255] to the [0.0, 1.0] range.
# preprocess and normalize the data
x_train = x_train.reshape(60000, 784)
x_test = x_test.reshape(10000, 784)
x_train = x_train.astype("float32")
x_test = x_test.astype("float32")
x_train /= 255
x_test /= 255
print(x_train.shape[0], "train samples")
print(x_test.shape[0], "test samples")
Now, we convert class vectors to binary class matrices.
y_train = keras.utils.to_categorical(y_train, 10)
y_test = keras.utils.to_categorical(y_test, 10)Building the Model
The dataset has been preprocessed, and now we will define all the parameters we want to use in our deep learning model. We will build our deep learning model using those parameters.
# Defining the parameters used in hypertuning the model
batch_size = 128
num_classes = 10
epochs = 20
num_nodes = 64
optimizer = 'adam'
activation = 'relu'
#Building the model
def build_model_graph(input_shape=(784,)):
model = Sequential()
model.add(Dense(num_nodes, activation='relu', input_shape=(784,)))
model.add(Dense(256, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy',
optimizer=optimizer,
metrics=['accuracy'])
return model
model = build_model_graph()
model.summary()
Visualization
Once the model has been created, we will display the results in our notebook. We use display() function in the experiment class to show the results of our model in the notebook only.
experiment.display()
Same as charts, we can log the performance of our metrics in the same experiments. To log the metrics, we can use the log_metrics function in the comet_ml package to log any metric, such as the loss during the model training or the accuracy of our deep-learning model.
with experiment.train():
history = model.fit(x_train, y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(x_test, y_test),
callbacks=[EarlyStopping(monitor='val_loss', min_delta=1e-4,patience=3, verbose=1, mode='auto')])
# Logging the metrics
with experiment.test():
loss, accuracy = model.evaluate(x_test, y_test)
metrics = {
'loss':loss,
'accuracy':accuracy
}
experiment.log_metrics(metrics)Here, we can see our logged metrics and parameters. In the left panel, we can see other information, such as our hyperparameters, the code we use, and much more.
If you want to end the experiment, you can use the end method of the Experiment object to mark the experiment as complete.
# End the experiment
experiment.end()Ending an experiment this way will make Comet stop logging information about the experiment and mark it as complete on the platform.
This is how we can track all our experiments with the comet_ml library. We visualized all the results in our notebook, the same way we can track our model and view the logged information for the experiment using the Comet web interface or API.
Comet Web Interface
In the Comet web interface, we can view the experiment history, the details of each experiment and compare the results of different experiments.
We can also view the code and logs associated with each experiment and any metrics and plots generated during the experiment.
Conclusion
That's all from this article. We learned how to create experiments in the Comet platform, track the performance of our machine learning model, and view the logs on the interface.
Thanks for reading!
Editor's Note: Heartbeat is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We're committed to supporting and inspiring developers and engineers from all walks of life.
Editorially independent, Heartbeat is sponsored and published by Comet, an MLOps platform that enables data scientists & ML teams to track, compare, explain, & optimize their experiments. We pay our contributors, and we don't sell ads.
If you'd like to contribute, head on over to our call for contributors. You can also sign up to receive our weekly newsletter (Deep Learning Weekly), check out the Comet blog, join us on Slack, and follow Comet on Twitter and LinkedIn for resources, events, and much more that will help you build better ML models, faster.
