How to Deploy Fast.ai models to Google Cloud Functions for Predictions

Photo by Franck V. on Unsplash

If you are going through the Fast.ai course, one of the things you might be looking for would be an affordable yet reliable option to deploy your models online for predictions or inferences.

Google Cloud Functions together with its free tier is definitely one of the good options out there to get started. It being serverless also means that you don’t get charged for anything if you are not using the services! That’s perfect for a beginner’s use case.

So how can we deploy to Google Cloud Functions?

Let’s get started with this 7 simple steps.

Source codes and sample model are available on Github.

This tutorial is based on Fast.ai — 2019 Lesson 2 Download notebook but its source codes are intentionally kept generic enough for most image classifications models.

1. Export the Trained Model

At the end of Lesson 2 — Download notebook, there is a section that teaches you to export your model via learn.export(). This command will generate a export.pkl model file in you respective folder.

Here we’ll create a model.pkl model file by using learn.export('model.pkl') .

If you could not get the model file, I have uploaded a sample model.pkl on Github.

Fast.ai — Lesson 2

2. Upload model.pkl Online

Since GCF does not come with a persistent storage, we will have to put the model somewhere else and fetch it as needed. We have 2 choices:

1. Store it on a Cloud Storage bucket, then fetch the file privately
2. Upload it to somewhere with a publicly downloadable link

For simplicity, we’ll continue with option #2 though for best practises, you would want to go with option #1.

Note that if you put it on Cloud Storage, you can set the permission to allow public access for the particular object and generate a publicly accessible link for this tutorial.

3. Create Our Cloud Functions

Since Lesson 2 was about bears, let’s create a Cloud Function called bears. We are going to need 512 MB for memory as parts of it would be used to load our large model file model.pkl.

Trigger would be for HTTP. We want to create an endpoint that in the future our web apps could interface with.

Finally, remember to choose Python 3.7 as the runtime.

Settings for our new Cloud Function

4. Setting Up the requirements.txt

Under Runtime, you will see 2 tabs for main.py and requirements.txt. Let’s start with requirements.txt. This is where we specify the libraries we need for our Cloud Function.

We only needed 2 i.e. fastai and pytorch. For pytorch we will be pulling the libraries via direct links. Since GCF does not come with GPU, we need the custom pytorch versions that do not come with CUDA or GCF will throw errors.

fastai>=1.0
# PyTorch 1.1.0 without CUDA
https://download.pytorch.org/whl/cpu/torch-1.1.0-cp37-cp37m-linux_x86_64.whl
https://download.pytorch.org/whl/cpu/torchvision-0.3.0-cp37-cp37m-linux_x86_64.whl

requirements.txt

5. Filling Up the main.py

We only need around 24 lines of codes to make this work! Copy and paste the following codes into themain.py tab.

Then, for Function to execute field, enter handler.

import json
import urllib.request
from fastai.vision import *
def handler(request):
    defaults.device = torch.device('cpu')
    model = request.args['model']
    urllib.request.urlretrieve(model, '/tmp/model.pkl')
    path = Path('/tmp')
    learner = load_learner(path, 'model.pkl')
    
    image = request.args['image']
    urllib.request.urlretrieve(image, '/tmp/image.jpg')
    img = open_image('/tmp/image.jpg')
    pred_class,pred_idx,outputs = learner.predict(img)
    return json.dumps({
        "predictions": sorted(
            zip(learner.data.classes, map(float, outputs)),
            key=lambda p: p[1],
            reverse=True
        )
    })

main.py

We are going to leave the other stuff like Environment variables etc as default for now.

6. Deployment

Now click deploy and wait patiently has Google builds our Cloud Function, it may take a while.

While waiting, let’s grab the HTTP URL for testing later. Click into our Cloud Function bears again, select the Trigger tab, and copy the URL.

Trigger URL

7. Prediction!

When you see the green tick beside the function name bears appears, it’s now ready!

Give it a try at: https://<region-name>-<project-name>.cloudfunctions.net/bears?image=<image_url>&model=<model_url>

Hopefully everything works OK for you and you would see the right prediction scores as followed:

{"predictions": [["teddys", 0.9998297691345215], ["grizzly", 0.0001574010675540194], ["black", 1.285488178837113e-05]]}

That’s all folks! Thanks for following along.

Now you can build a web app on top of this JSON output and extend its functions anyway you like!

References:

1. Fast.ai Lesson 2 Download Notebook
2. https://medium.com/swlh/deploying-pytorch-models-to-serverless-environments-8fcd25dc7b5c
3. https://cloud.google.com/blog/products/ai-machine-learning/how-to-serve-deep-learning-models-using-tensorflow-2-0-with-cloud-functions