How to make a Flask Web App for Keras Model?

So, you have built some kind of machine learning model in Keras. Now, you want to add a frontend interface so people can actually interact with your model in an approachable way. Look no further.

Introduction

So, you have built some kind of machine learning model in Keras. Now, you want to add a frontend interface so people can actually interact with your model in an approachable way. Look no further. In this tutorial, I will cover how to package your Keras model into a powerful web application using Flask.

This is the second article in a series talking about how I made and deployed a neural network to rate songs like the popular music reviewer Anthony Fantano. You can read about the process of researching and developing the neural network in the first article. The final article covers the issues I encountered deploying a Flask Web Application to Heroku.

Click here to skip to a demo of what we will be building in this article. Click here to see all my code.

Article Overview

Cloning the Starter Code

In researching this topic myself, I stumbled upon this Github repository which promised to help “Deploy Keras Model with Flask as Web App in 10 Minutes.” While it took me more than 10 minutes to get to the finished product I wanted, this repository helped tremendously with speeding up the process. I recommend you all use it as a starting point if your web app only requires simple I/O from the user.

So, let’s go ahead and clone the starter project and check out how the code runs out of the box.

# 1. First, clone the repo 
$ git clone https://github.com/mtobeiyf/keras-flask-deploy-webapp.git 
$ cd keras-flask-deploy-webapp 

# 2. Install Python packages 
$ pip install -r requirements.txt 

# 3. Run! 
$ python app.py

Now that you have the Flask app running, you should be able to navigate to http://localhost:5000 and see the web app. Below is a demo of the built in functionality.

As you can see, the starter project includes basically all the deliverables for a flask web application using a keras model. If you have experience with web development and python, this may be enough for you to go off on your own and start building.

Now, I will work through my process of converting the starter code to meet my requirements.

Mocking Up The Design

I always like to start with a quick mockup before I dive into front end design. I wanted to fit the aesthetic that Anthony Fantano uses in theneedledrop. By using the same colors and fonts that Fantano uses in his videos, I settled on a mockup I was happy with.

Here is my quick mockup that I made in photoshop. I used this as a reference while I was developing the front end.

Next, I moved on to converting this mockup into HTML and CSS.

Writing the HTML and CSS

I will walk through my code file by file for this section.

base.html

This file simply declares the meta information for your web app, including the title, google search image, and search description. It also connects the html to our css and javascript files. Additionally, I added the custom font I will be using for the project, League Gothic.

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <title>The Needle Bot: How would Anthony Fantano rate your songs?</title>
    <meta name="description" content="Using Artificial Intelligence, this bot will try to rate your songs just like Anthony Fantano of theneedledrop."/>
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <meta property="og:title" content="The Needle Bot: Rate Your Songs" />
    <meta property="og:image" content="https://analyticsarora.com/wp-content/uploads/2021/06/the-needle-bot.jpeg" />
    <link rel="stylesheet" type="text/css" href="{{ url_for('static',filename='main.css') }}" />
    <link rel="stylesheet" href="https://use.typekit.net/uys8pat.css">
    <script type="text/javascript" src="https://code.jquery.com/jquery-1.7.1.min.js"></script>
    <script type="text/javascript" src="https://unpkg.com/meyda@<VERSION>/dist/web/meyda.min.js"></script>
  </head>

  <body>
    {% block content %}{% endblock %}
  </body>

  <footer>
    <script src="{{ url_for('static',filename='main.js') }}"></script>
  </footer>
</html>

index.html

The index file extends base.html and the code written here appears between the <body> tags in the base file. Specifically, the line {% block content %}{% endblock %} specified where the code from index.html will populate.

{% extends "base.html" %} {% block content %}

<div class="main">
<!-- Title Row with Image and App Name -->
  <div class="title">
    <img src="{{url_for('static', filename='melon_head.jpeg')}}" align="middle" class="melonHead"/>
    <h3 class="titleText">The Needle Bot</h3>
  </div>
<!-- -------- -->
<!-- File Upload Input Box -->
  <div class="panel">
    <input id="file-upload" class="hidden" type="file" accept="audio/*" name="audio_file"/>
    <label for="file-upload" id="file-drag" class="upload-box">
      <div id="upload-caption">Drop your song here or click to select (.wav)</div>
      <p id="image-preview" class="hidden" />
    </label>
  </div>
<!-- -------- -->
<!-- Submit & Clear Buttons -->
  <div style="margin-bottom: 2rem;">
    <input type="button" value="Submit" class="button" onclick="submitImage();" />
    <input type="button" value="Clear" class="button" onclick="clearImage();" />
  </div>
<!-- -------- -->
<!-- Instructions and Loading Icon -->
  <div class="text-column">
    <h3 class="subText1" id='willhetext'>Will he love it? Will he hate it?<br>What will he rate it?</h3>
    <h3 class="subText2 hidden" id="was-success">File Uploaded Successfully! Click submit to see results.</h3>
    <div class="lds-ripple hidden" id='spinner'><div></div><div></div></div>
  </div>
<!-- -------- -->
<!-- Model Results (Hidden at first) -->
  <div id="image-box">
    <h3 class="subText3 hidden" id="forthissong">For this song, I'm feeling a</h3>
    <img id="image-display" />
    <div id="pred-result" class="hidden"></div>
    <h3 class="subText2 hidden" id="justopinion">Y'all know this is just my opinion, right?</h3>
    <h4 class="subText4"><a href="http://google.com" target="_blank" rel="noopener noreferrer">(See how Melon Bot was made <u>here</u>)</a></h4>
  </div>
<!-- -------- -->
</div>

{% endblock %}

For now, I’ll leave the html up to your own interpretation. I go into more detail about the file upload code and showing/hiding elements later in the article.

My CSS is pretty long and also self explanatory, so the code is omitted in this article. You can see all of my code in this repo.

Let’s move on to specifics of including a keras model in a flask application.

Creating the Model Operation Functions

Saving the Best Model

Before we can think about putting our model into the flask web app, we have to save it. You can save your keras model as either a single .h5 file or as a directory.

To save your model as a .h5 file do the following:

model.save("my_h5_model.h5")

To save your model as a directory, simply remove the file extension as shown:

model.save("my_h5_model")

Now, with the model saved we can move back to the flask project. I opted to contain my model, the relevant data preprocessing functions, and the predict function in a separate file from app.py . This is not necessary but it became helpful when it came time to deploy the web app to the cloud.

modeloperations.py

All the code I show here comes from the file modeloperations.py . I will explain it piece by piece. First, let’s get the imports out of the way.

import librosa

from sklearn.preprocessing import LabelEncoder

# TensorFlow and tf.keras
import tensorflow as tf
from tensorflow import keras

from tensorflow.keras.models import load_model
import numpy as np

import os
import random

from base64 import b64decode

After importing, it’s time to instantiate the model and the tensorflow session. If you used a LabelEncoder to decode the output of the model.predict() function, this is also the place to instantiate that.

config = tf.ConfigProto(
                intra_op_parallelism_threads=1,
                allow_soft_placement=True
            )
session = tf.Session(config=config)

keras.backend.set_session(session)

model = load_model("best_fantano.h5", compile=False)
model._make_predict_function()
print('Model loaded. Start serving...')

le = LabelEncoder()
le.classes_ = np.load('static/classes.npy')

Data Preprocessing Function

Next, I wrote the function to preprocess the user inputted file and extract the necessary datapoints for my model. I am doing audio feature analysis so I need to load in the file from the tmp directory, extract the features, and finally delete the temporary file.

def extract_features_and_predict(path):
    # Construct relative path
    filepath = './tmp/' + path + '.wav'
    
    # Extract Audio Features
    x, sr = librosa.load(filepath)

    rmse = librosa.feature.rms(y=x)
    chroma_stft = librosa.feature.chroma_stft(y=x, sr=sr)
    spec_cent = librosa.feature.spectral_centroid(y=x, sr=sr)
    spec_bw = librosa.feature.spectral_bandwidth(y=x, sr=sr)
    rolloff = librosa.feature.spectral_rolloff(y=x, sr=sr)
    zcr = librosa.feature.zero_crossing_rate(x)
    mfcc = librosa.feature.mfcc(y=x, sr=sr)

    to_append = f'{np.mean(chroma_stft)} {np.mean(rmse)} {np.mean(spec_cent)} {np.mean(spec_bw)} {np.mean(rolloff)} {np.mean(zcr)}'    
    for e in mfcc:
        to_append += f' {np.mean(e)}'
    
    # Convert string to array of floats
    vect = to_append.split()

    for i in range(len(vect)):
        vect[i] = float(vect[i])

    # Make prediction
    preds = model_predict(vect, model)

    # Remove temporary file
    os.remove(filepath)

    # Return JSON formatted Predictions
    responseObject = {
        "result": preds.tolist(),
        "errors": 'none'
    }
    
    return responseObject

Prediction Function

Once I get the features, I call my prediction function, model_predict(vect, model), which is shown below:

def model_predict(vector, model):

    try:
        with session.as_default():
                with session.graph.as_default():
                    preds = model.predict(np.array([vector]))

                    return le.inverse_transform([np.argmax(preds)])
    
    except Exception as ex:
        print(ex)

    return [2.0]

This may look verbose but the try except block is important here. Flask uses multiple threads. So, if you do not explicitly invoke the session and the session graph, your keras model will be loaded and used on different threads. If you encounter the error “Tensor is not an element of this graph“, this is likely your issue.

My model outputs a number between 1 and 10, so I default to 2 in the event of any error. You could implement better error handling that actually updates the front end.

That’s all for our modeloperations file!

Taking in User Input

Getting input from the user is basically essential to any web app with a machine learning model. The workflow that most follow is:

  1. Accept user input
  2. Send it to the backend via a post request
  3. Save the file in some temporary directory / an s3 bucket
  4. Do the actual analysis and return the results

Let’s start with accepting user input. The way to do this is using a single or multiple <input> tags and specifying the datatypes each accepts. You can also set a placeholder value to tell people what goes in each field.

<input id="file-upload" class="hidden" type="file" accept="audio/*" name="audio_file"/>

This is my input that accepts all types of audio files, including mp3 and wav. You can see that the type is set to file and I assigned the name ‘audio_file’. This name element will be important for sending the data via post if you’re sending files.

Accepting File Drag Events

Letting the user drag and drop files really makes the web app feel more polished. This can be accomplished easily with the following code. First, the HTML elements are shown:

<div class="panel">
    <input id="file-upload" class="hidden" type="file" accept="audio/*" name="audio_file"/>
    <label for="file-upload" id="file-drag" class="upload-box">
      <div id="upload-caption">Drop your song here or click to select</div>
      <p id="image-preview" class="hidden" />
    </label>
  </div>

Now, let’s take a look at the javascript code that powers file dragging:

//======================================================================
// Drag and drop image handling
//======================================================================

var fileDrag = document.getElementById("file-drag");
var fileSelect = document.getElementById("file-upload");

// Add event listeners
fileDrag.addEventListener("dragover", fileDragHover, false);
fileDrag.addEventListener("dragleave", fileDragHover, false);
fileDrag.addEventListener("drop", fileSelectHandler, false);
fileSelect.addEventListener("change", fileSelectHandler, false);


function fileDragHover(e) {
  // prevent default behaviour
  e.preventDefault();
  e.stopPropagation();

  fileDrag.className = e.type === "dragover" ? "upload-box dragover" : "upload-box";
}

function fileSelectHandler(e) {
  // handle file selecting
  var files = e.target.files || e.dataTransfer.files;
  fileDragHover(e);
  for (var i = 0, f; (f = files[i]); i++) {
    previewFile(f);
  }
}

When a file drag and drop event occurs, the previewFile(f) function is triggered, which is shown below.

function previewFile(file) {
  // save a reference to the file and update the frontend

  globFile = file;
  var fileName = encodeURI(file.name);
  imagePreview.innerText = fileName;

  show(imagePreview);
  hide(uploadCaption);
  show(fileSuccess)
  hide(willhetext)

}

In this case, I save a reference to the file object to be used in another function whenever the user hits “Submit”. I also update the front end to display the name of the file uploaded. I cover showing and hiding elements here.

Sending Files through a Post Request

Once we have gotten the file uploaded on the frontend, it is time to send it via POST to the backend. 

function predictImage(file) {

  var form = new FormData();
  form.append('audio_file', file, "poopy.wav")

  // "Content-Type": "multipart/form-data"

  fetch("/predict", {
    method: "POST",
    headers: {
    },
    body: form
  })
    .then(resp => {
      if (resp.ok)
        resp.json().then(data => {

          displayResult(data);
    
        });
    })
    .catch(err => {
      console.log("An error occured", err.message);
      window.alert("Oops! Something went wrong.");
    });
}

In order to send the file, I create a form data object to hold the file as a key value pair. I added the file to the form under the key ‘audio_file’. This is the same as the name that I set in the input field. I also assign a filename as the third parameter, in this case poopy.wav . This third parameter is required otherwise you won’t see the file in your endpoint code.

An important note is that I had to remove all the headers in order for my file to actually appear in my flask endpoint code. So, do not include the Content-Type header if you have multipart form data. That sounds counterintuitive so here is a quote that I thought was quite good:

you can absolutely send files via POST without using multipart/form-data. What you can’t do is do that using an ordinary HTML form submission, without JavaScript. Setting a form to use multipart/form-data is the only mechanism that HTML provides to let you POST files without using JavaScript.

Mark Amery

Saving Files to Temporary Directory

The post request gets sent off to the /predict route in the flask code. Let’s take a look at how I receive the request, extract the file, and save it to the temporary directory.

@app.route('/predict', methods=['GET', 'POST'])

def predict():

    if request.method == 'POST':

        file = request.files['audio_file']

        random_number = random.randint(00000, 99999)

        filepath = './tmp/' + str(random_number) + '.wav'

        file.save(filepath)
        filename = str(random_number) + '.wav'

        res = extract_features_and_predict(filename)

        # create a dictionary with the ID of the task

        responseObject = {"status": "success", "data": res}

        # return the dictionary

        return jsonify(responseObject)

    return None

I grab the file from request.files using the key that I set in the form data object. I then save the file to the temporary directory and pass on the file path to the data extraction / prediction function. Once the model finishes its prediction, I return the results in a JSON format to my javascript code.

Updating the Frontend with Javascript

Now, let’s dig in to how I am using Javascript to show and hide elements. Then, I will give you a full overview of what happens when the user clicks the Submit and Clear buttons.

Showing and Hiding Elements

Showing and hiding specific HTML elements is made easy with two utility functions:

function hide(el) {
  // hide an element
  el.classList.add("hidden");
}

function show(el) {
  // show an element
  el.classList.remove("hidden");
}

Calling these functions adds and removes the hidden class from a given element. The hidden class is shown below:

.hidden {
  display: none;
}

Submit Function

When the user hits the submit button, the following functions are executed in the following order. First, the submit function is called:

function submitImage() {
  // action for the submit button
  console.log("submit");

  // call the predict function of the backend
  

  if (globFile !== null) {
    if (globFile.type.split('/')[0] == 'audio') {
      hide(fileSuccess)
      show(spinner)
      predictImage(globFile);
    } else {
      window.alert("Invalid File Type. Please submit an audio file");
    }
  } else {
    window.alert("Please upload a .wav file before submitting");
  }

}

The submit function performs some basic input validation, hides the label telling them their file has been uploaded, and shows them a loading icon. Then, a call is made to the predictImage(file) function.

function predictImage(file) {

  var form = new FormData();
  form.append('audio_file', file, "poopy.wav")

  // "Content-Type": "multipart/form-data"

  fetch("/predict", {
    method: "POST",
    headers: {
    },
    body: form
  })
    .then(resp => {
      if (resp.ok)
        resp.json().then(data => {

          displayResult(data);
    
        });
    })
    .catch(err => {
      console.log("An error occured", err.message);
      window.alert("Oops! Something went wrong.");
    });
}

Here, we package the file into the form data as I previously explained. The endpoint will return an array with a single number in it. This is passed off to the displayResult(data) function which shows the user the model output.

function displayResult(data, errors) {

  hide(spinner)

  imageDisplay.src = './static/' + String(data[0]) + '.png'

  if (imageDisplay.classList.contains('hidden')) {
    show(imageDisplay)
  }
  
  show(justopinion)
  show(forthissong)
  
}

The loading icon is hidden. The imageDisplay is updated to show the corresponding image for the model output, and two <p> tags are shown to explain the results. That’s it for submit!

Clear Function

The clear function resets the state of our web app back to how it looks on page load. For me, this just means showing and hiding certain elements and resetting my global file.

function clearImage() {
  // reset selected files
  fileSelect.value = "";

  // remove image sources and hide them
  imagePreview.src = "";
  imageDisplay.src = "";
  predResult.innerHTML = "";

  globFile = null

  hide(imagePreview);
  hide(imageDisplay);
  hide(predResult);
  hide(justopinion)
  hide(forthissong)

  show(uploadCaption);
  show(willhetext)
  
  imageDisplay.classList.remove("loading");
}

Great! Now you should have a good understanding of how I made my flask web app for keras from start to end.

Demo and Conclusion

Let’s see the finished product of all that code! Below is a demo of my web app working.

Hopefully this article has given you some clarity on the process of creating a flask web app for keras machine learning models. Once you have created your own web application, you likely want to deploy it to the cloud so that other people can interact with it. Check out the final tutorial in this series where I detail the common issues you may encounter deploying to Heroku .

Related Posts:


Avi Arora
Avi Arora

Avi is a Computer Science student at the Georgia Institute of Technology pursuing a Masters in Machine Learning. He is a software engineer working at Capital One, and the Co Founder of the company Octtone. His company creates software products in the Health & Wellness space.

2 Comments

  2. Hello very nice web site!! Guy .. Excellent .. Superb ..
    I will bookmark your website and take the feeds also?

    I am satisfied to seek out a lot of helpful info here in the publish, we
    want develop more strategies in this regard, thank
    you for sharing. . . . . .

Comments are closed.