Series 2 Exporting LSTM Gender Classification and Serving With Tensorflowserving
October 1, 2020
Tensorflow
Text Classification
NLP
LSTM
So this is the second part of the series, in the previous part we successfully train our model and test the model directly from trained model instance. in this part we will export the model and serve the model with tensorflow serving so it can be accessed with REST api or gRPC endpoints, in short you can serve your model like in production, so others can performing request to ask a gender prediction to your model.
Exporting Trained Model for Tensorflow Serving
In this section, we will use (again) docker to handle our tensorflow serving instance and serving model, but first we need to look back to our previous training notebook, we need to save our character dictionary for later prediction use. this is use to encode our sequence input to model prediction later,
print(vocab_index)
#>> {'b': 1, 'w': 2, 'i': 3, '.': 4, 'd': 5, 'r': 6, 's': 7, 'h': 8, 'm': 9, "'": 10, 'f': 11, 'q': 12, 'g': 13, 'c': 14, 'l': 15, 'y': 16, 'u': 17, 'p': 18, ' ': 19, 't': 20, 'n': 21, 'k': 22, 'v': 23, 'o': 24, 'z': 25, 'e': 26, 'j': 27, 'a': 28}
# save to json file for later use
import json
with open('char_dictionary.json', 'w') as f:
json.dump(vocab_index, f)
okay, first we need to export our trained model for tensorflow serving, bear in mind tensorflow serving using name directory as versioning system (1, 2… etc)
# first create lstm_gender directory for our export directory through your notebook
! mkdir lstm_gender
then export
import os
MODEL_DIR = "lstm_gender"
version = 1
export_path = os.path.join(MODEL_DIR, str(version))
print('export_path = {}\n'.format(export_path))
tf.keras.models.save_model(
model,
export_path,
overwrite=True,
include_optimizer=True,
save_format=None,
signatures=None,
options=None
)
print('\nSaved model:')
!ls -l {export_path}
it will produce:
export_path = lstm_gender/1
INFO:tensorflow:Assets written to: lstm_gender/1/assets
Saved model:
total 632
drwxr-xr-x 2 root root 4096 Sep 27 15:31 assets
-rw-r--r-- 1 root root 637555 Sep 27 15:31 saved_model.pb
drwxr-xr-x 2 root root 4096 Sep 27 15:31 variables
all of those files and folder under 1 directory is our exported model, you then can download to your local directory, later we will copy to our tensorflow docker.
Tensorflow Serving with Docker
first we create new TFServing directory for docker and add Dockerfile, docker-compose.yml file and models.conf like this tree
Kinaras-MBP:TFServing kinara$ tree
.
├── Dockerfile
├── docker-compose.yml
└── modeldir
├── models.conf
└── tf-gender-prediction-model
└── 1
├── assets
├── saved_model.pb
└── variables
├── variables.data-00000-of-00001
└── variables.index
first for Dockerfile you put
FROM tensorflow/serving:1.14.0-rc0
docker-compose.yml
version: '3'
services:
tfserving:
container_name: 'whale_tf_serving'
restart: 'always'
build: .
volumes:
- ./modeldir:/models # set shared volume here
ports:
- 8501:8501
- 9000:9000
command: ["tensorflow_model_server", "--model_config_file=/models/models.conf"] # we set model config file here in /models/models.conf
models.conf
model_config_list: {
config: {
name: "gender_model", # this we defined our model name
base_path: "/models/tf-gender-prediction-model", # this we place our exported model directory
model_platform: "tensorflow"
}
}
as you can see from above directory tree, you put your exported model under TFServing/modeldir/tf-gender-prediction-model/ directory
thats it, to run the docker container, we can use docker-compose up command
docker-compose up
if this console output already appear then your model are ready to rock and roll
whale_tf_serving | 2020-10-01 07:11:46.623913: I tensorflow_serving/model_servers/server.cc:324] Running gRPC ModelServer at 0.0.0.0:8500 ...
whale_tf_serving | [warn] getaddrinfo: address family for nodename not supported
whale_tf_serving | [evhttp_server.cc : 239] RAW: Entering the event loop ...
whale_tf_serving | 2020-10-01 07:11:46.627376: I tensorflow_serving/model_servers/server.cc:344] Exporting HTTP/REST API at:localhost:8501 ...
Testing and Making Query Prediction
to make a query prediction, we could use REST api method, just like any other API call, but first we need to prepare our payload, encode our input with dictionary we exported before.
import json
import requests
import numpy as np
from tensorflow.keras.preprocessing.sequence import pad_sequences
# open character dictionary from trained model before
with open("char_dictionary.json", "r") as f:
vocab_index = json.loads(f.read())
name = "Susi Susanti"
name = list(name.lower())
test_dt = [vocab_index[x] for x in name]
test_dt = pad_sequences([test_dt], maxlen=32)
payload = {
"signature_name": "serving_default",
"instances": test_dt.tolist()
}
headers = {"content-type": "application/json"}
resp = requests.post('http://<your local machine ip address>:8501/v1/models/gender_model:predict', data=json.dumps(payload), headers=headers)
prediction = json.loads(resp.text)['predictions'][0]
if np.argmax(prediction) == 0:
print('Female with confidence score: {:.2f}%'.format(prediction[0] * 100))
elif np.argmax(prediction) == 1:
print('Male with confidence score: {:.2f}%'.format(prediction[1] * 100))
## sample output
## name: "Susi Susanti"
## Female with confidence score: 82.21%
## name: "Sudjiwo Tedjo"
## Male with confidence score: 99.93%
Congratulation, you just have had your model deployed into production level and ready to handle queries from other applications or users.
In summary, we can deploy our deep learning model trained with tensorflow model with Tensorflow Serving, Tensorflow Serving provide best feature for serving model with efficient performance, multi versions and also support multi models serving. Other applications can access our model with REST API call or gRCP for asking prediction.
This is the end of the second part of our series about developing and deploying deep learning model the repo also had been updated https://github.com/yudanta/lstm-gender-classification, if you haven’t read the first post about how to train the model you can visit this page: https://yudanta.github.io/posts/series-1-lstm-gender-classification-tensorflow/
for the next post, will be add an alternative ways to serving our deep learning model with Flask, Thank you… stay tune…