A very useful functionality was added to OpenCV’s DNN module: a Tensorflow net importer.
To use the DNN, the opencv_contrib is needed, make sure to install it. This article is focused on the Python language, where the function has the following format:
1 | cv2.dnn.readNetFromTensorflow('frozen_inference_graph.pb', 'graph.pbtxt') |
As you might have seen, to use it, two files are needed:
- frozen_inference_graph.pb
- graph.pbtxt
About Tensorflow’s .pb and .pbtxt files
Tensorflow models usually have a fairly high number of parameters. Freezing is the process to identify and save just the required ones (graph, weights, etc) into a single file that you can use later. So, in other words, it’s the TF way to “export” your model. The freezing process produces a Protobuf ( .pb) file.
The good news is: There are a bunch of trained, optimized and widely used models on the Tensorflow’s detection model zoo repository that you can use freely. You won’t need to train one (if the available models, trained with well know datasets, fit your needs).
Additionally, OpenCV requires an extra configuration file based on the .pb, the .pbtxt. It is possible to import your own models and generate your own .pbtxt files by using one of the following files from the OpenCV Github repository.
If you want to use pre-trained models, the amazing OpenCV community already did the hard work for the following models.
| Model | Version | ||
|---|---|---|---|
| MobileNet-SSD v1 | 2017_11_17 | weights | config |
| MobileNet-SSD v1 PPN | 2018_07_03 | weights | config |
| MobileNet-SSD v2 | 2018_03_29 | weights | config |
| Inception-SSD v2 | 2017_11_17 | weights | config |
| Faster-RCNN Inception v2 | 2018_01_28 | weights | config |
| Faster-RCNN ResNet-50 | 2018_01_28 | weights | config |
| Mask-RCNN Inception v2 | 2018_01_28 | weights | config |
Font: OpenCV’s Github wiki.
The Python code
The entire import/use procedure could be split into 5 steps:
- Load your model using the downloaded files;
- Load your images;
- Use those images as network inputs;
- Get the output with the detected objects.
Take a look at the Python code snippet for doing these steps:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 | # How to load a Tensorflow model using OpenCV # Jean Vitor de Paulo Blog - https://jeanvitor.com/tensorflow-object-detecion-opencv/ import cv2 # Load a model imported from Tensorflow tensorflowNet = cv2.dnn.readNetFromTensorflow('frozen_inference_graph.pb', 'graph.pbtxt') # Input image img = cv2.imread('img.jpg') rows, cols, channels = img.shape # Use the given image as input, which needs to be blob(s). tensorflowNet.setInput(cv2.dnn.blobFromImage(img, size=(300, 300), swapRB=True, crop=False)) # Runs a forward pass to compute the net output networkOutput = tensorflowNet.forward() # Loop on the outputs for detection in networkOutput[0,0]: score = float(detection[2]) if score > 0.2: left = detection[3] * cols top = detection[4] * rows right = detection[5] * cols bottom = detection[6] * rows #draw a red rectangle around detected objects cv2.rectangle(img, (int(left), int(top)), (int(right), int(bottom)), (0, 0, 255), thickness=2) # Show the image with a rectagle surrounding the detected objects cv2.imshow('Image', img) cv2.waitKey() cv2.destroyAllWindows() |
Output examples
Have fun with your projects!
If this post helped you, please consider buying me a coffee 🙂
Hello,
thanks for your demo. I have a questions:
why scale factor and mean are not used in your code?
Hi,
This article is just to show how to load the models. I really tried to make it as simple as possible.
Either way, if you think that you have something to add up, please feel free to share!
Best
Hi Jean,
your demo is very helpful.
1- in opencv GitHub (https://github.com/opencv/opencv/tree/master/samples/dnn) I see scale factor and mean values for tensorflow SSDs. I tried to used them, but no object was detected, but with your settings it works. Do you know why? usually they should be employed in generating blob.
2- it is good if you also add a few lines to print label of each object on it.
Thanks!
The scale factor represents the confidence level, in simple term higher the scale higher is the threshold (that’s how much the neural network is confident about the guess it made.
would this work with tf lite models? or quantized ones
Great post but I am having this error
cv2.error: OpenCV(3.4.2) C:\Miniconda3\conda-bld\opencv-suite_1534379934306\work\modules\dnn\src\layers\permute_layer.cpp:137: error: (-215:Assertion failed) (int)_numAxes == inputs[0].size() in function ‘cv::dnn::PermuteLayerImpl::getMemoryShapes’
Hi,
Seems to be something with your network input. Could you provide more details (code or screenshot)?
hi did you find a solution for this problem
You may checkout the input path in the config file before generating the graph
Hey! How can I use these pre-trained models to detect only some pre-defined objects in real time and labeling them in the video. And If I want to run “XYZ” piece of code after the detection of “a” obejct, how can I achieve this.
Any help would be great.
Hi.
For the real time detection, I suggest you taking a look on models such as “Yolo“.
For running a XYZ code after a detection, it’s not rocket science, it would be very similar to the lines 20-32 of the last code snippet.
Good luck!
Can you explain what are the .py scripts you showed? I need to create the pbtxt from the pb file, and not sure how to use the scripts you mentioned.
Hi,
I suggest you take look at the Tensorflow’s function freeze_graph.
Hi,
I am getting segmentation fault at line 17. Any idea why?
Hi,
Please check your input images dimensions.
I did I changed it to 1080×1080 in the code as my images are of that dimensions. Even then I am getting segmentation fault.
Hello Jean,
Thank you for the tutorial. I have one question. With the rectangles around the detected objects, how can I print the class labels as well using the dnn module. The model is trained for printing the labels and I can print them using the below function:
# Draw the results of the detection (aka ‘visulaize the results’)
vis_util.visualize_boxes_and_labels_on_image_array(
image,
np.squeeze(boxes),
np.squeeze(classes).astype(np.int32),
np.squeeze(scores),
category_index,
use_normalized_coordinates=True,
line_thickness=8,
min_score_thresh=0.60)
How can I do the same thing in your code.
Thank you
dear Jean,
thank you for all the valuable information. i am using your files in order to create a .pbtxt file from a .pb file that i have, but where do we get the *.config file from, required in tf_text_graph_ssd.py ? thanks
Hi.
There are links to some of them on the article.
Either way, you can find more here https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md
thank you
hi Jean,
how can I use my Tensorflow model for tracking process like SSD model? If you have any source code or suggestion please replay
Hi.
Do you wanna train your own SSD model?
yes i want to train my own model
Hi,
I have a query regarding the OpenCV-dnn classification. the documentation says that the support caffe,TF and pytorch. But it seems that caffe is the default choice in case of classification while TF API is for obejct detection. I wanted to use TF trained squeeze-net for classification using dnn. Do you think that i can reproduce the similar results as using caffe model of squeezenet.
Hello.
I think its possible, there are specific functions for the Caffe.
Please take a look here https://docs.opencv.org/3.4/d6/d0f/group__dnn.html#ga946b342af1355185a7107640f868b64a
Good Luck!
Pingback: Tinkering with TensorFlow (and OpenCV) | Matthew Tran's Blog!
hey Jean ,
I need to implement robust and effective human detection and tracking
What is the best algo to do it
Yolo v3 ?
I have to do it in openCV
is there a need to load tensorflow framework , or it is possible without using tensorflow ?
which is faster
Hi,
The OpenCV’s DNN module is capable of doing a reasonable job. You wont need tensorflow if you just want to load and use the trained models (try Keras if you need to train the models to make things simpler).
The YOLO V3 is indeed a good solution and is pretty fast. Search also for Single Shot Object Detecion (SSD) and Faster-RCNN to see other alternatives.
Good luck.
Hi .
I am getting an error
cv2.dnn.readNetFromTensorflow
No module dnn available
Hi,
Please make sure to have opencv-contrib installed, the dnn module is in it.
What changes to make for faster RCNN
Hi Jean,
Thanks for a well-explained guide.
I’m trying to run this (https://github.com/nwojke/cosine_metric_learning) network’s protobuf file using the code snipped you have shared. However, I am getting this error,
Where the first layer of network is named as “map/while/Enter” (I get that using .getLayerNames() ).
Do you have any idea how to solve this error ?
Thanks for your help!
Hi. Glad you liked the post.
Would you mind to give more details about that error??
Thanks for your quick reply and sorry for responding late.
1) when I pass .pb and .pbtxt file in the arguments of “readNetfromTensorflow”, it fails to parse the pbtxt file with
the following error.
(OpenCV(4.1.1) /io/opencv/modules/dnn/src/tensorflow/tf_io.cpp:54: error: (-2:Unspecified error) FAILED:
ReadProtoFromTextFile(param_file, param). Failed to parse GraphDef file:
./experimented_networks/graph.pbtxt in function
‘ReadTFNetParamsFromTextFileOrDie’)
2) when I pass the .pb file only to readNetFromTensorflow, it gives me the following error.
( OpenCV(4.1.1) /io/opencv/modules/dnn/src/dnn.cpp:525: error: (-2:Unspecified error) Can’t create layer
“map/while/Enter” of type “Enter” in function ‘getLayerInstance’ )
here are the layers names that my network has:
[‘map/while/Enter’,
‘map/while/Enter_1’,
‘map/Shape’,
‘map/strided_slice’,
‘map/while/Less/Enter’,
‘map/while/Less_1’,
‘map/while/Less’,
‘map/while/LogicalAnd’,
‘map/while/LoopCond’,
‘map/TensorArray_1’,
‘map/while/Enter_2’,
‘map/while/Exit_2’,
‘map/TensorArrayStack/TensorArraySizeV3’,
‘map/TensorArrayStack/range’,
‘map/TensorArrayStack/TensorArrayGatherV3’,
‘conv1_1/Conv2D’,
‘conv1_1/conv1_1/bn/FusedBatchNorm’,
‘conv1_1/Elu’,
‘conv1_2/Conv2D’,
‘conv1_2/conv1_2/bn/FusedBatchNorm’,
‘conv1_2/Elu’,
‘conv2_1/1/Conv2D’,
‘conv2_1/1/conv2_1/1/bn/FusedBatchNorm’,
‘conv2_1/1/Elu’,
‘conv2_1/2/Conv2D’,
‘add’,
‘conv2_3/bn/FusedBatchNorm’,
‘Elu’,
‘conv2_3/1/Conv2D’,
‘conv2_3/1/conv2_3/1/bn/FusedBatchNorm’,
‘conv2_3/1/Elu’,
‘conv2_3/2/Conv2D’,
‘add_1’,
‘pool1/MaxPool’,
‘Flatten/flatten/Reshape/nchw’,
‘Flatten/flatten/Reshape’,
‘fc1/MatMul’,
‘fc1/fc1/bn/Reshape’,
‘fc1/fc1/bn/FusedBatchNorm’,
‘fc1/fc1/bn/Reshape_1’,
‘fc1/Elu’,
‘l2_normalize’]
And this network has been taken from this (https://github.com/nwojke/cosine_metric_learning) source.
here’s the link to ‘graph.pbtxt’ file:
(https://drive.google.com/file/d/1OTe5PiGy8jcO4JAypwpnqOaTE1gQu4aN/view?usp=sharing)
Hi .
I am getting an error
File “tf_text_graph_ssd.py”, line 15, in
from tf_text_graph_common import *
ModuleNotFoundError: No module named ‘tf_text_graph_common’
i installed opencv with anaconda by conda install -c conda-forge opencv .Can you help me, pls. Thanks
Hi. Which OpenCV version are you using?
hi i have an error when i try run tf_text_graph_ssd.py
File “tf_text_graph_ssd.py”, line 15, in
from tf_text_graph_common import *
ModuleNotFoundError: No module named ‘tf_text_graph_common’
i installed opencv in anaconda3 by command conda install -c conda-forge opencv
Can you help me, Thanks
Hi , is it possible to read and print the weights and biases for a model file using readNetFromTensorflow()?
Hi,
Yes it’s possible. The readNetFromTensorflow function returns a ‘Net’ type that has a lot of utility functions.
Thank you Jean. Is it possible to load a custom Tensorflow model using openCV DNN APIs? If not, is there a workaround?
Thanks for your time and effort.
Hi,
There are some Tensorflow scripts to correctly freeze your models for further use on OpenCV DNN.
Does this work with quantized models like ssd_mobilenet_v2_quantized_300x300_coco_2019_01_03?
Hi,
I think so. I’ve never tried using quantized models.
Hello Jean! Have you ever exported your custom model to .pb , .pbtxt and read it in opencv?
I have never tried custom ones. If you manage to try your own, please report back your experience!
Is there is any way to put labels on detected objects
hello, I’m getting the following error while trying to load a tensorflow model ssd mobilenet V2. Using the pretrained model in Model Zoo everything works fine, but after a fine tuning of the model I get error. Could it be the version of tensorflow used for finetuning? Currently I’m using Tf 1.15. Thanks
Can’t create layer “FeatureExtractor/MobilenetV2/Conv/BatchNorm/FusedBatchNormV3” of type “FusedBatchNormV3” in function ‘getLayerInstance’
hi all and thanks for help
when i’m using my custom object detection
tensorflowNet = cv2.dnn.readNetFromTensorflow(‘models/my_custom.pb’, ‘models/my_custom.pbtxt’)
i get error :
networkOutput = tensorflowNet.forward()
cv2.error: OpenCV(4.1.1) C:\projects\opencv-python\opencv\modules\dnn\src\dnn.cp
p:693: error: (-215:Assertion failed) inputs.size() == requiredOutputs in functi
on ‘cv::dnn::dnn4_v20190621::DataLayer::getMemoryShapes’
hey jean,i am using this code for my custom tensorflow graph of mobilenet ssd,but getting this error:
error: OpenCV(4.2.0) C:\projects\opencv-python\opencv\modules\dnn\src\dnn.cpp:730: error: (-215:Assertion failed) inputs.size() == requiredOutputs in function ‘cv::dnn::dnn4_v20191202::DataLayer::getMemoryShapes’
could you help me please.
Hi,
Please check your images (resolution, channels, format…) to match with what your network expects.
Hi Jean, may I know how to generate tensorflow .pb file to .pbtxt?
I have yolo v3 model with cfg and weights file. I already convert them into .pb file (tensorflow), but now i need .pbtxt. How to i generate the .pbtxt file.?
Thank you.
Look for your reply soon~
how we will get .pbtxt file as I have only .pb file can you please help in this…thank you
Hi Jean,
Thank you for sharing!
I am trying to use OpenCV to recognized a new object based on a trained mobilenet-ssd-v2 model, but always get an error. In Tensorflow it recognizes the new objects, but in OpenCV there is an error.
I am unable to generate .pb and .pbtxt files to use in the cv.dnn.readNetFromTensorflow call.
If you get the already made file, it works, but if you need to generate them, seems pretty impossible.
Could you point me some resource to research how to solve this?
Thank you,
Pingback: How to load Pytorch models with OpenCV • Jean Vitor
Traceback (most recent call last):
File “c:/Users/RR/Desktop/project/project1.py”, line 7, in
tensorflowNet = cv2.dnn.readNetFromTensorflow(‘frozen_inference_graph.pb’, ‘graph.pbtxt’)
cv2.error: OpenCV(4.4.0) C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-h4wtvo23\opencv\modules\dnn\src\caffe\caffe_io.cpp:1133: error: (-2:Unspecified error) FAILED: fs.is_open(). Can’t open “frozen_inference_graph.pb” in function ‘cv::dnn::ReadProtoFromBinaryFile’
what could be the reason for this error?
Anyone has come across this issue:
tensorflowNet = cv2.dnn.readNetFromTensorflow(
cv2.error: OpenCV(4.4.0) /private/var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/pip-req-build-wv7rsg8n/opencv/modules/dnn/src/tensorflow/tf_importer.cpp:672: error: (-215:Assertion failed) const_layers.insert(std::make_pair(name, li)).second in function ‘addConstNodes’
???
Please any hints?
Thanks in advance
Hi, thank you so much for the info.
I was able to find the models and config files but not the classification/labels for the models. How do I get these?
These are the models with pbtxt files:
https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md
Pingback: Sử dụng OpenCV deep learning với model Tensorflow 1