Thursday, November 23, 2017

OpenCV DNN speed compare in Python, C#, C++

OpenCVDnnCompare_4
Last time I tried SSD on Python + OpenCV DNN, this time I will implement same test using C# and C++ to test performance and the difficult of implement.

To be fair, I will use same model and same test image on those 3 tests, model will be "VGG_VOC0712Plus_SSD_512x512_ft_iter_160000.caffemodel" from here (07++12+COCO: SSD512), image will be one picture I shot at Bail few year ago.
bali-crop


Test environment

  • Hardware: Intel Core i7-7820HQ @ 2.90GHz (OpenCV DNN can't enable GPU so didn't list)
  • Python: Python 3.6.3, OpenCv-Python 3.3.0.10
  • C#: .NET Framework 4.7, EMGU.CV 3.3.0.2824
  • C++: OpenCv 3.3.1

SSD with Python OpenCV DNN

Detail please check my previous post.

RunTime: 1510 ms
OpenCVDnnCompare_1


SSD with C# EmguCV DNN

Because OpenCvSharp still didn't support OpenCV 3.3, so I have to change to EmguCV, it can easy install via NuGet, but EmguCV API namespace are way different with OpenCV and it can't direct access Mat's pixel, so I spent a lot of time on implement my demo code from OpenCv to EmguCV.

RunTime: 4041 ms
OpenCVDnnCompare_2

And here has a problem, using EmguCV to evaluate test image, the result will different with other framework, you can obviously see here have a wrong rectangle at bottom of image labeled as person.

I think it is Emgu CV bug, not sure come from DNN module or not, if you evaluate other image, the wrong rectangle still appear here.
OpenCVDnnCompare_5


SSD with C++ OpenCV DNN

I running C++/OpenCV on Windows, follow this introduction can easy install OpenCv to your pc, and make sure follow this guide to setup your environment and solution if you want to run C++ on Visual Studio like me.

RunTime: 9306 ms
OpenCVDnnCompare_3


Summary

OpenCVDnnCompare_6

Speed: Python > C# > C++

Yes, it was weird, but i can't figure it out the real reason, I only can guess it come from framework or not native code or something else...

Anyway, my test result was here, please left comments if you have any opinions, I really want to know why running OpenCv Dnn in C++ was 6 times slow than running in python.



Source Code

All source code already put on GuiHub/OpenCvDnnSpeedCompare.


Python code in previous post.

C++ code part from OpenCv Dnn Moudle Sample, so I didn't add much comments.
#include "stdafx.h"
#include <opencv2/opencv.hpp>
#include <opencv2/dnn.hpp>
using namespace cv;
using namespace cv::dnn;
using namespace std;

const char* classNames[] = { "background","aeroplane", "bicycle", "bird", "boat","bottle", "bus", "car", "cat", "chair","cow", "diningtable", "dog", "horse","motorbike", "person", "pottedplant","sheep", "sofa", "train", "tvmonitor" };

int main()
{
 Scalar colors[21];
 srand((unsigned)time(NULL));
 for (int i = 0; i < 21; i++) {
  colors[i] = Scalar(rand() % 256, rand() % 256, rand() % 256);
 }

 String modelTxt = "deploy.prototxt";
 String modelBin = "VGG_VOC0712Plus_SSD_512x512_ft_iter_160000.caffemodel";
 String imageFile = "bali-crop.jpg";

 Net net = dnn::readNetFromCaffe(modelTxt, modelBin);
 Mat img = cv::imread(imageFile);

 Mat inputBlob = blobFromImage(img, 1, Size(512, 512));   //Convert Mat to batch of images
 Mat prob;

 cv::TickMeter t;
 net.setInput(inputBlob, "data");        //set the network input
 t.start();
 prob = net.forward("detection_out");                          //compute output
 t.stop();
 cout << "Runtime: " << (double)t.getTimeMilli() << " ms " << endl;

 Mat detectionMat(prob.size[2], prob.size[3], CV_32F, prob.ptr());

 for (int i = 0; i < detectionMat.rows; i++)
 {
  float confidence = detectionMat.at(i, 2);
  if (confidence > 0.4)
  {
   size_t objectClass = (size_t)(detectionMat.at(i, 1));

   int xLeftBottom = static_cast(detectionMat.at(i, 3) * img.cols);
   int yLeftBottom = static_cast(detectionMat.at(i, 4) * img.rows);
   int xRightTop = static_cast(detectionMat.at(i, 5) * img.cols);
   int yRightTop = static_cast(detectionMat.at(i, 6) * img.rows);

   Rect object(xLeftBottom, yLeftBottom, xRightTop - xLeftBottom, yRightTop - yLeftBottom);

   rectangle(img, object, colors[objectClass],2);
   String label = String(classNames[objectClass]) + ": " + to_string(confidence);
   std::cout << label << endl;

   int baseLine = 0;
   Size labelSize = getTextSize(label, FONT_HERSHEY_TRIPLEX, 0.5, 1, &baseLine);
   rectangle(img, Rect(Point(xLeftBottom, yLeftBottom - labelSize.height),
    Size(labelSize.width, labelSize.height + baseLine)), colors[objectClass], CV_FILLED);
   putText(img, label, Point(xLeftBottom, yLeftBottom), FONT_HERSHEY_TRIPLEX, 0.5, Scalar(0,0,0));
  }
 }

 cv::imshow("image", img);
 cv::waitKey(0);
 return 0;
}

C# code was here, seems no one using Emgu CV Dnn with SSD(at least I didn't see it on google search), so this sample may help you if you want to use OpenCV DNN on C# (until OpenCvSharp support 3.3).

But be honest, I don't recommend Emgu CV, as a wrapper of C# over openCV, Emgu CV has part different with OpenCV, so most of time you can't using OpenCV resource in troubleshooting since Emgu CV got fewer user and fewer support.

using System;
using System.Diagnostics;
using System.Drawing;
using Emgu.CV;
using Emgu.CV.CvEnum;
using Emgu.CV.Dnn;
using Emgu.CV.Structure;

namespace emguCvSsd
{
    class Program
    {
        private static readonly string[] Labels = {"background", "aeroplane", "bicycle", "bird", "boat","bottle", "bus", "car", "cat", "chair","cow", "diningtable", "dog", "horse", "motorbike","person", "pottedplant", "sheep", "sofa", "train", "tvmonitor"};
        private static readonly MCvScalar[] Colors = new MCvScalar[21];

        static void Main()
        {
            //set random color
            Random rnd = new Random();
            for (int i = 0; i < 21; i++)
            {
                Colors[i] = new Rgb(rnd.Next(0,256), rnd.Next(0, 256), rnd.Next(0, 256)).MCvScalar;
            }
            //get image and set model
            Mat img = CvInvoke.Imread("bali-crop.jpg");
            var blob = DnnInvoke.BlobFromImage(img, 1, new Size(512, 515));
            var prototxt = "deploy.prototxt";
            var model = "VGG_VOC0712Plus_SSD_512x512_ft_iter_160000.caffemodel";
            var net = new Net();
            var import = Importer.CreateCaffeImporter(prototxt, model);
            import.PopulateNet(net);
            net.SetInput(blob, "data");
            
            Stopwatch sw = new Stopwatch();
            sw.Start();
            //forward model
            var prob = net.Forward("detection_out");
            sw.Stop();
            Console.WriteLine($"Runtime:{sw.ElapsedMilliseconds} ms");

            //copy result to byte due to egmucv can not access Mat pixel.
            byte[] data = new byte[5600];
            prob.CopyTo(data);

            //draw result
            for (int i = 0; i < prob.SizeOfDimemsion[2]; i++)
            {
                var d = BitConverter.ToSingle(data, i * 28 + 8);
                if (d > 0.4)
                {
                    var idx = (int)BitConverter.ToSingle(data, i * 28 + 4);
                    var w1 = (int) (BitConverter.ToSingle(data, i * 28 + 12) * img.Width);
                    var h1 = (int)(BitConverter.ToSingle(data, i * 28 + 16) * img.Height);
                    var w2 = (int)(BitConverter.ToSingle(data, i * 28 + 20) * img.Width);
                    var h2 = (int)(BitConverter.ToSingle(data, i * 28 + 24) * img.Height);

                    var label = $"{Labels[idx]} {d * 100:0.00}%";
                    Console.WriteLine(label);
                    CvInvoke.Rectangle(img,new Rectangle(w1,h1,w2-w1,h2-h1),Colors[idx],2);
                    int baseline = 0;
                    var textSize = CvInvoke.GetTextSize(label, FontFace.HersheyTriplex, 0.5, 1, ref baseline);
                    var y = h1 - textSize.Height < 0 ? h1 + textSize.Height : h1;
                    CvInvoke.Rectangle(img,new Rectangle(w1,y-textSize.Height,textSize.Width, textSize.Height),Colors[idx],-1);
                    CvInvoke.PutText(img, label, new Point(w1, y), FontFace.HersheyTriplex, 0.5, new Bgr(0, 0, 0).MCvScalar);
                }
            }

            //Show the image
            CvInvoke.Imshow("image", img); 
            CvInvoke.WaitKey();  
            CvInvoke.DestroyAllWindows();
        }
    }
}

10 comments:

  1. I have the same result in C++ ,OpenCV 3.3.1 DNN SSD . I think the problem may from code inside OpenCV DNN .

    ReplyDelete
    Replies
    1. You mean got wrong result? or the long runtime?

      Delete
  2. This comment has been removed by the author.

    ReplyDelete
  3. This comment has been removed by the author.

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
    Replies
    1. I will try it when I got free time,thank you.

      Delete
  5. Glad to know this page :D
    thank you

    I wonder what's the most accurate and fastest between this and that > https://github.com/opencv/opencv/pull/9705
    I would love to see with OpencvSharp from you @Died Liu

    You need to see this:
    https://docs.opencv.org/3.4.0/da/d9d/tutorial_dnn_yolo.html
    https://github.com/opencv/opencv/blob/master/samples/dnn/yolo_object_detection.cpp
    and the video demo:
    https://www.youtube.com/watch?v=NHtRlndE2cg

    ReplyDelete
    Replies
    1. Hi Blaise,
      Thank you for the request, after some try and error, I succeed to get some result, please check https://i.imgur.com/19do5Hx.png .
      Write to blog will cost some time so maybe tomorrow I will publish it.

      Died

      Delete
    2. @Blaise Thunderbytes, please check this
      http://www.died.tw/2018/01/c-opencvsharp-dnn-with-yolo2.html

      Delete
    3. thank you so much Master @Died Liu

      Delete