Friday, December 15, 2017

[C#] OpenCvSharp DNN test with SSD

OpenCvDnn_2
Last time, we compared different language running in OpenCv DNN at this article OpenCV DNN Speed Test in Python / C# / C++, this time we can test the newest OpenCvSharp with DNN moudule supported.


Let's look the code, most past same as before.

            var file = "bali.jpg";
            var prototxt = "deploy.prototxt";
            var model = "VGG_VOC0712Plus_SSD_512x512_ft_iter_160000.caffemodel";
            var colors = Enumerable.Repeat(false, 21).Select(x=> Scalar.RandomColor()).ToArray();
Prepare things for later use.


            //get image
            var org = Cv2.ImRead(file);
            var blob = CvDnn.BlobFromImage(org,1,new Size(512,512));
            //setup model
            var net = CvDnn.ReadNetFromCaffe(prototxt, model);
            net.SetInput(blob, "data");

            //forward model
            var prob = net.Forward("detection_out");
Setup model and forward it.


            //reshape from [1,1,200,7] to [200,7]
            var p = prob.Reshape(1, prob.Size(2));
Model output 4 dimensions, here reshape it to 2 dimensions for easy using.


            for (int i = 0; i < prob.Size(2); i++)
            {
                var confidence = p.At(i, 2);
                if (confidence > 0.4)
                {
                    //get value what we need
                    var idx = (int)p.At(i, 1);
                    var w1 = (int)(org.Width * p.At(i, 3));
                    var h1 = (int)(org.Width * p.At(i, 4));
                    var w2 = (int)(org.Width * p.At(i, 5));
                    var h2 = (int)(org.Width * p.At(i, 6));
                    var label = $"{Labels[idx]} {confidence * 100:0.00}%";
                    Console.WriteLine($"{label}");
                    //draw result
                    Cv2.Rectangle(org, new Rect(w1, h1, w2 - w1, h2 - h1), colors[idx], 2);
                    var textSize = Cv2.GetTextSize(label, HersheyFonts.HersheyTriplex, 0.5, 1, out var baseline);
                    Cv2.Rectangle(org,new Rect(new Point(w1, h1 - textSize.Height),
                            new Size(textSize.Width, textSize.Height + baseline)), colors[idx], Cv2.FILLED);
                    Cv2.PutText(org, label, new Point(w1, h1), HersheyFonts.HersheyTriplex, 0.5, Scalar.Black);
                }
            }
Get value from result and draw on image like before.

And here is our result image.
OpenCvDnn_1

Runtime on my pc are about 1900 ms~2000 ms, a bit slower than python, but still can be accepted for me.

The Important things is, OpenCvSharp DNN got right result and good runtime, way better than EmguCv.


Full code:
using System;
using System.Diagnostics;
using System.Linq;
using OpenCvSharp;
using OpenCvSharp.Dnn;

namespace OpenCvSharpDnn
{
    class Program
    {
        private static readonly string[] Labels = { "background", "aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair", "cow", "diningtable", "dog", "horse", "motorbike", "person", "pottedplant", "sheep", "sofa", "train", "tvmonitor" };

        static void Main()
        {
            var file = "bali.jpg";
            var prototxt = "deploy.prototxt";
            var model = "VGG_VOC0712Plus_SSD_512x512_ft_iter_160000.caffemodel";
            var colors = Enumerable.Repeat(false, 21).Select(x=> Scalar.RandomColor()).ToArray();
            //get image
            var org = Cv2.ImRead(file);
            var blob = CvDnn.BlobFromImage(org,1,new Size(512,512));
            //setup model
            var net = CvDnn.ReadNetFromCaffe(prototxt, model);
            net.SetInput(blob, "data");

            Stopwatch sw = new Stopwatch();
            sw.Start();
            //forward model
            var prob = net.Forward("detection_out");
            sw.Stop();
            Console.WriteLine($"Runtime:{sw.ElapsedMilliseconds} ms");

            //reshape from [1,1,200,7] to [200,7]
            var p = prob.Reshape(1, prob.Size(2));

            for (int i = 0; i < prob.Size(2); i++)
            {
                var confidence = p.At(i, 2);
                if (confidence > 0.4)
                {
                    //get value what we need
                    var idx = (int)p.At(i, 1);
                    var w1 = (int)(org.Width * p.At(i, 3));
                    var h1 = (int)(org.Width * p.At(i, 4));
                    var w2 = (int)(org.Width * p.At(i, 5));
                    var h2 = (int)(org.Width * p.At(i, 6));
                    var label = $"{Labels[idx]} {confidence * 100:0.00}%";
                    Console.WriteLine($"{label}");
                    //draw result
                    Cv2.Rectangle(org, new Rect(w1, h1, w2 - w1, h2 - h1), colors[idx], 2);
                    var textSize = Cv2.GetTextSize(label, HersheyFonts.HersheyTriplex, 0.5, 1, out var baseline);
                    Cv2.Rectangle(org,new Rect(new Point(w1, h1 - textSize.Height),
                            new Size(textSize.Width, textSize.Height + baseline)), colors[idx], Cv2.FILLED);
                    Cv2.PutText(org, label, new Point(w1, h1), HersheyFonts.HersheyTriplex, 0.5, Scalar.Black);
                }
            }

            using(new Window("image", org))
            {
                Cv2.WaitKey();
            }
        }
    }
}

Or you can check it on github.

Hope you enjoy it.

12 comments:

  1. Hi, thanks for posting your code its a great help. I got it working easy enough. Only problem I had is with the positioning of the bounding boxes, if you change to a different picture they don't seem to be in the right place.

    I also tired some other models from https://github.com/BVLC/caffe/tree/master/models but all gave me an error at this line var net = CvDnn.ReadNetFromCaffe(prototxt, model);

    I am very new to this are so it might be a problem this end.

    ReplyDelete
    Replies
    1. Can you give more info about your bounding boxes issue and the error when you change model ?

      Delete
  2. var w1 = (int)(org.Width * p.At(i, 3));
    var h1 = (int)(org.Width * p.At(i, 4)); //should be org.Height
    var w2 = (int)(org.Width * p.At(i, 5));
    var h2 = (int)(org.Width * p.At(i, 6)); //should be org.Height

    ReplyDelete
  3. I downloaded the caffemodel and deploy.prototxt from here https://github.com/BVLC/caffe/tree/master/models/bvlc_googlenet and changed the path in the code. I get an exception here

    var net = CvDnn.ReadNetFromCaffe(prototxt, model);

    OpenCvSharp.OpenCVException: 'FAILED: ReadProtoFromTextFile(param_file, param). Failed to parse NetParameter file: C:\Users\derekmoc\Documents\Visual Studio 2017\Projects\OpenCvDnnSpeedCompare-master\OpenCvSharpDnn\GoogleNet\deploy.prototxt'

    I am probably doing something really dumb but I am only just starting with OpenCV.

    Thanks.

    ReplyDelete
    Replies
    1. https://docs.opencv.org/3.4/d5/de7/tutorial_dnn_googlenet.html}
      You can try opecv's demo, they have provide caffemodel and prototxt, maybe it will work.

      Delete
  4. Greetings Died Liu, I was able run your OpenCVSharp version (thank you), but would like to use Emgu-CV because I have other software needs. My problem is where it gets to:

    var confidence = prob.At(i, 4);
    if (confidence > threshold)
    {
    //get classes probability
    Cv2.MinMaxLoc(prob.Row[i].ColRange(prefix, prob.Cols), out _, out Point max);

    As you know Emgu-CV doesn't have an .At() and the code in your past post using Emgu-CV the array math doesn't work with this model. Any ideas? Thanks Bruce Kingsley

    ReplyDelete
    Replies
    1. It's a long time I didn't using Emgu-CV, so I have no idea about it, sorry.

      Delete
  5. Hi Died Liu,
    Many thanks for your efforts.
    I would like to know if you have tried MobileNetSSD with opeNCVSharp 3.4.1 ? I have somewhat failed to get it working. Prediction object always gives nothing even though I an using the proper model and config files. Any idea?

    ReplyDelete
    Replies
    1. Maybe target different output layer on net.Forward() ?

      Delete
  6. var import = Emgu.CV.Dnn.Importer.CreateCaffeImporter(prototxt, model);
    I get error in Importer. Function not found. I have install EmguCV 3.4.1.2980 from nuget. Can you help me. i was import Emgu.CV.World

    thanks for response

    ReplyDelete
    Replies
    1. It's a long time I didn't using Emgu-CV, so I have no idea about it, sorry.

      Delete