JeVoisBase  1.21
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
Loading...
Searching...
No Matches
ObjectTracker.C
Go to the documentation of this file.
1// ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2//
3// JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4// California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5//
6// This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7// redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8// Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9// without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10// License for more details. You should have received a copy of the GNU General Public License along with this program;
11// if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12//
13// Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14// Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15// ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16/*! \file */
17
18#include <jevois/Core/Module.H>
20#include <jevois/Debug/Log.H>
21#include <jevois/Util/Utils.H>
23#include <jevois/Debug/Timer.H>
25
26#include <linux/videodev2.h>
27#include <opencv2/imgproc/imgproc.hpp>
28#include <string.h>
29
30// This code was loosely inspired by:
31
32// https://raw.githubusercontent.com/kylehounslow/opencv-tuts/master/object-tracking-tut/objectTrackingTut.cpp
33
34// Written by Kyle Hounslow 2013
35
36// Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
37// documentation files (the "Software") , to deal in the Software without restriction, including without limitation the
38// rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
39// permit persons to whom the Software is furnished to do so, subject to the following conditions:
40//
41// The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
42// Software.
43//
44// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
45// WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
46// COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
47// OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
48
49// icon by Roundicons in miscellaneous at flaticon
50
51//! Simple color-based object detection/tracking
52/*! This modules isolates pixels within a given HSV range (hue, saturation, and value of color pixels), does some
53 cleanups, and extracts object contours. It sends information about object centers over serial.
54
55 This module usually works best with the camera sensor set to manual exposure, manual gain, manual color balance, etc
56 so that HSV color values are reliable. See the file \b script.cfg file in this module's directory for an example of
57 how to set the camera settings each time this module is loaded.
58
59 This code was loosely inspired by:
60 https://raw.githubusercontent.com/kylehounslow/opencv-tuts/master/object-tracking-tut/objectTrackingTut.cpp written
61 by Kyle Hounslow, 2013.
62
63 Serial Messages
64 ---------------
65
66 This module can send standardized serial messages as described in \ref UserSerialStyle. One message is issued on
67 every video frame for each detected and good object (good objects have a pixel area within the range specified by \p
68 objectarea, and are only reported when the image is clean enough according to \p maxnumobj). The \p id field in the
69 messages simply is \b blob for all messages.
70
71 - Serial message type: \b 2D
72 - `id`: always \b blob
73 - `x`, `y`, or vertices: standardized 2D coordinates of blob center or of corners of bounding box
74 (depending on \p serstyle)
75 - `w`, `h`: standardized object size
76 - `extra`: none (empty string)
77
78 \jvversion{1.8.1}: Note that this module sends one message per detected object (blob) and per frame. You may want to
79 specify `setpar serstamp Frame` if you need to get all the objects on one given frame (they will have the same frame
80 number prefix). Here is an example output with `setpar serstyle Detail` and `setpar serstamp Frame` (see \ref
81 UserSerialStyle for more info about \p serstamp and \p serstyle):
82
83 \verbatim
84 2940 D2 blob 4 -238 237 -238 19 13 19 13 237
85 2940 D2 blob 4 266 209 194 206 213 -277 285 -274
86 2940 D2 blob 4 575 -243 499 -511 613 -544 689 -275
87 2940 D2 blob 4 -150 -413 -150 -613 50 -613 50 -413
88 2940 D2 blob 4 -913 -306 -1000 -306 -1000 -750 -913 -750
89 2941 D2 blob 4 -313 119 -313 -106 -63 -106 -63 119
90 2941 D2 blob 4 189 105 115 99 157 -385 231 -379
91 2941 D2 blob 4 469 -275 469 -500 575 -500 575 -275
92 2941 D2 blob 4 -200 -531 -200 -744 -13 -744 -13 -531
93 2942 D2 blob 4 -381 -13 -381 -244 -113 -244 -113 -13
94 2942 D2 blob 4 478 -213 310 -402 414 -494 582 -305
95 2942 D2 blob 4 144 -6 53 -15 114 -668 206 -660
96 \endverbatim
97 In this example, we detected 5 blobs on frame 2940, then 4 blobs on frame 2941, then 3 blobs on frame 2942.
98
99 See \ref UserSerialStyle for more on standardized serial messages, and \ref coordhelpers for more info on
100 standardized coordinates.
101
102 Trying it out
103 -------------
104
105 The default parameter settings (which are set in \b script.cfg explained below) attempt to detect light blue
106 objects. Present a light blue object to the JeVois camera and see whether it is detected. When detected and good
107 enough according to \p objectarea and \p maxnumobj, a green circle will be drawn at the center of each good object.
108
109 For further use of this module, you may want to check out the following tutorials:
110
111 - [Tuning the color-based object tracker using a python graphical
112 interface](http://jevois.org/tutorials/UserColorTracking.html)
113 - [Making a motorized pan-tilt head for JeVois and tracking
114 objects](http://jevois.org/tutorials/UserPanTilt.html)
115 - \ref ArduinoTutorial
116
117 Tuning
118 ------
119
120 You should adjust parameters \p hrange, \p srange, and \p vrange to isolate the range of Hue, Saturation, and Value
121 (respectively) that correspond to the objects you want to detect. Note that there is a \b script.cfg file in this
122 module's directory that provides a range tuned to a lighgt blue object, as shown in the demo screenshot.
123
124 Tuning the parameters is best done interactively by connecting to your JeVois camera while it is looking at some
125 object of the desired color. Once you have achieved a tuning, you may want to set the hrange, srange, and vrange
126 parameters in your \b script.cfg file for this module on the microSD card (see below).
127
128 Typically, you would start by narrowing down on the hue, then the value, and finally the saturation. Make sure you
129 also move your camera around and show it typical background clutter so check for false positives (detections of
130 things which you are not interested, which can happen if your ranges are too wide).
131
132 Config file
133 -----------
134
135 JeVois allows you to store parameter settings and commands in a file named \b script.cfg stored in the directory of
136 a module. The file \b script.cfg may contain any sequence of commands as you would type them interactively in the
137 JeVois command-line interface. For the \jvmod{ObjectTracker} module, a default script is provided that sets the
138 camera to manual color, gain, and exposure mode (for more reliable color values), and to setup communication with a
139 pan/tilt head as described in \ref ArduinoTutorial.
140
141 The \b script.cfg file for \jvmod{ObjectTracker} is stored on your microSD at
142 <b>JEVOIS:/modules/JeVois/ObjectTracker/script.cfg</b> and is shown in \ref ArduinoTutorial as an example.
143
144
145 @author Laurent Itti
146
147 @videomapping YUYV 320 254 60.0 YUYV 320 240 60.0 JeVois ObjectTracker
148 @videomapping NONE 0 0 0.0 YUYV 320 240 60.0 JeVois ObjectTracker
149 @email itti\@usc.edu
150 @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
151 @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
152 @mainurl http://jevois.org
153 @supporturl http://jevois.org/doc
154 @otherurl http://iLab.usc.edu
155 @license GPL v3
156 @distribution Unrestricted
157 @restrictions None
158 \ingroup modules */
160{
161 public:
162 //! Constructor
163 ObjectTracker(std::string const & instance) :
164 jevois::StdModule(instance)
165 {
166 itsDetector = addSubComponent<BlobDetector>("detector");
167 }
168
169 //! Virtual destructor for safe inheritance
170 virtual ~ObjectTracker() { }
171
172 //! Processing function, no USB video output
173 virtual void process(jevois::InputFrame && inframe) override
174 {
175 // Wait for next available camera image. Any resolution and format ok:
176 jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
177
178 // Convert input image to BGR24, then to HSV:
179 cv::Mat imgbgr = jevois::rawimage::convertToCvBGR(inimg);
180 cv::Mat imghsv; cv::cvtColor(imgbgr, imghsv, cv::COLOR_BGR2HSV);
181
182 // Let camera know we are done processing the input image:
183 inframe.done();
184
185 // Detect blobs and get their contours:
186 auto contours = itsDetector->detect(imghsv);
187
188 // Send a serial message for each detected blob:
189 for (auto const & c : contours) sendSerialContour2D(w, h, c, "blob");
190 }
191
192 //! Processing function, with USB video output
193 virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
194 {
195 static jevois::Timer timer("processing");
196
197 // Wait for next available camera image. Any resolution ok, but require YUYV since we assume it for drawings:
198 jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
199 inimg.require("input", w, h, V4L2_PIX_FMT_YUYV);
200
201 timer.start();
202
203 // While we process it, start a thread to wait for output frame and paste the input image into it:
204 jevois::RawImage outimg; // main thread should not use outimg until paste thread is complete
205 auto paste_fut = jevois::async([&]() {
206 outimg = outframe.get();
207 outimg.require("output", w, h + 14, inimg.fmt);
208 jevois::rawimage::paste(inimg, outimg, 0, 0);
209 jevois::rawimage::writeText(outimg, "JeVois Color Object Tracker", 3, 3, jevois::yuyv::White);
210 jevois::rawimage::drawFilledRect(outimg, 0, h, w, outimg.height-h, 0x8000);
211 });
212
213 // Convert input image to BGR24, then to HSV:
214 cv::Mat imgbgr = jevois::rawimage::convertToCvBGR(inimg);
215 cv::Mat imghsv; cv::cvtColor(imgbgr, imghsv, cv::COLOR_BGR2HSV);
216
217 // Detect blobs and get their contours:
218 auto contours = itsDetector->detect(imghsv);
219
220 // Wait for paste to finish up:
221 paste_fut.get();
222
223 // Let camera know we are done processing the input image:
224 inframe.done();
225
226 // Draw all detected contours in a thread:
227 std::future<void> draw_fut = jevois::async([&]() {
228 // We reinterpret the top portion of our YUYV output image as an opencv 8UC2 image:
229 cv::Mat outuc2 = jevois::rawimage::cvImage(outimg); // pixel data shared
230 cv::drawContours(outuc2, contours, -1, jevois::yuyv::LightPink, 2, 8);
231 });
232
233 // Send a serial message and draw a circle for each detected blob:
234 for (auto const & c : contours)
235 {
236 sendSerialContour2D(w, h, c, "blob");
237
238 cv::Moments moment = cv::moments(c);
239 double const area = moment.m00;
240 int const x = int(moment.m10 / area + 0.4999);
241 int const y = int(moment.m01 / area + 0.4999);
242 jevois::rawimage::drawCircle(outimg, x, y, 20, 1, jevois::yuyv::LightGreen);
243 }
244
245 // Show number of detected objects:
246 jevois::rawimage::writeText(outimg, "Detected " + std::to_string(contours.size()) + " objects.",
247 3, h + 2, jevois::yuyv::White);
248
249 // Show processing fps:
250 std::string const & fpscpu = timer.stop();
251 jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
252
253 // Wait until all contours are drawn, if they had been requested:
254 draw_fut.get();
255
256 // Send the output image with our processing results to the host over USB:
257 outframe.send();
258 }
259
260 private:
261 std::shared_ptr<BlobDetector> itsDetector;
262};
263
264// Allow the module to be loaded as a shared object (.so) file:
int h
double area(const std::vector< Point2D< T > > &polygon, const bool getsigned=false)
What is the area of a polygon?
Definition Point2D.H:422
Simple color-based object detection/tracking.
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function, with USB video output.
virtual void process(jevois::InputFrame &&inframe) override
Processing function, no USB video output.
ObjectTracker(std::string const &instance)
Constructor.
virtual ~ObjectTracker()
Virtual destructor for safe inheritance.
unsigned int fmt
unsigned int width
unsigned int height
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
StdModule(std::string const &instance)
void sendSerialContour2D(unsigned int camw, unsigned int camh, std::vector< cv::Point_< T > > points, std::string const &id="", std::string const &extra="")
std::string const & stop(double *seconds)
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
cv::Mat cvImage(RawImage const &src)
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
cv::Mat convertToCvBGR(RawImage const &src)
void drawCircle(RawImage &img, int x, int y, unsigned int rad, unsigned int thick, unsigned int col)
std::future< std::invoke_result_t< std::decay_t< Function >, std::decay_t< Args >... > > async(Function &&f, Args &&... args)
JEVOIS_REGISTER_MODULE(TutorialPassThrough)