JeVoisBase  1.21
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
Loading...
Searching...
No Matches
DemoArUco.C
Go to the documentation of this file.
1// ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2//
3// JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4// California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5//
6// This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7// redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8// Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9// without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10// License for more details. You should have received a copy of the GNU General Public License along with this program;
11// if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12//
13// Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, BA 90089-2520 - USA.
14// Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15// ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16/*! \file */
17
18#include <jevois/Core/Module.H>
19#include <jevois/Debug/Timer.H>
22#include <opencv2/core/core.hpp>
23
24//! Simple demo of ArUco and AprilTag augmented reality markers detection and decoding
25/*! Detect and decode patterns known as ArUco markers, which are small 2D barcodes often used in augmented
26 reality and robotics.
27
28 ArUco and AprilTag markers are small 2D barcodes. Each marker corresponds to a number, encoded into a small grid of
29 black and white pixels. The marker decoding algorithm is capable of locating, decoding, and of estimating the pose
30 (location and orientation in space) of any compatible markers in the camera's field of view.
31
32 ArUco and AprilTag markers are very useful as tags for many robotics and augmented reality applications. For
33 example, one may place an ArUco next to a robot's charging station, an elevator button, or an object that a robot
34 should manipulate.
35
36 The implementation of ArUco used by JeVois is the one of OpenCV-Contrib, documented here:
37 https://docs.opencv.org/4.x/d5/dae/tutorial_aruco_detection.html
38
39 ArUco and AprilTag markers can be created with several standard dictionaries. Different dictionaries give rise to
40 different numbers of pixels in the markers, and to different numbers of possible symbols that can be created using
41 the dictionary. The default dictionary used by JeVois is ArUco 4x4 with 50 symbols. Other dictionaries are also
42 supported by setting the parameter \p dictionary over serial port or in a config file, up to 7x7 with 1000 symbols.
43
44 Creating and printing markers
45 -----------------------------
46
47 We have created the 50 markers available in the default ArUco dictionary (4x4_50) as PNG images that you can
48 download and print, at http://jevois.org/data/ArUco.zip
49
50 To make your own, for example, using another dictionary, see the documentation of the \ref ArUco component of
51 JeVoisBase. Some utilities are provided with the component.
52
53 Serial Messages
54 ---------------
55
56 This module can send standardized serial messages as described in \ref UserSerialStyle.
57
58 When \p dopose is turned on, 3D messages will be sent, otherwise 2D messages.
59
60 One message is issued for every detected ArUco, on every video frame.
61
62 2D messages when \p dopose is off:
63
64 - Serial message type: \b 2D
65 - `id`: decoded ArUco marker ID, with a prefix 'U'
66 - `x`, `y`, or vertices: standardized 2D coordinates of marker center or corners
67 - `w`, `h`: standardized marker size
68 - `extra`: none (empty string)
69
70 3D messages when \p dopose is on:
71
72 - Serial message type: \b 3D
73 - `id`: decoded ArUco marker ID, with a prefix 'U'
74 - `x`, `y`, `z`, or vertices: 3D coordinates in millimeters of marker center or corners
75 - `w`, `h`, `d`: marker size in millimeters, a depth of 1mm is always used
76 - `extra`: none (empty string)
77
78 If you will use the quaternion data (Detail message style; see \ref UserSerialStyle), you should probably set the \p
79 serprec parameter to something non-zero to get enough accuracy in the quaternion values.
80
81 See \ref UserSerialStyle for more on standardized serial messages, and \ref coordhelpers for more info on
82 standardized coordinates.
83
84 Things to try
85 -------------
86
87 - First, use a video viewer software on a host computer and select one of the video modes with video output over
88 USB. Point your JeVois camera towards one of the screenshots provided with this module, or towards some ArUco
89 markers that you find on the web or that you have printed from the collection above (note: the default dictionary
90 is 4x4_50, see parameter \p dictionary).
91
92 - Then try it with no video output, as it would be used by a robot. Connect to the command-line interface of your
93 JeVois camera through the serial-over-USB connection (see \ref UserCli; on Linux, you would use <b>sudo screen
94 /dev/ttyACM0 115200</b>) and try:
95 \verbatim
96 setpar serout USB
97 setmapping2 YUYV 320 240 30.0 JeVois DemoArUco
98 streamon
99 \endverbatim
100 and point the camera to some markers; the camera should issue messages about all the markers it identifies.
101
102 Computing and showing 3D pose
103 -----------------------------
104
105 The OpenCV ArUco module can also compute the 3D location and orientation of each marker in the world when \p dopose
106 is true. The requires that the camera be calibrated, see the documentation of the \ref ArUco component in
107 JeVoisBase. A generic calibration that is for a JeVois camera with standard lens is included in files \b
108 calibration640x480.yaml, \b calibration352x288.yaml, etc in the jevoisbase share directory (on the MicroSD, this is
109 in <b>JEVOIS:/share/camera/</b>).
110
111 When doing pose estimation, you should set the \p markerlen parameter to the size (width) in millimeters of your
112 actual physical markers. Knowing that size will allow the pose estimation algorithm to know where in the world your
113 detected markers are.
114
115 For more about camera calibration, see [this tutorial](http://jevois.org/tutorials/UserArUcoCalib.html) and
116 http://jevois.org/basedoc/ArUco_8H_source.html
117
118 Tutorial and video
119 ------------------
120
121 Check out this tutorial on how to [build a simple visually-guided toy robot car for under $100 with
122 JeVois](http://jevois.org/tutorials/UserRobotCar.html), which uses ArUco at its core. A demo video is here:
123
124 \youtube{7cMtD-ef83E}
125
126
127 @author Laurent Itti
128
129 @displayname Demo ArUco
130 @videomapping NONE 0 0 0 YUYV 320 240 30.0 JeVois DemoArUco
131 @videomapping YUYV 320 260 30.0 YUYV 320 240 30.0 JeVois DemoArUco
132 @videomapping YUYV 640 500 20.0 YUYV 640 480 20.0 JeVois DemoArUco
133 @email itti\@usc.edu
134 @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
135 @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
136 @mainurl http://jevois.org
137 @supporturl http://jevois.org/doc
138 @otherurl http://iLab.usc.edu
139 @license GPL v3
140 @distribution Unrestricted
141 @restrictions None
142 \ingroup modules */
144{
145 public:
146 // ####################################################################################################
147 //! Constructor
148 // ####################################################################################################
149 DemoArUco(std::string const & instance) : jevois::StdModule(instance)
150 {
151 itsArUco = addSubComponent<ArUco>("aruco");
152 }
153
154 // ####################################################################################################
155 //! Virtual destructor for safe inheritance
156 // ####################################################################################################
157 virtual ~DemoArUco()
158 { }
159
160 // ####################################################################################################
161 //! Processing function, no video output
162 // ####################################################################################################
163 virtual void process(jevois::InputFrame && inframe) override
164 {
165 // Wait for next available camera image, any format and resolution ok here:
166 jevois::RawImage const inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
167
168 // Convert the image to grayscale and process:
169 cv::Mat cvimg = jevois::rawimage::convertToCvGray(inimg);
170 std::vector<int> ids;
171 std::vector<std::vector<cv::Point2f> > corners;
172 itsArUco->detectMarkers(cvimg, ids, corners);
173
174 // Do pose computation if desired:
175 std::vector<cv::Vec3d> rvecs, tvecs;
176 if (itsArUco->dopose::get() && ids.empty() == false)
177 itsArUco->estimatePoseSingleMarkers(corners, rvecs, tvecs);
178
179 // Let camera know we are done processing the input image:
180 inframe.done();
181
182 // Send serial output:
183 itsArUco->sendSerial(this, ids, corners, w, h, rvecs, tvecs);
184 }
185
186 // ####################################################################################################
187 //! Processing function with video output to USB
188 // ####################################################################################################
189 virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
190 {
191 static jevois::Timer timer("processing", 100, LOG_DEBUG);
192
193 // Wait for next available camera image:
194 jevois::RawImage const inimg = inframe.get();
195
196 timer.start();
197
198 // We only handle one specific pixel format, any size in this demo:
199 unsigned int const w = inimg.width, h = inimg.height;
200 inimg.require("input", w, h, V4L2_PIX_FMT_YUYV);
201
202 // While we process it, start a thread to wait for out frame and paste the input into it:
203 jevois::RawImage outimg;
204 auto paste_fut = jevois::async([&]() {
205 outimg = outframe.get();
206 outimg.require("output", w, h + 20, inimg.fmt);
207 jevois::rawimage::paste(inimg, outimg, 0, 0);
208 jevois::rawimage::writeText(outimg, "JeVois ArUco Demo", 3, 3, jevois::yuyv::White);
210 });
211
212 // Convert the image to grayscale and process:
213 cv::Mat cvimg = jevois::rawimage::convertToCvGray(inimg);
214 std::vector<int> ids;
215 std::vector<std::vector<cv::Point2f> > corners;
216 std::vector<cv::Vec3d> rvecs, tvecs;
217 itsArUco->detectMarkers(cvimg, ids, corners);
218
219 if (itsArUco->dopose::get() && ids.empty() == false)
220 itsArUco->estimatePoseSingleMarkers(corners, rvecs, tvecs);
221
222 // Wait for paste to finish up:
223 paste_fut.get();
224
225 // Let camera know we are done processing the input image:
226 inframe.done();
227
228 // Show all the results:
229 itsArUco->drawDetections(outimg, 3, h+5, ids, corners, rvecs, tvecs);
230
231 // Send serial output:
232 itsArUco->sendSerial(this, ids, corners, w, h, rvecs, tvecs);
233
234 // Show processing fps:
235 std::string const & fpscpu = timer.stop();
236 jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
237
238 // Send the output image with our processing results to the host over USB:
239 outframe.send();
240 }
241
242#ifdef JEVOIS_PRO
243 // ####################################################################################################
244 //! Processing function with zero-copy and GUI on JeVois-Pro
245 // ####################################################################################################
246 virtual void process(jevois::InputFrame && inframe, jevois::GUIhelper & helper) override
247 {
248 static jevois::Timer timer("processing", 100, LOG_DEBUG);
249
250 // Start the GUI frame:
251 unsigned short winw, winh;
252 helper.startFrame(winw, winh);
253
254 // Draw the camera frame:
255 int x = 0, y = 0; unsigned short iw = 0, ih = 0;
256 helper.drawInputFrame("camera", inframe, x, y, iw, ih);
257
258 // Wait for next available camera image:
259 jevois::RawImage const inimg = inframe.getp();
260 unsigned int const w = inimg.width, h = inimg.height;
261 helper.itext("JeVois-Pro ArUco Marker Detection");
262
263 timer.start();
264
265 // Convert the image to grayscale and process:
266 cv::Mat cvimg = jevois::rawimage::convertToCvGray(inimg);
267 std::vector<int> ids;
268 std::vector<std::vector<cv::Point2f> > corners;
269 std::vector<cv::Vec3d> rvecs, tvecs;
270 itsArUco->detectMarkers(cvimg, ids, corners);
271
272 if (itsArUco->dopose::get() && ids.empty() == false)
273 itsArUco->estimatePoseSingleMarkers(corners, rvecs, tvecs);
274
275 // Let camera know we are done processing the input image:
276 inframe.done();
277
278 // Show all the results:
279 itsArUco->drawDetections(helper, ids, corners, rvecs, tvecs);
280
281 // Send serial output:
282 itsArUco->sendSerial(this, ids, corners, w, h, rvecs, tvecs);
283
284 // Show processing fps:
285 std::string const & fpscpu = timer.stop();
286 helper.iinfo(inframe, fpscpu, winw, winh);
287
288 // Render the image and GUI:
289 helper.endFrame();
290 }
291#endif
292
293 // ####################################################################################################
294 protected:
295 std::shared_ptr<ArUco> itsArUco;
296};
297
298// Allow the module to be loaded as a shared object (.so) file:
JEVOIS_REGISTER_MODULE(ArUcoBlob)
int h
Simple demo of ArUco and AprilTag augmented reality markers detection and decoding.
Definition DemoArUco.C:144
virtual void process(jevois::InputFrame &&inframe, jevois::GUIhelper &helper) override
Processing function with zero-copy and GUI on JeVois-Pro.
Definition DemoArUco.C:246
virtual ~DemoArUco()
Virtual destructor for safe inheritance.
Definition DemoArUco.C:157
DemoArUco(std::string const &instance)
Constructor.
Definition DemoArUco.C:149
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function with video output to USB.
Definition DemoArUco.C:189
std::shared_ptr< ArUco > itsArUco
Definition DemoArUco.C:295
virtual void process(jevois::InputFrame &&inframe) override
Processing function, no video output.
Definition DemoArUco.C:163
void drawInputFrame(char const *name, InputFrame const &frame, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool casync=false)
bool startFrame(unsigned short &w, unsigned short &h)
void iinfo(jevois::InputFrame const &inframe, std::string const &fpscpu, unsigned short winw=0, unsigned short winh=0)
void itext(char const *txt, ImU32 const &col=IM_COL32_BLACK_TRANS, int line=-1)
unsigned int fmt
unsigned int width
unsigned int height
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
StdModule(std::string const &instance)
std::string const & stop(double *seconds)
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
cv::Mat convertToCvGray(RawImage const &src)
void drawFilledRect(RawImage &img, int x, int y, unsigned int w, unsigned int h, unsigned int col)
std::future< std::invoke_result_t< std::decay_t< Function >, std::decay_t< Args >... > > async(Function &&f, Args &&... args)
unsigned short constexpr Black
unsigned short constexpr White