JeVoisBase  1.21
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
Loading...
Searching...
No Matches
PassThrough.C
Go to the documentation of this file.
1// ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2//
3// JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4// California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5//
6// This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7// redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8// Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9// without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10// License for more details. You should have received a copy of the GNU General Public License along with this program;
11// if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12//
13// Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14// Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15// ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16/*! \file */
17
18#include <jevois/Core/Module.H>
19
21#include <linux/videodev2.h>
22
23// icon by Catalin Fertu in cinema at flaticon
24
25//! Simple module that just passes the captured camera frames through to USB host
26/*! This module makes your JeVois smart camera operate like a regular "dumb" camera. It is intended mainly for use in
27 programming tutorials, and to allow you to debug new machine vision modules that you test on your host computer,
28 using the JeVois camera in pass-through mode as input, to simulate what will happen when your code runs on the
29 JeVois embedded processor.
30
31 Any video mapping is possible here, as long as camera and USB pixel types match, and camera and USB image
32 resolutions also match.
33
34 See \ref PixelFormats for information about pixel formats; with this module you can use the formats supported by the
35 camera sensor: YUYV, BAYER, RGB565, and resolutions:
36
37 - SXGA (1280 x 1024): up to 15 fps
38 - VGA (640 x 480): up to 30 fps
39 - CIF (352 x 288): up to 60 fps
40 - QVGA (320 x 240): up to 60 fps
41 - QCIF (176 x 144): up to 120 fps
42 - QQVGA (160 x 120): up to 60 fps
43 - QQCIF (88 x 72): up to 120 fps
44
45 Things to try
46 -------------
47
48 Edit <b>JEVOIS:/config/videomappings.cfg</b> on your MicroSD card (see \ref VideoMapping) and try to add some new
49 pass-through mappings. Not all of the possible pass-through mappings have been included in the card to avoid having
50 too many of these simple "dumb camera" mappings in the base software distribution. For example, you can try
51
52 \verbatim
53 YUYV 176 144 115.0 YUYV 176 144 115.0 JeVois PassThrough
54 \endverbatim
55
56 will grab YUYV frames on the sensor, with resolution 176x144 at 115 frames/s, and will directly send them to the
57 host computer over the USB link. To test this mapping, select the corresponding resolution and framerate in your
58 video viewing software (here, YUYV 176x144 \@ 115fps). Although the sensor can capture at up to 120fps at this
59 resolution, here we used 115fps to avoid a conflict with a mapping using YUYV 176x144 \@ 120fps USB output and the
60 \jvmod{SaveVideo} module that is already in the default <b>videomappings.cfg</b> file.
61
62 Note that this module may suffer from DMA coherency artifacts if the \p camturbo parameter of the jevois::Engine is
63 turned on, which it is by default. The \p camturbo parameter relaxes some of the cache coherency constraints on the
64 video buffers captured by the camera sensor, which allows the JeVois processor to access video pixel data from
65 memory faster. But with modules that do not do much processing, sometimes this yields video artifacts, we presume
66 because some of the video data from previous frames still is in the CPU cache and hence is not again fetched from
67 main memory by the CPU. If you see short stripes of what appears to be wrong pixel colors in the video, try to
68 disable \p camturbo, by editing <b>JEVOIS:/config/params.cfg</b> on your MicroSD card and in there turning \p
69 camturbo to false.
70
71
72 @author Laurent Itti
73
74 @videomapping YUYV 1280 1024 7.5 YUYV 1280 1024 7.5 JeVois PassThrough
75 @videomapping YUYV 640 480 30.0 YUYV 640 480 30.0 JeVois PassThrough
76 @videomapping YUYV 640 480 19.6 YUYV 640 480 19.6 JeVois PassThrough
77 @videomapping YUYV 640 480 12.0 YUYV 640 480 12.0 JeVois PassThrough
78 @videomapping YUYV 640 480 8.3 YUYV 640 480 8.3 JeVois PassThrough
79 @videomapping YUYV 640 480 7.5 YUYV 640 480 7.5 JeVois PassThrough
80 @videomapping YUYV 640 480 5.5 YUYV 640 480 5.5 JeVois PassThrough
81 @email itti\@usc.edu
82 @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
83 @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
84 @mainurl http://jevois.org
85 @supporturl http://jevois.org/doc
86 @otherurl http://iLab.usc.edu
87 @license GPL v3
88 @distribution Unrestricted
89 @restrictions None
90 \ingroup modules */
92{
93 public:
94 //! Default base class constructor ok
96
97 //! Virtual destructor for safe inheritance
98 virtual ~PassThrough() { }
99
100 //! Processing function
101 virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
102 {
103 // Wait for next available camera image:
104 jevois::RawImage const inimg = inframe.get(true);
105
106 // Wait for an image from our gadget driver into which we will put our results:
107 jevois::RawImage outimg = outframe.get();
108
109 // Enforce that the input and output formats and image sizes match:
110 outimg.require("output", inimg.width, inimg.height, inimg.fmt);
111
112 // Just copy the pixel data over:
113 memcpy(outimg.pixelsw<void>(), inimg.pixels<void>(), std::min(inimg.buf->length(), outimg.buf->length()));
114
115 // Camera outputs RGB565 in big-endian, but most video grabbers expect little-endian:
116 if (outimg.fmt == V4L2_PIX_FMT_RGB565) jevois::rawimage::byteSwap(outimg);
117
118 // Let camera know we are done processing the input image:
119 inframe.done(); // NOTE: optional here, inframe destructor would call it anyway
120
121 // Send the output image with our processing results to the host over USB:
122 outframe.send(); // NOTE: optional here, outframe destructor would call it anyway
123 }
124
125#ifdef JEVOIS_PRO
126 //! Processing function with zero-copy and GUI on JeVois-Pro
127 virtual void process(jevois::InputFrame && inframe, jevois::GUIhelper & helper) override
128 {
129 // Start the frame: Internally, this initializes or resizes the display as needed, polls and handles events
130 // (inputs, window resize, etc), and clears the frame. Variables winw and winh are set by startFrame() to the
131 // current window size, and true is returned if no keyboard/mouse action in a while (can be used to hide any GUI
132 // elements when user is not interacting with JeVois):
133 unsigned short winw, winh;
134 bool idle = helper.startFrame(winw, winh);
135
136 // Test mode: replace frame by checkerboard:
137 static bool testmode = false;
138 static bool halt = true;
139 static bool valt = false;
140 static bool noalias = true;
141 static bool showscaled = true;
142 static bool usedma = false;
143 static int scaledx = 0;
144 static int scaledy = 0;
145
146 if (testmode)
147 {
148 jevois::RawImage const & img = inframe.get();
149 unsigned char * pix = (unsigned char *)img.pixels<unsigned char>();
150 int w = img.width, h = img.height;
151 bool oddv = false, oddh = false;
152
153 switch (img.fmt)
154 {
155 case V4L2_PIX_FMT_RGB24:
156 {
157 for (int j = 0; j < h; ++j)
158 {
159 bool hh = valt ? oddv : halt;
160 oddh = hh ? false : oddv;
161 for (int i = 0; i < w; ++i)
162 {
163 if (oddh) { *pix++ = 0; *pix++ = 0; *pix++ = 0; }
164 else { *pix++ = 255; *pix++ = 255; *pix++ = 255; }
165 if (hh) oddh = !oddh;
166 }
167 oddv = !oddv;
168 }
169 }
170 break;
171 case V4L2_PIX_FMT_YUYV:
172 {
173 for (int j = 0; j < h; ++j)
174 {
175 bool hh = valt ? oddv : halt;
176 oddh = hh ? false : oddv;
177 for (int i = 0; i < w; ++i)
178 {
179 if (oddh) { *pix++ = 0; *pix++ = 0x80; }
180 else { *pix++ = 0xff; *pix++ = 0x80; }
181 if (hh) oddh = !oddh;
182 }
183 oddv = !oddv;
184 }
185 }
186 break;
187 case V4L2_PIX_FMT_RGB32:
188 {
189 for (int j = 0; j < h; ++j)
190 {
191 bool hh = valt ? oddv : halt;
192 oddh = hh ? false : oddv;
193 for (int i = 0; i < w; ++i)
194 {
195 if (oddh) { *pix++ = 0; *pix++ = 0; *pix++ = 0; *pix++ = 255; }
196 else { *pix++ = 255; *pix++ = 255; *pix++ = 255; *pix++ = 255; }
197 if (hh) oddh = !oddh;
198 }
199 oddv = !oddv;
200 }
201 }
202 break;
203 }
204 }
205
206 jevois::RawImage const & imgdebug = inframe.get(); // to get dims only
207
208 // In the PassThrough module, just draw the camera input frame, as large as we can, which is achieved by passing
209 // zero dims. The helper with compute the image position and size to make it as large as possible without changing
210 // aspect ratio, and size and position variables will be updated so we know what they are:
211 int x = 0, y = 0; unsigned short w = 0, h = 0;
212 if (usedma) helper.drawInputFrame("dc", inframe, x, y, w, h, noalias);
213 else
214 {
215 jevois::RawImage const & img = inframe.get();
216 helper.drawImage("c", img, x, y, w, h, noalias);
217 }
218 inframe.done();
219
220 // If we have a second image from the ISP, display it:
221 if (inframe.hasScaledImage() && showscaled)
222 {
223 jevois::RawImage const & img2 = inframe.get2();
224 unsigned short w2 = img2.width, h2 = img2.height;
225 if (usedma) helper.drawInputFrame2("ds", inframe, scaledx , scaledy, w2, h2, noalias);
226 else helper.drawImage("s", img2, scaledx, scaledy, w2, h2, noalias);
227
228 inframe.done2();
229 }
230
231 if (idle == false)
232 {
233 // To draw things on top of input video but behind ImGui windows, use ImGui global background draw list:
234 auto dlb = ImGui::GetBackgroundDrawList(); // or use GetForegroundDrawList to draw in front of ImGui
235 ImVec2 const p = ImGui::GetMousePos();
236 dlb->AddRect(ImVec2(p.x-30, p.y-30), ImVec2(p.x+30, p.y+30), ImColor(255, 0, 0, 255) /* red */);
237
238 // Just draw a simple ImGui window that shows fps if we are not idle:
239 ImGuiIO & io = ImGui::GetIO();
240 if (ImGui::Begin("JeVois-Pro PassThrough Module"))
241 {
242 ImGui::Text("Framerate: %3.2f fps", io.Framerate);
243 ImGui::Text("Video: raw %dx%d aspect=%f, render %dx%d @ %d,%d", imgdebug.width, imgdebug.height,
244 float(imgdebug.width) / float(imgdebug.height), w, h, x, y);
245 if (inframe.hasScaledImage())
246 {
247 ImGui::Checkbox("Show ISP scaled image", &showscaled);
248 ImGui::SliderInt("Scaled image x", &scaledx, 0, winw);
249 ImGui::SliderInt("Scaled image y", &scaledy, 0, winh);
250 }
251
252 ImGui::Checkbox("Use DMABUF", &usedma);
253 ImGui::Checkbox("Test mode", &testmode);
254 if (testmode)
255 {
256 ImGui::Checkbox("Pattern 1", &halt);
257 ImGui::Checkbox("Pattern 2", &valt);
258 ImGui::Checkbox("No aliasing", &noalias);
259 }
260 }
261 ImGui::End();
262
263 // To draw things on top of input video and on top of ImGui windows, use ImGui global foregound draw list:
264 auto dlf = ImGui::GetForegroundDrawList();
265 dlf->AddCircle(ImVec2(p.x, p.y), 20, ImColor(0, 255, 0, 128) /* semi transparent green */);
266 }
267
268 // Render the image and GUI:
269 helper.endFrame();
270 }
271#endif // JEVOIS_PRO
272};
273
274// Allow the module to be loaded as a shared object (.so) file:
JEVOIS_REGISTER_MODULE(ArUcoBlob)
int h
Simple module that just passes the captured camera frames through to USB host.
Definition PassThrough.C:92
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function.
virtual void process(jevois::InputFrame &&inframe, jevois::GUIhelper &helper) override
Processing function with zero-copy and GUI on JeVois-Pro.
virtual ~PassThrough()
Virtual destructor for safe inheritance.
Definition PassThrough.C:98
friend friend class Module
void drawInputFrame(char const *name, InputFrame const &frame, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool casync=false)
bool startFrame(unsigned short &w, unsigned short &h)
void drawInputFrame2(char const *name, InputFrame const &frame, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool casync=false)
void drawImage(char const *name, RawImage const &img, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool isoverlay=false)
unsigned int fmt
T const * pixels() const
unsigned int width
unsigned int height
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
std::shared_ptr< VideoBuf > buf
void byteSwap(RawImage &img)