JeVoisBase  1.21
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
Loading...
Searching...
No Matches
DemoCPUGPU.C
Go to the documentation of this file.
1// ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2//
3// JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4// California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5//
6// This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7// redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8// Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9// without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10// License for more details. You should have received a copy of the GNU General Public License along with this program;
11// if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12//
13// Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14// Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15// ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16/*! \file */
17
18#include <jevois/Core/Module.H>
19#include <jevois/Types/Enum.H>
20
24
26#include <linux/videodev2.h>
27
28#include <opencv2/imgproc/imgproc.hpp>
29#include <future>
30
31// icon by Freepik in computer at flaticon
32
33//! Live saliency computation and image filtering using 4-core CPU and OpenGL-ES 2.0 shaders on the Mali-400MP2 GPU
34/*! This module computes saliency and gist over our 4 CPU cores while also computing 4 different image filters over the
35 GPU, finally combining all results into a single grayscale image:
36
37 - saliency: multicore CPU-based detection of the most conspicuous (most attention-grabbing) object in the field of
38 view.
39 - GPU filter 1: Sobel edge detector
40 - GPU filter 2: Median filter
41 - GPU filter 3: Morphological erosion filter
42 - GPU filter 4: Morphological dilation filter
43
44 For an introduction to visual saliency, see http://ilab.usc.edu/bu/
45
46 Also see \jvmod{DemoSaliency}, \jvmod{JeVoisIntro}, \jvmod{DarknetSaliency} for more about saliency.
47
48 Video output
49 ------------
50
51 The video output is arranged vertically, with, from top to bottom:
52 - Sobel filter results (same size as input image)
53 - Median filter results (same size as input image)
54 - Morphological erosion filter results (same size as input image)
55 - Morphological dilation filter results (same size as input image)
56 - Saliency results (those are very small): from left to right: saliency map, color map, intensity map, orientation
57 map, flicker map, motion map. Map size is input size divided by 8 horizontally and vertically. Gist vector is
58 appended to the right but note that at 160x120 it is truncated (see source code for details).
59
60 Serial Messages
61 ---------------
62
63 This module can send standardized serial messages as described in \ref UserSerialStyle, where all coordinates and
64 sizes are standardized using \ref coordhelpers. One message is issued on every video frame at the temporally
65 filtered attended (most salient) location:
66
67 - Serial message type: \b 2D
68 - `id`: always \b sm (shorthand for saliency map)
69 - `x`, `y`: standardized 2D coordinates of temporally-filtered most salient point
70 - `w`, `h`: always 0, 0
71 - `extra`: none (empty string)
72
73 See \ref UserSerialStyle for more on standardized serial messages, and \ref coordhelpers for more info on
74 standardized coordinates.
75
76 @author Laurent Itti
77
78 @displayname Demo CPU GPU
79 @videomapping GREY 160 495 60.0 YUYV 160 120 60.0 JeVois DemoCPUGPU
80 @email itti\@usc.edu
81 @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
82 @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
83 @mainurl http://jevois.org
84 @supporturl http://jevois.org/doc
85 @otherurl http://iLab.usc.edu
86 @license GPL v3
87 @distribution Unrestricted
88 @restrictions None
89 \ingroup modules */
91{
92 public:
93 //! Constructor
94 DemoCPUGPU(std::string const & instance) : jevois::StdModule(instance)
95 {
96 itsFilter = addSubComponent<FilterGPU>("gpu");
97 itsSaliency = addSubComponent<Saliency>("saliency");
98 itsKF = addSubComponent<Kalman2D>("kalman");
99
100 // Use some fairly large saliency and feature maps so we can see them:
101 itsSaliency->centermin::set(1);
102 itsSaliency->smscale::set(3);
103 }
104
105 //! Virtual destructor for safe inheritance
106 virtual ~DemoCPUGPU() { }
107
108 //! Set our GPU program after we are fully constructed and our Component path has been set
109 void postInit() override
110 {
111 itsFilter->setProgram("shaders/simplevertshader.glsl", "shaders/combofragshader.glsl");
112 itsFilter->setProgramParam2f("offset", -1.0F, -1.0F);
113 itsFilter->setProgramParam2f("scale", 2.0F, 2.0F);
114 }
115
116 //! Processing function
117 virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
118 {
119 // Wait for next available camera image:
120 jevois::RawImage const inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
121 inimg.require("input", w, h, V4L2_PIX_FMT_YUYV); // accept any image size but require YUYV pixels
122
123 // In this demo, the GPU completes earlier than the CPU. Hence, we are going to wait for the output frame in the
124 // main thread (which runs the GPU code). We use a mutex to signal to the saliency thread when the output image is
125 // available. Using a mutex and unique_lock here ensures that we will be exception-safe (which may be trickier
126 // with a condition variable or such):
127 jevois::RawImage outimg;
128 std::unique_lock<std::mutex> lck(itsOutMtx); // mutex will be released by main thread when outimg is available
129
130 // Launch the saliency computation in a thread:
131 auto sal_fut = jevois::async([&](){
132 // Compute saliency and gist:
133 itsSaliency->process(inimg, true);
134
135 // Find most salient point:
136 int mx, my; intg32 msal; itsSaliency->getSaliencyMax(mx, my, msal);
137
138 // Compute attended location in original frame coordinates:
139 int const smlev = itsSaliency->smscale::get();
140 int const smadj = smlev > 0 ? (1 << (smlev-1)) : 0; // half a saliency map pixel adjustment
141 unsigned int const dmx = (mx << smlev) + smadj;
142 unsigned int const dmy = (my << smlev) + smadj;
143 unsigned int const mapw = itsSaliency->salmap.dims.w, maph = itsSaliency->salmap.dims.h;
144 //unsigned int const gistsize = itsSaliency->gist_size;
145 unsigned int const gistw = w - 6 * mapw; // width for gist block
146 //unsigned int const gisth = (gistsize + gistw - 1) / gistw; // divide gist_size by w and round up
147
148 // Filter over time the salient location coordinates:
149 itsKF->set(dmx, dmy, w, h);
150 float kfxraw, kfyraw; itsKF->get(kfxraw, kfyraw, 1.0F); // round to int for serial
151
152 // Send kalman-filtered most-salient-point info to serial port (for arduino, etc):
153 sendSerialStd2D(kfxraw, kfyraw, 0.0F, 0.0F, "sm");
154
155 // Wait for output image to be available:
156 std::lock_guard<std::mutex> _(itsOutMtx);
157
158 // Paste saliency and feature maps:
159 unsigned int offset = 0;
160 pasteGrayMap(outimg, itsSaliency->salmap, offset, h*4, 20);
161 pasteGrayMap(outimg, itsSaliency->color, offset, h*4, 18);
162 pasteGrayMap(outimg, itsSaliency->intens, offset, h*4, 18);
163 pasteGrayMap(outimg, itsSaliency->ori, offset, h*4, 18);
164 pasteGrayMap(outimg, itsSaliency->flicker, offset, h*4, 18);
165 pasteGrayMap(outimg, itsSaliency->motion, offset, h*4, 18);
166
167 // Paste gist. Note: at 160x120 we will only end up pasting the first 600 gist entries. This code may fail at
168 // different resolutions, beware:
169 unsigned char * d = outimg.pixelsw<unsigned char>() + 4*w*h + 6*mapw;
170 for (unsigned int i = 0; i < maph; ++i) memcpy(d + i*w, itsSaliency->gist + i*gistw, gistw);
171 });
172
173 // Convert input image to grayscale:
174 cv::Mat grayimg = jevois::rawimage::convertToCvGray(inimg);
175
176 // Once saliency is done using the input image, let camera know we are done with it:
177 itsSaliency->waitUntilDoneWithInput();
178 inframe.done();
179
180 // Upload the greyscale image to GPU and apply 4 grayscale GPU filters, storing the 4 results into an RGBA image:
181 cv::Mat gpuout(h, w, CV_8UC4);
182 itsFilter->process(grayimg, gpuout);
183
184 // Wait for an image from our gadget driver into which we will put our results:
185 outimg = outframe.get();
186 lck.unlock(); // let saliency thread proceed with using the output image
187 outimg.require("output", w, h * 4 + (h >> itsSaliency->smscale::get()), V4L2_PIX_FMT_GREY);
188
189 // Unpack the GPU output into 4 gray images, starting from top-left of the output image:
191
192 // Wait until saliency computation is complete:
193 sal_fut.get();
194
195 // Send the output image with our processing results to the host over USB:
196 outframe.send();
197 }
198
199 // ####################################################################################################
200 //! Paste a map and add its width to the dx offset
201 /*! Beware this is for a gray outimg only. */
202 void pasteGrayMap(jevois::RawImage & outimg, env_image const & fmap, unsigned int & dx, unsigned int dy,
203 unsigned int bitshift)
204 {
205 env_size_t const fw = fmap.dims.w, fh = fmap.dims.h;
206 unsigned int const ow = outimg.width, oh = outimg.height;
207
208 if (dy + fh > oh) LFATAL("Map would extend past output image bottom");
209 if (fw + dx > ow) LFATAL("Map would extend past output image width");
210
211 unsigned int const stride = ow - fw;
212
213 intg32 * s = fmap.pixels; unsigned char * d = outimg.pixelsw<unsigned char>() + dx + dy * ow;
214
215 for (unsigned int j = 0; j < fh; ++j)
216 {
217 for (unsigned int i = 0; i < fw; ++i)
218 {
219 intg32 v = (*s++) >> bitshift; if (v > 255) v = 255;
220 *d++ = (unsigned char)(v);
221 }
222 d += stride;
223 }
224
225 // Add the map width to the dx offset:
226 dx += fw;
227 }
228
229 private:
230 std::shared_ptr<FilterGPU> itsFilter;
231 std::shared_ptr<Saliency> itsSaliency;
232 std::shared_ptr<Kalman2D> itsKF;
233 std::mutex itsOutMtx;
234};
235
236// Allow the module to be loaded as a shared object (.so) file:
JEVOIS_REGISTER_MODULE(ArUcoBlob)
int h
Live saliency computation and image filtering using 4-core CPU and OpenGL-ES 2.0 shaders on the Mali-...
Definition DemoCPUGPU.C:91
DemoCPUGPU(std::string const &instance)
Constructor.
Definition DemoCPUGPU.C:94
virtual ~DemoCPUGPU()
Virtual destructor for safe inheritance.
Definition DemoCPUGPU.C:106
void postInit() override
Set our GPU program after we are fully constructed and our Component path has been set.
Definition DemoCPUGPU.C:109
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function.
Definition DemoCPUGPU.C:117
void pasteGrayMap(jevois::RawImage &outimg, env_image const &fmap, unsigned int &dx, unsigned int dy, unsigned int bitshift)
Paste a map and add its width to the dx offset.
Definition DemoCPUGPU.C:202
unsigned int width
unsigned int height
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
StdModule(std::string const &instance)
void sendSerialStd2D(float x, float y, float w=0.0F, float h=0.0F, std::string const &id="", std::string const &extra="")
ENV_INTG32_TYPE intg32
32-bit signed integer
Definition env_types.h:52
unsigned long env_size_t
Definition env_types.h:71
#define LFATAL(msg)
cv::Mat convertToCvGray(RawImage const &src)
void unpackCvRGBAtoGrayRawImage(cv::Mat const &src, RawImage &dst)
std::future< std::invoke_result_t< std::decay_t< Function >, std::decay_t< Args >... > > async(Function &&f, Args &&... args)
env_size_t w
The width.
Definition env_types.h:82
env_size_t h
The height.
Definition env_types.h:83
Basic image class.
Definition env_image.h:44
struct env_dims dims
Definition env_image.h:45
intg32 * pixels
Definition env_image.h:46