JeVoisBase  1.21
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
Loading...
Searching...
No Matches
ColorFiltering.C
Go to the documentation of this file.
1// ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2//
3// JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2017 by Laurent Itti, the University of Southern
4// California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5//
6// This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7// redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8// Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9// without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10// License for more details. You should have received a copy of the GNU General Public License along with this program;
11// if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12//
13// Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14// Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15// ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16/*! \file */
17
18#include <jevois/Core/Module.H>
19#include <jevois/Types/Enum.H>
20#include <jevois/Debug/Timer.H>
22
23#include <opencv2/imgproc/imgproc.hpp>
24
31
32// icon by Freepik in other at flaticon
33
34static jevois::ParameterCategory const ParamCateg("Color Filtering Parameters");
35
36//! Enum \relates ColorFiltering
37JEVOIS_DEFINE_ENUM_CLASS(Effect, (NoEffect) (Blur) (Median) (Morpho) (Laplacian) (Bilateral) );
38
39//! Parameter \relates ColorFiltering
40JEVOIS_DECLARE_PARAMETER_WITH_CALLBACK(effect, Effect, "Image processing effect to apply",
41 Effect::Morpho, Effect_Values, ParamCateg);
42
43//! Image filtering using OpenCV
44/*! This module is to learn about basic image filtering. It was developed to allow students to instantly observe the
45 effects of different filters and their parameters on live video. The module implements a variety of filters using
46 OpenCV. Each filter exposes some paremeters (e.g., kernel size) that can be set interactively to understand their
47 effects onto the filter behavior.
48
49 Available filters:
50
51 - Blur: Replace each pixel by the average of all pixels in a box around the pixel of interest. Leads to a
52 blur effect on the image, more pronounced with bigger box sizes. See
53 http://docs.opencv.org/3.2.0/d4/d86/group__imgproc__filter.html#ga8c45db9afe636703801b0b2e440fce37
54
55 - Median: Replaces each pixel by the median value within a box around that pixel. Tends to blur and remove
56 salt-and-peper noise from the
57 image. http://docs.opencv.org/3.2.0/d4/d86/group__imgproc__filter.html#ga564869aa33e58769b4469101aac458f9
58
59 - Morpho: mathematical morphology operations, see
60 http://docs.opencv.org/3.2.0/d9/d61/tutorial_py_morphological_ops.html for an introduction,
61 and http://docs.opencv.org/3.2.0/d4/d86/group__imgproc__filter.html
62
63 - Laplacian: Computes second spatial derivative of the image. Tends to amplify edges and noise. See
64 http://docs.opencv.org/3.2.0/d4/d86/group__imgproc__filter.html#gad78703e4c8fe703d479c1860d76429e6
65
66 - Bilateral: bi-lateral filter, very slow, see
67 http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/MANDUCHI1/Bilateral_Filtering.html and
68 http://docs.opencv.org/3.2.0/d4/d86/group__imgproc__filter.html#ga9d7064d478c95d60003cf839430737ed
69
70 How to use this module
71 ----------------------
72
73 - On JeVois-Pro, select the effect and set its parameters in the Parameters tab of the GUI. You can also drag the
74 cyan handles left and right on screen to change which portion of the image is shown as original vs processed.
75
76 - On JeVois-A33, either try it with JeVois Inventor using <b>YUYV 640x240 @@ 30 fps</b>. Note that each time you
77 select a new \p effect, this will affect the set of parameters that are available for that effect, but currently
78 JeVois Inventor has no way of being notified of that change. So just click to another tab (e.g., the \b Info tab),
79 and then back to the \b Parameters tab each time you change the effect. This will refresh the parameter list.
80
81 - Or, open a video viewer on your host computer and select <b>YUYV 640x240 @@ 30 fps</b> (see \ref UserQuick)
82
83 - Open a serial communication to JeVois (see \ref UserCli)
84
85 - Type `help` and observe the available parameter called \p effect
86
87 - Start by setting the \p effect parameter to a given effect type. For example:
88 `setpar effect Median` or `setpar effect Morpho` (commands are case-sensitive).
89
90 - Then type \c help to see what additional parameters are available for each effect. For example, for Median, you
91 can adjust the kernel size (parameter \p ksize). For Morpho, you can select the type of morphological operation
92 (parameter \p op), structuring element shape (parameter \p kshape) and size (parameter \p ksize), etc.
93
94 Complete example:
95
96 \code
97 setpar effect Morpho
98 help
99 setpar op Open
100 setpar ksize 7 7
101 \endcode
102
103 With \jvversion{1.5} and above, you may want to use `help2` instead of `help`, which is a shorter and more compact
104 help message that shows parameters and commands of the running machine vision module only (and no general
105 parameters related to the JeVois core).
106
107
108 @author Laurent Itti
109
110 @videomapping YUYV 640 240 30.0 YUYV 320 240 30.0 JeVois ColorFiltering
111 @email itti\@usc.edu
112 @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
113 @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
114 @mainurl http://jevois.org
115 @supporturl http://jevois.org/doc
116 @otherurl http://iLab.usc.edu
117 @license GPL v3
118 @distribution Unrestricted
119 @restrictions None
120 \ingroup modules */
122 public jevois::Parameter<effect>
123{
124 public:
125 // ####################################################################################################
126 //! Inherited constructor ok
127 // ####################################################################################################
129
130 // ####################################################################################################
131 //! Virtual destructor for safe inheritance
132 // ####################################################################################################
134 { }
135
136 // ####################################################################################################
137 //! Processing function with video output to USB
138 // ####################################################################################################
139 virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
140 {
141 static jevois::Timer timer("processing", 30, LOG_DEBUG);
142
143 // Wait for next available camera image:
144 jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
145 inimg.require("input", w, h, V4L2_PIX_FMT_YUYV); // accept any image size but require YUYV pixels
146
147 timer.start();
148
149 // While we process it, start a thread to wait for output frame and paste the input image into it:
150 jevois::RawImage outimg; // main thread should not use outimg until paste thread is complete
151 auto paste_fut = jevois::async([&]() {
152 outimg = outframe.get();
153 outimg.require("output", w * 2, h, inimg.fmt);
154 jevois::rawimage::paste(inimg, outimg, 0, 0);
155 jevois::rawimage::writeText(outimg, "JeVois Input Image", 3, 3, jevois::yuyv::White);
156 });
157
158 // Convert source image to BGR:
159 cv::Mat src = jevois::rawimage::convertToCvBGR(inimg);
160
161 // Apply the filter (if any):
162 cv::Mat dst; std::string settings;
163 if (itsFilter) settings = itsFilter->process(src, dst);
164 else dst = cv::Mat(h, w, CV_8UC3, cv::Scalar(0,0,0));
165
166 // Wait for paste to finish up:
167 paste_fut.get();
168
169 // Paste the results into our output:
170 jevois::rawimage::pasteBGRtoYUYV(dst, outimg, w, 0);
171
172 std::string const & fpscpu = timer.stop();
173
174 // Write a few things:
175 jevois::rawimage::writeText(outimg, "JeVois image filter: " + effect::strget(), w + 3, 3, jevois::yuyv::White);
176
177 std::vector<std::string> svec = jevois::split(settings, "\n");
178 for (size_t i = 0; i < svec.size(); ++i)
179 jevois::rawimage::writeText(outimg, svec[i], w + 3, 15 + 12 * i, jevois::yuyv::White);
180
181 jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
182
183 // Send the output image with our processing results to the host over USB:
184 outframe.send();
185 }
186
187#ifdef JEVOIS_PRO
188 // ####################################################################################################
189 //! Processing function with zero-copy and GUI on JeVois-Pro
190 // ####################################################################################################
191 virtual void process(jevois::InputFrame && inframe, jevois::GUIhelper & helper) override
192 {
193 static jevois::Timer timer("processing", 100, LOG_DEBUG);
194 static int left = -1, right = -1;
195
196 // Start the GUI frame:
197 unsigned short winw, winh;
198 helper.startFrame(winw, winh);
199
200 // Draw the camera frame:
201 int x = 0, y = 0; unsigned short iw = 0, ih = 0;
202 helper.drawInputFrame("camera", inframe, x, y, iw, ih);
203
204 helper.itext("JeVois-Pro Color Filtering");
205
206 timer.start();
207
208 // Wait for next available camera image:
209 cv::Mat src = inframe.getCvRGBp(); // should be BGR but RGB is faster and should not affect processing...
210 int const sw = src.cols; int const sh = src.rows;
211 inframe.done();
212
213 // Apply the filter (if any):
214 cv::Mat dst; std::string settings;
215 if (itsFilter) settings = itsFilter->process(src, dst);
216 else dst = cv::Mat(sh, sw, CV_8UC3, cv::Scalar(0,0,0));
217
218 // We will display the processed image as a partial overlay between two vertical lines defined by 'left' and
219 // 'right', where each also has a mouse drag handle (small square).
220
221 ImU32 const col = IM_COL32(128,255,255,255); // color of the vertical lines and square handles
222 int const siz = 20; // size of the square handles, in image pixels
223 static bool dragleft = false, dragright = false;
224
225 // Initialize the handles at 1/4 and 3/4 of image width on first video frame after module is loaded:
226 if (jevois::frameNum() == 0) { left = sw / 4; right = 3 * sw / 4; }
227
228 // Make sure the handles do not overlap and/or get out of the image bounds:
229 if (left > right - siz) { if (dragright) left = right - siz; else right = left + siz; }
230 left = std::max(siz, std::min(sw - siz * 2, left));
231 right = std::max(left + siz, std::min(sw - siz, right));
232
233 // Mask and draw the overlay. To achieve this, we convert the whole result image to RGBA and then assign a zero
234 // alpha channel to all pixels to the left of the 'left' bound and to the right of the 'right' bound:
235 cv::Mat ovl; cv::cvtColor(dst, ovl, cv::COLOR_RGB2RGBA);
236 ovl(cv::Rect(0, 0, left, sh)) = 0; // make left side transparent
237 ovl(cv::Rect(right, 0, sw-right, sh)) = 0; // make right side transparent
238 int ox = 0, oy = 0; unsigned short ow = 0, oh = 0;
239 helper.drawImage("filtered", ovl, true, ox, oy, ow, oh, false, true /* overlay */);
240
241 helper.drawLine(left, 0, left, sh, col);
242 helper.drawRect(left-siz, (sh-siz)/2, left, (sh+siz)/2, col);
243 helper.drawLine(right, 0, right, sh, col);
244 helper.drawRect(right, (sh-siz)/2, right+siz, (sh+siz)/2, col);
245
246 // Adjust the left and right handles if they get clicked and dragged:
247 ImVec2 const ip = helper.d2i(ImGui::GetMousePos());
248
249 if (ImGui::IsMouseClicked(0))
250 {
251 // Are we clicking on the left or right handle?
252 if (ip.x > left-siz && ip.x < left && ip.y > (sh-siz)/2 && ip.y < (sh+siz)/2) dragleft = true;
253 if (ip.x > right && ip.x < right+siz && ip.y > (sh-siz)/2 && ip.y < (sh+siz)/2) dragright = true;
254 }
255
256 if (ImGui::IsMouseDragging(0))
257 {
258 if (dragleft) left = ip.x + 0.5F * siz;
259 if (dragright) right = ip.x - 0.5F * siz;
260 // We will enforce validity of left and right on next frame.
261 }
262
263 if (ImGui::IsMouseReleased(0))
264 {
265 dragleft = false;
266 dragright = false;
267 }
268
269 // Write a few things:
270 std::string const & fpscpu = timer.stop();
271 helper.itext("JeVois image filter: " + effect::strget());
272 std::vector<std::string> svec = jevois::split(settings, "\n");
273 for (std::string const & s : svec) helper.itext(s);
274 helper.iinfo(inframe, fpscpu, winw, winh);
275
276 // Render the image and GUI:
277 helper.endFrame();
278 }
279#endif
280
281 // ####################################################################################################
282 protected:
283 //! Parameter callback: set the selected filter algo
284 void onParamChange(effect const & /*param*/, Effect const & val) override
285 {
286 if (itsFilter) { removeSubComponent(itsFilter); itsFilter.reset(); }
287
288 switch (val)
289 {
290 case Effect::NoEffect: break;
291 case Effect::Blur: itsFilter = addSubComponent<BlurFilter>("filter"); break;
292 case Effect::Median: itsFilter = addSubComponent<MedianFilter>("filter"); break;
293 case Effect::Morpho: itsFilter = addSubComponent<MorphologyFilter>("filter"); break;
294 case Effect::Laplacian: itsFilter = addSubComponent<LaplacianFilter>("filter"); break;
295 case Effect::Bilateral: itsFilter = addSubComponent<BilateralFilter>("filter"); break;
296 }
297 }
298
299 private:
300 std::shared_ptr<Filter> itsFilter;
301};
302
303// Allow the module to be loaded as a shared object (.so) file:
JEVOIS_REGISTER_MODULE(ArUcoBlob)
int h
Image filtering using OpenCV.
void onParamChange(effect const &, Effect const &val) override
Parameter callback: set the selected filter algo.
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function with video output to USB.
JEVOIS_DEFINE_ENUM_CLASS(Effect,(NoEffect)(Blur)(Median)(Morpho)(Laplacian)(Bilateral))
Enum.
virtual ~ColorFiltering()
Virtual destructor for safe inheritance.
JEVOIS_DECLARE_PARAMETER_WITH_CALLBACK(effect, Effect, "Image processing effect to apply", Effect::Morpho, Effect_Values, ParamCateg)
Parameter.
virtual void process(jevois::InputFrame &&inframe, jevois::GUIhelper &helper) override
Processing function with zero-copy and GUI on JeVois-Pro.
friend friend class Module
void removeSubComponent(std::shared_ptr< Comp > &component)
void drawInputFrame(char const *name, InputFrame const &frame, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool casync=false)
bool startFrame(unsigned short &w, unsigned short &h)
void drawRect(float x1, float y1, float x2, float y2, ImU32 col=IM_COL32(128, 255, 128, 255), bool filled=true)
void iinfo(jevois::InputFrame const &inframe, std::string const &fpscpu, unsigned short winw=0, unsigned short winh=0)
void drawLine(float x1, float y1, float x2, float y2, ImU32 col=IM_COL32(128, 255, 128, 255))
void itext(char const *txt, ImU32 const &col=IM_COL32_BLACK_TRANS, int line=-1)
ImVec2 d2i(ImVec2 p, char const *name=nullptr)
void drawImage(char const *name, RawImage const &img, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool isoverlay=false)
unsigned int fmt
unsigned int width
unsigned int height
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
std::string const & stop(double *seconds)
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
cv::Mat convertToCvBGR(RawImage const &src)
void pasteBGRtoYUYV(cv::Mat const &src, RawImage &dst, int dx, int dy)
std::future< std::invoke_result_t< std::decay_t< Function >, std::decay_t< Args >... > > async(Function &&f, Args &&... args)
std::vector< std::string > split(std::string const &input, std::string const &regex="\\s+")
unsigned short constexpr White