JeVois  1.20
JeVois Smart Embedded Machine Vision Toolkit
Share this page:
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2021 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See and for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - - -
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
18 #pragma once
21 #include <opencv2/core/core.hpp>
23 namespace jevois
24 {
25  namespace dnn
26  {
27  //! Pre-Processor for neural network pipeline
28  /*! This is the first step in a deep neural network processing Pipeline.
30  This pre-processor works as follows. As an example, assume a 1024x576 camera input frame and a 224x224 neural
31  network input:
33  - If camera frame is not RGB or BGR, convert to that (e.g., YUYV to RGB or BGR)
35  - If \p letterbox is specified, fit the largest possible rectangle, with the aspect ratio of the network input,
36  within the camera frame. For example, 224x224 is square so that would compute a 576x576 square box around the
37  center of the camera frame. Otherwise, use the whole camera frame.
39  - Crop that rectangle and resize it to network input size (possibly with stretching if \p letterbox was off)
41  - Swap BGR/RGB if needed (combination of \p rgb parameter and color order in the received camera frame)
43  - Most accurate but also slowest path (may be replaced by an optimized path below):
44  + convert pixel data to float32
45  + subtract \p mean if not zero
46  + divide by \p stdev if not 1
47  + multiply by \p scale if not 1. At this point, values will typically be in [0..1] or [-1..1]
48  + quantize if needed. For example, if the network expects uint8 with asymmetric affine quantization
49  NHWC:8U:1x224x224x3:AA:0.0078125:128, divide by quantizer scale (here, 0.0078125, so that multiplies the
50  pixel values by 128) then add zero point (here, 128). The goal here is to use as much of the 8-bit dynamic
51  range as possible. What the network wants (specified by its intensors parameter) is determined during the
52  network quantization process.
53  + convert to desired data type for the network (e.g., uint8)
54  + possibly convert from packed RGB from the camera (NHWC) to planar (NCHW)
55  + convert shape to 4D, with batch size (N) always 1
57  - Because for uint8 (and also signed int8 and dynamic-fixed-point) this leads to nearly a no-op (first transform
58  from native camera range [0..255] to, say, [0..1], then, during quantization, stretch back to [0..255]), fast
59  paths are implemented for these special cases (e.g., uint8 camera input to quantized asymmetric affine uint8
60  network input). For dynamic fixed point, the fast path uses fast bit-shifting operations; for uint8
61  asymmetric affine, it is sometimes a no-op.
63  You can see these steps in the JeVois-Pro GUI (in the window that shows network processing details) by
64  enabling pre-processor parameter \p details
66  \ingroup dnn */
68  public jevois::Parameter<preprocessor::letterbox, preprocessor::scale, preprocessor::mean,
69  preprocessor::stdev, preprocessor::interp, preprocessor::numin>
70  {
71  public:
72  //! Inherited constructor ok
75  //! Destructor
76  virtual ~PreProcessorBlob();
78  //! Freeze/unfreeze parameters that users should not change while running
79  void freeze(bool doit) override;
81  protected:
82  //! Extract blobs from input image
83  std::vector<cv::Mat> process(cv::Mat const & img, bool swaprb, std::vector<vsi_nn_tensor_attr_t> const & attrs,
84  std::vector<cv::Rect> & crops) override;
86  //! Report what happened in last process() to console/output video/GUI
87  void report(jevois::StdModule * mod, jevois::RawImage * outimg = nullptr,
88  jevois::OptGUIhelper * helper = nullptr, bool overlay = true, bool idle = false) override;
90  std::vector<std::string> itsInfo;
91  };
93  } // namespace dnn
94 } // namespace jevois
std::vector< cv::Mat > process(cv::Mat const &img, bool swaprb, std::vector< vsi_nn_tensor_attr_t > const &attrs, std::vector< cv::Rect > &crops) override
Extract blobs from input image.
Definition: PreProcessorBlob.C:42
void report(jevois::StdModule *mod, jevois::RawImage *outimg=nullptr, jevois::OptGUIhelper *helper=nullptr, bool overlay=true, bool idle=false) override
Report what happened in last process() to console/output video/GUI.
Definition: PreProcessorBlob.C:379
virtual ~PreProcessorBlob()
Definition: PreProcessorBlob.C:32
A raw image as coming from a V4L2 Camera and/or being sent out to a USB Gadget.
Definition: RawImage.H:110
Helper class to assist modules in creating graphical and GUI elements.
Definition: GUIhelper.H:128
Pre-Processor for neural network pipeline.
Definition: PreProcessorBlob.H:67
Definition: Concepts.dox:1
Pre-Processor for neural network pipeline.
Definition: PreProcessor.H:108
std::vector< std::string > itsInfo
Definition: PreProcessorBlob.H:90
Base class for a module that supports standardized serial messages.
Definition: Module.H:232
PreProcessor(std::string const &instance)
Definition: PreProcessor.C:26
void freeze(bool doit) override
Freeze/unfreeze parameters that users should not change while running.
Definition: PreProcessorBlob.C:36