JeVois  1.20
JeVois Smart Embedded Machine Vision Toolkit
Share this page:
jevois::dnn::PreProcessorBlob Class Reference

#include <jevois/DNN/PreProcessorBlob.H>

Pre-Processor for neural network pipeline.

This is the first step in a deep neural network processing Pipeline.

This pre-processor works as follows. As an example, assume a 1024x576 camera input frame and a 224x224 neural network input:

  • If camera frame is not RGB or BGR, convert to that (e.g., YUYV to RGB or BGR)
  • If letterbox is specified, fit the largest possible rectangle, with the aspect ratio of the network input, within the camera frame. For example, 224x224 is square so that would compute a 576x576 square box around the center of the camera frame. Otherwise, use the whole camera frame.
  • Crop that rectangle and resize it to network input size (possibly with stretching if letterbox was off)
  • Swap BGR/RGB if needed (combination of rgb parameter and color order in the received camera frame)
  • Most accurate but also slowest path (may be replaced by an optimized path below):
    • convert pixel data to float32
    • subtract mean if not zero
    • divide by stdev if not 1
    • multiply by scale if not 1. At this point, values will typically be in [0..1] or [-1..1]
    • quantize if needed. For example, if the network expects uint8 with asymmetric affine quantization NHWC:8U:1x224x224x3:AA:0.0078125:128, divide by quantizer scale (here, 0.0078125, so that multiplies the pixel values by 128) then add zero point (here, 128). The goal here is to use as much of the 8-bit dynamic range as possible. What the network wants (specified by its intensors parameter) is determined during the network quantization process.
    • convert to desired data type for the network (e.g., uint8)
    • possibly convert from packed RGB from the camera (NHWC) to planar (NCHW)
    • convert shape to 4D, with batch size (N) always 1
  • Because for uint8 (and also signed int8 and dynamic-fixed-point) this leads to nearly a no-op (first transform from native camera range [0..255] to, say, [0..1], then, during quantization, stretch back to [0..255]), fast paths are implemented for these special cases (e.g., uint8 camera input to quantized asymmetric affine uint8 network input). For dynamic fixed point, the fast path uses fast bit-shifting operations; for uint8 asymmetric affine, it is sometimes a no-op.

    You can see these steps in the JeVois-Pro GUI (in the window that shows network processing details) by enabling pre-processor parameter details

Definition at line 67 of file PreProcessorBlob.H.

Inheritance diagram for jevois::dnn::PreProcessorBlob:
Collaboration diagram for jevois::dnn::PreProcessorBlob:

Public Member Functions

virtual ~PreProcessorBlob ()
 Destructor. More...
 
void freeze (bool doit) override
 Freeze/unfreeze parameters that users should not change while running. More...
 
 PreProcessor (std::string const &instance)
 Inherited constructor ok. More...
 
- Public Member Functions inherited from jevois::dnn::PreProcessor
 PreProcessor (std::string const &instance)
 Constructor. More...
 
virtual ~PreProcessor ()
 Destructor. More...
 
std::vector< cv::Mat > process (jevois::RawImage const &img, std::vector< vsi_nn_tensor_attr_t > const &attrs)
 Extract blobs from input image. More...
 
virtual void sendreport (jevois::StdModule *mod, jevois::RawImage *outimg=nullptr, jevois::OptGUIhelper *helper=nullptr, bool overlay=true, bool idle=false)
 Report what happened in last process() to console/output video/GUI. More...
 
const cv::Size & imagesize () const
 Access the last processed image size. More...
 
const std::vector< cv::Mat > & blobs () const
 Access the last computed blobs (or empty if process() has not yet been called) More...
 
cv::Size blobsize (size_t num) const
 Access the width and height of a given blob, accounting for NCHW or NHWC. More...
 
void b2i (float &x, float &y, size_t blobnum=0)
 Convert coordinates from blob back to original image. More...
 
void b2i (float &x, float &y, cv::Size const &bsiz, bool letterboxed)
 Convert coordinates from blob back to original image, given a known blob size. More...
 
cv::Rect getUnscaledCropRect (size_t blobnum=0)
 Get unscaled crop rectangle in image coordinates. More...
 
void getUnscaledCropRect (size_t blobnum, int &tlx, int &tly, int &brx, int &bry)
 Get unscaled crop rectangle in image coordinates. More...
 
void i2b (float &x, float &y, size_t blobnum=0)
 Convert coordinates from image to blob. More...
 
void i2b (float &x, float &y, cv::Size const &bsiz, bool letterboxed)
 Convert coordinates from image to blob. More...
 
std::shared_ptr< PreProcessorForPythongetPreProcForPy () const
 Get a pointer to our python-friendly interface. More...
 
- Public Member Functions inherited from jevois::Component
 Component (std::string const &instance)
 Constructor. More...
 
virtual ~Component ()
 Virtual destructor for safe inheritance. More...
 
template<class Comp , typename... Args>
std::shared_ptr< Comp > addSubComponent (std::string const &instance, Args &&...args)
 Pseudo-constructor: construct and add another component as a subcomponent of this one. More...
 
template<class Comp >
void removeSubComponent (std::shared_ptr< Comp > &component)
 Remove a sub-Component from this Component, by shared_ptr. More...
 
void removeSubComponent (std::string const &instance, bool warnIfNotFound=true)
 Remove a sub-Component from this Component, by instance name. More...
 
template<class Comp = jevois::Component>
std::shared_ptr< Comp > getSubComponent (std::string const &instance) const
 Get a sub-component by instance name. More...
 
bool isTopLevel () const
 Returns true if this component is top-level, i.e., its parent is jevois::Manager. More...
 
Engineengine () const
 Get a handle to our Engine, or throw if we do not have an Engine as root ancestor. More...
 
bool initialized () const
 Has this component been initialized yet? More...
 
const std::string & className () const
 The class name of this component. More...
 
const std::string & instanceName () const
 The instance name of this component. More...
 
template<typename T >
std::vector< std::string > setParamVal (std::string const &paramdescriptor, T const &val)
 Set a parameter value. More...
 
template<typename T >
void setParamValUnique (std::string const &paramdescriptor, T const &val)
 Set a parameter value, simple version assuming only one parameter match. More...
 
template<typename T >
std::vector< std::pair< std::string, T > > getParamVal (std::string const &paramdescriptor) const
 Get parameter(s) value(s) by descriptor. More...
 
template<typename T >
getParamValUnique (std::string const &paramdescriptor) const
 Get a parameter value, simple version assuming only one parameter match. More...
 
std::vector< std::string > setParamString (std::string const &paramdescriptor, std::string const &val)
 Set a parameter value, by string. More...
 
void setParamStringUnique (std::string const &paramdescriptor, std::string const &val)
 Set a parameter value by string, simple version assuming only one parameter match. More...
 
std::vector< std::pair< std::string, std::string > > getParamString (std::string const &paramdescriptor) const
 Get a parameter value, by string. More...
 
std::string getParamStringUnique (std::string const &paramdescriptor) const
 Get a parameter value by string, simple version assuming only one parameter match. More...
 
void freezeParam (std::string const &paramdescriptor)
 Freeze a parameter, by name, see ParameterBase::freeze() More...
 
void unFreezeParam (std::string const &paramdescriptor)
 Unfreeze a parameter, by name, see ParameterBase::unFreeze() More...
 
void freezeAllParams ()
 Freeze all parameters. More...
 
void unFreezeAllParams ()
 Unfreeze all parameters. More...
 
std::string descriptor () const
 Get our full descriptor (including all parents) as [Instancename]:[...]:[...]. More...
 
void setParamsFromFile (std::string const &filename)
 Set some parameters from a file. More...
 
std::istream & setParamsFromStream (std::istream &is, std::string const &absfile)
 Set some parameters from an open stream. More...
 
virtual void paramInfo (std::shared_ptr< UserInterface > s, std::map< std::string, std::string > &categs, bool skipFrozen, std::string const &cname="", std::string const &pfx="")
 Get machine-oriented descriptions of all parameters. More...
 
void foreachParam (std::function< void(std::string const &compname, ParameterBase *p)> func, std::string const &cname="")
 Run a function on every param we hold. More...
 
template<typename T >
std::shared_ptr< DynamicParameter< T > > addDynamicParameter (std::string const &name, std::string const &description, T const &defaultValue, ParameterCategory const &category)
 Add a new parameter after the Component has already been constructed. More...
 
template<typename T , template< typename > class ValidValuesSpec>
std::shared_ptr< DynamicParameter< T > > addDynamicParameter (std::string const &name, std::string const &description, T const &defaultValue, ValidValuesSpec< T > const &validValuesSpec, ParameterCategory const &category)
 Add a new parameter after the Component has already been constructed. More...
 
template<typename T >
void setDynamicParameterCallback (std::string const &name, std::function< void(T const &)> cb, bool callnow=true)
 Register a callback with a previously created dynamic parameter. More...
 
void removeDynamicParameter (std::string const &name)
 Remove a previously added dynamic parameter. More...
 
void setPath (std::string const &path)
 Assign a filesystem path to this component. More...
 
std::filesystem::path absolutePath (std::filesystem::path const &path="")
 If given path is relative (not starting with /), prepend the Component path to it. More...
 
- Public Member Functions inherited from jevois::ParameterRegistry
virtual ~ParameterRegistry ()
 Virtual destructor for safe inheritance. More...
 

Protected Member Functions

std::vector< cv::Mat > process (cv::Mat const &img, bool swaprb, std::vector< vsi_nn_tensor_attr_t > const &attrs, std::vector< cv::Rect > &crops) override
 Extract blobs from input image. More...
 
void report (jevois::StdModule *mod, jevois::RawImage *outimg=nullptr, jevois::OptGUIhelper *helper=nullptr, bool overlay=true, bool idle=false) override
 Report what happened in last process() to console/output video/GUI. More...
 
- Protected Member Functions inherited from jevois::Component
virtual void preInit ()
 Called before all sub-Components are init()ed. More...
 
virtual void postInit ()
 Called after all sub-Components are init()ed. More...
 
virtual void preUninit ()
 Called before all sub-Components are uninit()ed. More...
 
virtual void postUninit ()
 Called after all sub-Components are uninit()ed. More...
 
- Protected Member Functions inherited from jevois::ParameterRegistry
void addParameter (ParameterBase *const param)
 The Parameter class uses this method to register itself on construction with its owning Component. More...
 
void removeParameter (ParameterBase *const param)
 The Parameter class uses this method to un-register itself on destruction with its owning Component. More...
 
void callbackInitCall ()
 For all parameters that have a callback which has never been called, call it with the default param value. More...
 

Protected Attributes

std::vector< std::string > itsInfo
 

Constructor & Destructor Documentation

◆ ~PreProcessorBlob()

jevois::dnn::PreProcessorBlob::~PreProcessorBlob ( )
virtual

Destructor.

Definition at line 32 of file PreProcessorBlob.C.

Member Function Documentation

◆ freeze()

void jevois::dnn::PreProcessorBlob::freeze ( bool  doit)
overridevirtual

Freeze/unfreeze parameters that users should not change while running.

Implements jevois::dnn::PreProcessor.

Definition at line 36 of file PreProcessorBlob.C.

◆ PreProcessor()

jevois::dnn::PreProcessor::PreProcessor

Inherited constructor ok.

Definition at line 26 of file PreProcessor.C.

◆ process()

std::vector< cv::Mat > jevois::dnn::PreProcessorBlob::process ( cv::Mat const &  img,
bool  swaprb,
std::vector< vsi_nn_tensor_attr_t > const &  attrs,
std::vector< cv::Rect > &  crops 
)
overrideprotectedvirtual

◆ report()

void jevois::dnn::PreProcessorBlob::report ( jevois::StdModule mod,
jevois::RawImage outimg = nullptr,
jevois::OptGUIhelper helper = nullptr,
bool  overlay = true,
bool  idle = false 
)
overrideprotectedvirtual

Report what happened in last process() to console/output video/GUI.

Implements jevois::dnn::PreProcessor.

Definition at line 379 of file PreProcessorBlob.C.

Member Data Documentation

◆ itsInfo

std::vector<std::string> jevois::dnn::PreProcessorBlob::itsInfo
protected

Definition at line 90 of file PreProcessorBlob.H.


The documentation for this class was generated from the following files: