JeVois  1.21
JeVois Smart Embedded Machine Vision Toolkit
Share this page:
Loading...
Searching...
No Matches
Tensor/Neural Processing networks

Classes and utilities to provide abstraction to deep neural networks. Provides interfacing to OpenCV backends (CPU, OpenCL), tensor processing units (TPU) such as Coral Edge TPU and neural processing units (NPU) such as Amlogic A311D NPU.

Collaboration diagram for Tensor/Neural Processing networks:

Classes

class  jevois::dnn::Network
 Abstract class to represent a neural network. More...
 
class  jevois::dnn::NetworkHailo
 Wrapper around an DNN neural network running on the Hailo8 neural accelerator. More...
 
class  jevois::dnn::NetworkNPU
 Wrapper around a DNN neural network running on Amlogic A311D NPU accelerator (Verisilicon) More...
 
class  jevois::dnn::NetworkONNX
 Wrapper around an ONNX-Runtime neural network. More...
 
class  jevois::dnn::NetworkOpenCV
 Wrapper around an OpenCV DNN neural network. More...
 
class  jevois::dnn::NetworkPython
 Wrapper around an DNN neural network invoked through python. More...
 
class  jevois::dnn::NetworkTPU
 Wrapper around a Coral TPU neural network. More...
 
class  jevois::dnn::Pipeline
 Neural processing pipeline. More...
 
class  jevois::dnn::PostProcessor
 Post-Processor for neural network pipeline. More...
 
class  jevois::dnn::PostProcessorClassify
 Post-Processor for neural network pipeline. More...
 
class  jevois::dnn::PostProcessorDetect
 Post-Processor for neural network pipeline. More...
 
class  jevois::dnn::PostProcessorPython
 Post-Processor for neural network pipeline. More...
 
class  jevois::dnn::PostProcessorSegment
 Post-Processor for neural network pipeline. More...
 
class  jevois::dnn::PostProcessorStub
 Post-Processor for neural network pipeline. More...
 
class  jevois::dnn::PostProcessorYuNet
 Post-Processor for YuNet face landmarks detector. More...
 
class  jevois::dnn::PreProcessor
 Pre-Processor for neural network pipeline. More...
 
class  jevois::dnn::PreProcessorBlob
 Pre-Processor for neural network pipeline. More...
 
class  jevois::dnn::PreProcessorPython
 Pre-Processor for neural network pipeline written in python. More...
 

Modules

 DNN-related processors written in python
 

Functions

std::map< int, std::string > jevois::dnn::readLabelsFile (std::string const &fname)
 Read a label file.
 
std::string jevois::dnn::getLabel (std::map< int, std::string > const &labels, int id)
 Get a label from an id.
 
int jevois::dnn::stringToRGBA (std::string const &label, unsigned char alpha=128)
 Compute a color from a label name.
 
void jevois::dnn::topK (float const *pfProb, float *pfMaxProb, uint32_t *pMaxClass, uint32_t outputCount, uint32_t topNum)
 Get top-k entries and their indices.
 
std::string jevois::dnn::shapestr (cv::Mat const &m)
 Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional cv::Mat with data type TYPE.
 
std::string jevois::dnn::shapestr (TfLiteTensor const *t)
 Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional TfLiteTensor with data type TYPE.
 
std::string jevois::dnn::shapestr (vsi_nn_tensor_attr_t const &attr)
 Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional NPU tensor with data type TYPE.
 
std::vector< size_t > jevois::dnn::strshape (std::string const &str)
 Get a vector of size_t from a string containing AxBxC...
 
int jevois::dnn::tf2cv (TfLiteType t)
 Convert from TensorFlow data type to OpenCV.
 
vsi_nn_type_e jevois::dnn::tf2vsi (TfLiteType t)
 Convert from TensorFlow data type to vsi_nn.
 
int jevois::dnn::vsi2cv (vsi_nn_type_e t)
 Convert from NPU data type to OpenCV.
 
void jevois::dnn::clamp (cv::Rect &r, int width, int height)
 Clamp a rectangle to within given image width and height.
 
void jevois::dnn::clamp (cv::Rect2f &r, float width, float height)
 Clamp a rectangle to within given image width and height.
 
std::vector< vsi_nn_tensor_attr_t > jevois::dnn::parseTensorSpecs (std::string const &specs)
 Parse tensor specification.
 
cv::Mat jevois::dnn::attrmat (vsi_nn_tensor_attr_t const &attr, void *dataptr=nullptr)
 Construct a cv::Mat from attr and possibly data pointer.
 
std::vector< int > jevois::dnn::attrdims (vsi_nn_tensor_attr_t const &attr)
 Get a tensor dims as a vector of int, useful to construct a matching cv::Mat.
 
cv::Size jevois::dnn::attrsize (vsi_nn_tensor_attr_t const &attr)
 Get a tensor's (width, height) size in cv::Size format, skipping over other dimensions.
 
std::string jevois::dnn::attrstr (vsi_nn_tensor_attr_t const &attr)
 Get a string describing the specs of a tensor, including quantification specs (not provided by shapestr())
 
bool jevois::dnn::attrmatch (vsi_nn_tensor_attr_t const &attr, cv::Mat const &blob)
 Check that a cv::Mat blob matches exactly the spec of an attr.
 
vsi_nn_tensor_attr_t jevois::dnn::tensorattr (TfLiteTensor const *t)
 Get tensor shape and type attributes for a TensorFlow Lite tensor.
 
size_t jevois::dnn::softmax (float const *input, size_t const n, size_t const stride, float const fac, float *output, bool maxonly)
 Apply softmax to a float vector.
 
cv::Mat jevois::dnn::quantize (cv::Mat const &m, vsi_nn_tensor_attr_t const &attr)
 Quantize from float32 to fixed-point according to the quantization spec in attr.
 
cv::Mat jevois::dnn::dequantize (cv::Mat const &m, vsi_nn_tensor_attr_t const &attr)
 Dequantize an output to float32 according to the quantization spec in attr.
 
size_t jevois::dnn::effectiveDims (cv::Mat const &m)
 Returns the number of non-unit dims in a cv::Mat.
 
cv::Mat jevois::dnn::concatenate (std::vector< cv::Mat > const &tensors, int axis)
 Concatenate several tensors into one.
 
std::string jevois::dnn::shapestr (hailo_vstream_info_t const &vi)
 Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional Hailo tensor with data type TYPE.
 
vsi_nn_tensor_attr_t jevois::dnn::tensorattr (hailo_vstream_info_t const &vi)
 Get tensor shape and type attributes for a Hailo tensor.
 
vsi_nn_type_e jevois::dnn::hailo2vsi (hailo_format_type_t t)
 Convert from Hailo data type to vsi_nn.
 
vsi_nn_type_e jevois::dnn::onnx2vsi (ONNXTensorElementDataType t)
 Convert from ONNX-Runtime data type to vsi_nn.
 
std::string jevois::dnn::shapestr (Ort::ConstTensorTypeAndShapeInfo const &ti)
 Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional ONNX tensor with data type TYPE.
 
vsi_nn_tensor_attr_t jevois::dnn::tensorattr (Ort::ConstTensorTypeAndShapeInfo const &ti)
 Get tensor shape and type attributes for an ONNX-runtime tensor.
 

Function Documentation

◆ attrdims()

std::vector< int > jevois::dnn::attrdims ( vsi_nn_tensor_attr_t const &  attr)

Get a tensor dims as a vector of int, useful to construct a matching cv::Mat.

Definition at line 478 of file Utils.C.

Referenced by jevois::dnn::attrmat(), jevois::dnn::NetworkHailo::load(), jevois::dnn::Network::process(), and jevois::dnn::quantize().

◆ attrmat()

cv::Mat jevois::dnn::attrmat ( vsi_nn_tensor_attr_t const &  attr,
void *  dataptr = nullptr 
)

Construct a cv::Mat from attr and possibly data pointer.

If dataptr is nullptr, new memory will be allocated for the cv::Mat. Caller must ensure data outlives the cv::Mat, and is responsible for eventually de-allocating the data. Usually, with non-null dataptr, this is only to be used as a temporary re-casting, e.g., to recast a received tensor into a Mat before dequantizing it, then forgetting about that Mat.

Definition at line 471 of file Utils.C.

References jevois::dnn::attrdims(), and jevois::dnn::vsi2cv().

Referenced by jevois::dnn::NetworkNPU::doprocess(), jevois::dnn::NetworkONNX::doprocess(), jevois::dnn::NetworkHailo::load(), and jevois::dnn::Network::process().

◆ attrmatch()

bool jevois::dnn::attrmatch ( vsi_nn_tensor_attr_t const &  attr,
cv::Mat const &  blob 
)

Check that a cv::Mat blob matches exactly the spec of an attr.

Definition at line 704 of file Utils.C.

References jevois::dnn::vsi2cv().

Referenced by jevois::dnn::dequantize(), jevois::dnn::NetworkHailo::doprocess(), and jevois::dnn::NetworkNPU::doprocess().

◆ attrsize()

cv::Size jevois::dnn::attrsize ( vsi_nn_tensor_attr_t const &  attr)

Get a tensor's (width, height) size in cv::Size format, skipping over other dimensions.

Definition at line 487 of file Utils.C.

References jevois::dnn::attrstr().

Referenced by jevois::dnn::PreProcessor::blobsize(), and jevois::dnn::PreProcessorBlob::process().

◆ attrstr()

std::string jevois::dnn::attrstr ( vsi_nn_tensor_attr_t const &  attr)

◆ clamp() [1/2]

void jevois::dnn::clamp ( cv::Rect &  r,
int  width,
int  height 
)

Clamp a rectangle to within given image width and height.

Definition at line 367 of file Utils.C.

Referenced by jevois::dnn::PostProcessorDetect::process(), and jevois::dnn::PostProcessorYuNet::process().

◆ clamp() [2/2]

void jevois::dnn::clamp ( cv::Rect2f &  r,
float  width,
float  height 
)

Clamp a rectangle to within given image width and height.

Definition at line 377 of file Utils.C.

◆ concatenate()

cv::Mat jevois::dnn::concatenate ( std::vector< cv::Mat > const &  tensors,
int  axis 
)

Concatenate several tensors into one.

Axis may be positive starting at 0 for the first dimension (when reading dims from left to right), or negative starting at -1 for the last dimension. For example, for a 10x20x30 tensor, axis 0 has size 10 and is also axis -3, axis 1 has size 20 and is also axis -2, and axis 2 has size 30 and is also axis -1. The input tensors must all have the same number of dimensions, same pixel type, and sizes must match for all dimensions except the one that is being concatenated.

Definition at line 842 of file Utils.C.

References jevois::cvBytesPerPix(), jevois::cvtypestr(), and LFATAL.

◆ dequantize()

cv::Mat jevois::dnn::dequantize ( cv::Mat const &  m,
vsi_nn_tensor_attr_t const &  attr 
)

Dequantize an output to float32 according to the quantization spec in attr.

attr should have the type and quantization details of m, returned tensor is float32

Definition at line 793 of file Utils.C.

References jevois::dnn::attrmatch(), LFATAL, and jevois::dnn::shapestr().

Referenced by jevois::dnn::NetworkHailo::doprocess(), and jevois::dnn::NetworkNPU::doprocess().

◆ effectiveDims()

size_t jevois::dnn::effectiveDims ( cv::Mat const &  m)

Returns the number of non-unit dims in a cv::Mat.

For example, returns 2 for a 4D Mat with size 1x1x224x224, since it effectively is a 224x224 2D array

Definition at line 832 of file Utils.C.

◆ getLabel()

std::string jevois::dnn::getLabel ( std::map< int, std::string > const &  labels,
int  id 
)

Get a label from an id.

If no entry is found in the map, return the id as a string.

Definition at line 68 of file Utils.C.

Referenced by jevois::dnn::PostProcessorClassify::process(), and jevois::dnn::PostProcessorDetect::process().

◆ hailo2vsi()

vsi_nn_type_e jevois::dnn::hailo2vsi ( hailo_format_type_t  t)

Convert from Hailo data type to vsi_nn.

Definition at line 353 of file Utils.C.

Referenced by jevois::dnn::tensorattr().

◆ onnx2vsi()

vsi_nn_type_e jevois::dnn::onnx2vsi ( ONNXTensorElementDataType  t)

Convert from ONNX-Runtime data type to vsi_nn.

Definition at line 208 of file Utils.C.

Referenced by jevois::dnn::tensorattr().

◆ parseTensorSpecs()

std::vector< vsi_nn_tensor_attr_t > jevois::dnn::parseTensorSpecs ( std::string const &  specs)

◆ quantize()

cv::Mat jevois::dnn::quantize ( cv::Mat const &  m,
vsi_nn_tensor_attr_t const &  attr 
)

Quantize from float32 to fixed-point according to the quantization spec in attr.

m should be float32, typically normalized to [0..1[ or [-1..1[ already. attr is the desired quantized type and method (DFP, AA, etc)

Definition at line 718 of file Utils.C.

References jevois::dnn::attrdims(), LFATAL, jevois::dnn::shapestr(), and jevois::dnn::vsi2cv().

Referenced by jevois::dnn::PreProcessorBlob::process().

◆ readLabelsFile()

std::map< int, std::string > jevois::dnn::readLabelsFile ( std::string const &  fname)

Read a label file.

Two formats are allowed: one class name per line, or one class number followed by one class name per file.

Definition at line 25 of file Utils.C.

References LERROR, LFATAL, LINFO, and jevois::replaceStringAll().

Referenced by jevois::dnn::PostProcessorClassify::onParamChange(), and jevois::dnn::PostProcessorDetect::onParamChange().

◆ shapestr() [1/5]

◆ shapestr() [2/5]

std::string jevois::dnn::shapestr ( hailo_vstream_info_t const &  vi)

Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional Hailo tensor with data type TYPE.

Definition at line 252 of file Utils.C.

References jevois::dnn::shapestr(), and jevois::dnn::tensorattr().

◆ shapestr() [3/5]

std::string jevois::dnn::shapestr ( Ort::ConstTensorTypeAndShapeInfo const &  ti)

Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional ONNX tensor with data type TYPE.

Definition at line 174 of file Utils.C.

◆ shapestr() [4/5]

std::string jevois::dnn::shapestr ( TfLiteTensor const *  t)

Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional TfLiteTensor with data type TYPE.

Definition at line 115 of file Utils.C.

◆ shapestr() [5/5]

std::string jevois::dnn::shapestr ( vsi_nn_tensor_attr_t const &  attr)

Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional NPU tensor with data type TYPE.

Definition at line 144 of file Utils.C.

◆ softmax()

size_t jevois::dnn::softmax ( float const *  input,
size_t const  n,
size_t const  stride,
float const  fac,
float *  output,
bool  maxonly 
)

Apply softmax to a float vector.

n is the number of elements to process, stride is the increment in the arrays from one element to the next. So the arrays should have size n * stride. Returns the index in [0..n*stride[ of the highest scoring element. If maxonly is true, only output[returned index] is valid.

Definition at line 668 of file Utils.C.

References LFATAL.

Referenced by jevois::dnn::PostProcessorClassify::process().

◆ stringToRGBA()

int jevois::dnn::stringToRGBA ( std::string const &  label,
unsigned char  alpha = 128 
)

Compute a color from a label name.

Definition at line 76 of file Utils.C.

Referenced by jevois::dnn::PostProcessorDetect::report().

◆ strshape()

std::vector< size_t > jevois::dnn::strshape ( std::string const &  str)

Get a vector of size_t from a string containing AxBxC...

Definition at line 277 of file Utils.C.

References jevois::split().

Referenced by jevois::dnn::parseTensorSpecs(), and jevois::dnn::Network::process().

◆ tensorattr() [1/3]

vsi_nn_tensor_attr_t jevois::dnn::tensorattr ( hailo_vstream_info_t const &  vi)

Get tensor shape and type attributes for a Hailo tensor.

Definition at line 602 of file Utils.C.

References jevois::dnn::hailo2vsi().

◆ tensorattr() [2/3]

vsi_nn_tensor_attr_t jevois::dnn::tensorattr ( Ort::ConstTensorTypeAndShapeInfo const &  ti)

Get tensor shape and type attributes for an ONNX-runtime tensor.

Definition at line 234 of file Utils.C.

References jevois::dnn::onnx2vsi().

◆ tensorattr() [3/3]

vsi_nn_tensor_attr_t jevois::dnn::tensorattr ( TfLiteTensor const *  t)

Get tensor shape and type attributes for a TensorFlow Lite tensor.

Definition at line 563 of file Utils.C.

References LFATAL, and jevois::dnn::tf2vsi().

Referenced by jevois::dnn::NetworkTPU::inputShapes(), jevois::dnn::NetworkHailo::load(), jevois::dnn::NetworkONNX::load(), jevois::dnn::NetworkTPU::outputShapes(), and jevois::dnn::shapestr().

◆ tf2cv()

int jevois::dnn::tf2cv ( TfLiteType  t)

Convert from TensorFlow data type to OpenCV.

Definition at line 286 of file Utils.C.

Referenced by jevois::dnn::NetworkTPU::doprocess().

◆ tf2vsi()

vsi_nn_type_e jevois::dnn::tf2vsi ( TfLiteType  t)

Convert from TensorFlow data type to vsi_nn.

Definition at line 330 of file Utils.C.

Referenced by jevois::dnn::tensorattr().

◆ topK()

void jevois::dnn::topK ( float const *  pfProb,
float *  pfMaxProb,
uint32_t *  pMaxClass,
uint32_t  outputCount,
uint32_t  topNum 
)

Get top-k entries and their indices.

Definition at line 85 of file Utils.C.

Referenced by jevois::dnn::PostProcessorClassify::process().

◆ vsi2cv()

int jevois::dnn::vsi2cv ( vsi_nn_type_e  t)