Classes and utilities to provide abstraction to deep neural networks. Provides interfacing to OpenCV backends (CPU, OpenCL), tensor processing units (TPU) such as Coral Edge TPU and neural processing units (NPU) such as Amlogic A311D NPU.

Collaboration diagram for Tensor/Neural Processing networks:

Classes
class	jevois::dnn::Network
	Abstract class to represent a neural network. More...

class	jevois::dnn::NetworkHailo
	Wrapper around an DNN neural network running on the Hailo8 neural accelerator. More...

class	jevois::dnn::NetworkNPU
	Wrapper around a DNN neural network running on Amlogic A311D NPU accelerator (Verisilicon) More...

class	jevois::dnn::NetworkONNX
	Wrapper around an ONNX-Runtime neural network. More...

class	jevois::dnn::NetworkOpenCV
	Wrapper around an OpenCV DNN neural network. More...

class	jevois::dnn::NetworkPython
	Wrapper around an DNN neural network invoked through python. More...

class	jevois::dnn::NetworkTPU
	Wrapper around a Coral TPU neural network. More...

class	jevois::dnn::Pipeline
	Neural processing pipeline. More...

class	jevois::dnn::PostProcessor
	Post-Processor for neural network pipeline. More...

class	jevois::dnn::PostProcessorClassify
	Post-Processor for neural network pipeline. More...

class	jevois::dnn::PostProcessorDetect
	Post-Processor for neural network pipeline. More...

class	jevois::dnn::PostProcessorPython
	Post-Processor for neural network pipeline. More...

class	jevois::dnn::PostProcessorSegment
	Post-Processor for neural network pipeline. More...

class	jevois::dnn::PostProcessorStub
	Post-Processor for neural network pipeline. More...

class	jevois::dnn::PostProcessorYuNet
	Post-Processor for YuNet face landmarks detector. More...

class	jevois::dnn::PreProcessor
	Pre-Processor for neural network pipeline. More...

class	jevois::dnn::PreProcessorBlob
	Pre-Processor for neural network pipeline. More...

class	jevois::dnn::PreProcessorPython
	Pre-Processor for neural network pipeline written in python. More...

Modules
	DNN-related processors written in python
	In addition to writing DNN pre/net/post processors in C++, JeVois supports writing them in Python.

Functions
std::map< int, std::string >	jevois::dnn::readLabelsFile (std::string const &fname)
	Read a label file. More...

std::string	jevois::dnn::getLabel (std::map< int, std::string > const &labels, int id)
	Get a label from an id. More...

int	jevois::dnn::stringToRGBA (std::string const &label, unsigned char alpha=128)
	Compute a color from a label name. More...

void	jevois::dnn::topK (float const pfProb, float pfMaxProb, uint32_t *pMaxClass, uint32_t outputCount, uint32_t topNum)
	Get top-k entries and their indices. More...

std::string	jevois::dnn::shapestr (cv::Mat const &m)
	Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional cv::Mat with data type TYPE. More...

std::string	jevois::dnn::shapestr (TfLiteTensor const *t)
	Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional TfLiteTensor with data type TYPE. More...

std::string	jevois::dnn::shapestr (vsi_nn_tensor_attr_t const &attr)
	Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional NPU tensor with data type TYPE. More...

std::vector< size_t >	jevois::dnn::strshape (std::string const &str)
	Get a vector of size_t from a string containing AxBxC... More...

int	jevois::dnn::tf2cv (TfLiteType t)
	Convert from TensorFlow data type to OpenCV. More...

vsi_nn_type_e	jevois::dnn::tf2vsi (TfLiteType t)
	Convert from TensorFlow data type to vsi_nn. More...

int	jevois::dnn::vsi2cv (vsi_nn_type_e t)
	Convert from NPU data type to OpenCV. More...

void	jevois::dnn::clamp (cv::Rect &r, int width, int height)
	Clamp a rectangle to within given image width and height. More...

void	jevois::dnn::clamp (cv::Rect2f &r, float width, float height)
	Clamp a rectangle to within given image width and height. More...

std::vector< vsi_nn_tensor_attr_t >	jevois::dnn::parseTensorSpecs (std::string const &specs)
	Parse tensor specification. More...

cv::Mat	jevois::dnn::attrmat (vsi_nn_tensor_attr_t const &attr, void *dataptr=nullptr)
	Construct a cv::Mat from attr and possibly data pointer. More...

std::vector< int >	jevois::dnn::attrdims (vsi_nn_tensor_attr_t const &attr)
	Get a tensor dims as a vector of int, useful to construct a matching cv::Mat. More...

cv::Size	jevois::dnn::attrsize (vsi_nn_tensor_attr_t const &attr)
	Get a tensor's (width, height) size in cv::Size format, skipping over other dimensions. More...

std::string	jevois::dnn::attrstr (vsi_nn_tensor_attr_t const &attr)
	Get a string describing the specs of a tensor, including quantification specs (not provided by shapestr()) More...

bool	jevois::dnn::attrmatch (vsi_nn_tensor_attr_t const &attr, cv::Mat const &blob)
	Check that a cv::Mat blob matches exactly the spec of an attr. More...

vsi_nn_tensor_attr_t	jevois::dnn::tensorattr (TfLiteTensor const *t)
	Get tensor shape and type attributes for a TensorFlow Lite tensor. More...

size_t	jevois::dnn::softmax (float const input, size_t const n, size_t const stride, float const fac, float output, bool maxonly)
	Apply softmax to a float vector. More...

cv::Mat	jevois::dnn::quantize (cv::Mat const &m, vsi_nn_tensor_attr_t const &attr)
	Quantize from float32 to fixed-point according to the quantization spec in attr. More...

cv::Mat	jevois::dnn::dequantize (cv::Mat const &m, vsi_nn_tensor_attr_t const &attr)
	Dequantize an output to float32 according to the quantization spec in attr. More...

size_t	jevois::dnn::effectiveDims (cv::Mat const &m)
	Returns the number of non-unit dims in a cv::Mat. More...

std::string	jevois::dnn::shapestr (hailo_vstream_info_t const &vi)
	Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional Hailo tensor with data type TYPE. More...

vsi_nn_tensor_attr_t	jevois::dnn::tensorattr (hailo_vstream_info_t const &vi)
	Get tensor shape and type attributes for a Hailo tensor. More...

vsi_nn_type_e	jevois::dnn::hailo2vsi (hailo_format_type_t t)
	Convert from Hailo data type to vsi_nn. More...

vsi_nn_type_e	jevois::dnn::onnx2vsi (ONNXTensorElementDataType t)
	Convert from ONNX-Runtime data type to vsi_nn. More...

std::string	jevois::dnn::shapestr (Ort::ConstTensorTypeAndShapeInfo const &ti)
	Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional ONNX tensor with data type TYPE. More...

vsi_nn_tensor_attr_t	jevois::dnn::tensorattr (Ort::ConstTensorTypeAndShapeInfo const &ti)
	Get tensor shape and type attributes for an ONNX-runtime tensor. More...

Function Documentation

◆ attrdims()

std::vector< int > jevois::dnn::attrdims ( vsi_nn_tensor_attr_t const & attr )

Get a tensor dims as a vector of int, useful to construct a matching cv::Mat.

Definition at line 477 of file Utils.C.

Referenced by jevois::dnn::attrmat(), jevois::dnn::NetworkHailo::load(), jevois::dnn::Network::process(), and jevois::dnn::quantize().

◆ attrmat()

cv::Mat jevois::dnn::attrmat	(	vsi_nn_tensor_attr_t const &	attr,
		void *	dataptr = `nullptr`
	)

Construct a cv::Mat from attr and possibly data pointer.

If dataptr is nullptr, new memory will be allocated for the cv::Mat. Caller must ensure data outlives the cv::Mat, and is responsible for eventually de-allocating the data. Usually, with non-null dataptr, this is only to be used as a temporary re-casting, e.g., to recast a received tensor into a Mat before dequantizing it, then forgetting about that Mat.

Definition at line 470 of file Utils.C.

References jevois::dnn::attrdims(), and jevois::dnn::vsi2cv().

Referenced by jevois::dnn::NetworkONNX::doprocess(), jevois::dnn::NetworkNPU::doprocess(), jevois::dnn::NetworkHailo::load(), and jevois::dnn::Network::process().

◆ attrmatch()

bool jevois::dnn::attrmatch	(	vsi_nn_tensor_attr_t const &	attr,
		cv::Mat const &	blob
	)

Check that a cv::Mat blob matches exactly the spec of an attr.

Definition at line 703 of file Utils.C.

References jevois::dnn::vsi2cv().

Referenced by jevois::dnn::dequantize(), jevois::dnn::NetworkHailo::doprocess(), and jevois::dnn::NetworkNPU::doprocess().

◆ attrsize()

cv::Size jevois::dnn::attrsize ( vsi_nn_tensor_attr_t const & attr )

Get a tensor's (width, height) size in cv::Size format, skipping over other dimensions.

Definition at line 486 of file Utils.C.

References jevois::dnn::attrstr().

Referenced by jevois::dnn::PreProcessor::blobsize(), and jevois::dnn::PreProcessorBlob::process().

◆ attrstr()

std::string jevois::dnn::attrstr ( vsi_nn_tensor_attr_t const & attr )

Get a string describing the specs of a tensor, including quantification specs (not provided by shapestr())

Definition at line 511 of file Utils.C.

References jevois::to_string().

Referenced by jevois::dnn::attrsize(), jevois::dnn::NetworkONNX::doprocess(), jevois::dnn::NetworkHailo::doprocess(), jevois::dnn::NetworkNPU::doprocess(), jevois::dnn::NetworkTPU::inputShapes(), jevois::dnn::NetworkHailo::load(), jevois::dnn::NetworkTPU::outputShapes(), jevois::dnn::PreProcessorPythonImpl::process(), and jevois::dnn::PreProcessorBlob::process().

◆ clamp() [1/2]

void jevois::dnn::clamp	(	cv::Rect &	r,
		int	width,
		int	height
	)

Clamp a rectangle to within given image width and height.

Definition at line 366 of file Utils.C.

Referenced by jevois::dnn::PostProcessorDetect::process(), and jevois::dnn::PostProcessorYuNet::process().

◆ clamp() [2/2]

void jevois::dnn::clamp	(	cv::Rect2f &	r,
		float	width,
		float	height
	)

Clamp a rectangle to within given image width and height.

Definition at line 376 of file Utils.C.

◆ dequantize()

cv::Mat jevois::dnn::dequantize	(	cv::Mat const &	m,
		vsi_nn_tensor_attr_t const &	attr
	)

Dequantize an output to float32 according to the quantization spec in attr.

attr should have the type and quantization details of m, returned tensor is float32

Definition at line 792 of file Utils.C.

References jevois::dnn::attrmatch(), LFATAL, and jevois::dnn::shapestr().

Referenced by jevois::dnn::NetworkHailo::doprocess(), and jevois::dnn::NetworkNPU::doprocess().

◆ effectiveDims()

size_t jevois::dnn::effectiveDims ( cv::Mat const & m )

Returns the number of non-unit dims in a cv::Mat.

For example, returns 2 for a 4D Mat with size 1x1x224x224, since it effectively is a 224x224 2D array

Definition at line 831 of file Utils.C.

◆ getLabel()

std::string jevois::dnn::getLabel	(	std::map< int, std::string > const &	labels,
		int	id
	)

Get a label from an id.

If no entry is found in the map, return the id as a string.

Definition at line 67 of file Utils.C.

References jevois::to_string().

Referenced by jevois::dnn::PostProcessorClassify::process(), and jevois::dnn::PostProcessorDetect::process().

◆ hailo2vsi()

vsi_nn_type_e jevois::dnn::hailo2vsi ( hailo_format_type_t t )

Convert from Hailo data type to vsi_nn.

Definition at line 352 of file Utils.C.

References jevois::to_string().

Referenced by jevois::dnn::tensorattr().

◆ onnx2vsi()

vsi_nn_type_e jevois::dnn::onnx2vsi ( ONNXTensorElementDataType t )

Convert from ONNX-Runtime data type to vsi_nn.

Definition at line 207 of file Utils.C.

References jevois::to_string().

Referenced by jevois::dnn::tensorattr().

◆ parseTensorSpecs()

std::vector< vsi_nn_tensor_attr_t > jevois::dnn::parseTensorSpecs ( std::string const & specs )

Parse tensor specification.

If the specification is empty, an empty vector is returned. Throws std::range_error on any parsing error.

Definition at line 386 of file Utils.C.

References jevois::split(), and jevois::dnn::strshape().

Referenced by jevois::dnn::NetworkPython::inputShapes(), jevois::dnn::NetworkTPU::inputShapes(), jevois::dnn::NetworkOpenCV::inputShapes(), jevois::dnn::NetworkNPU::inputShapes(), jevois::dnn::NetworkNPU::load(), jevois::dnn::Network::onParamChange(), jevois::dnn::NetworkPython::outputShapes(), jevois::dnn::NetworkOpenCV::outputShapes(), jevois::dnn::NetworkTPU::outputShapes(), and jevois::dnn::NetworkNPU::outputShapes().

◆ quantize()

cv::Mat jevois::dnn::quantize	(	cv::Mat const &	m,
		vsi_nn_tensor_attr_t const &	attr
	)

Quantize from float32 to fixed-point according to the quantization spec in attr.

m should be float32, typically normalized to [0..1[ or [-1..1[ already. attr is the desired quantized type and method (DFP, AA, etc)

Definition at line 717 of file Utils.C.

References jevois::dnn::attrdims(), LFATAL, jevois::dnn::shapestr(), and jevois::dnn::vsi2cv().

Referenced by jevois::dnn::PreProcessorBlob::process().

◆ readLabelsFile()

std::map< int, std::string > jevois::dnn::readLabelsFile ( std::string const & fname )

Read a label file.

Two formats are allowed: one class name per line, or one class number followed by one class name per file.

Definition at line 24 of file Utils.C.

References LERROR, LFATAL, LINFO, and jevois::replaceStringAll().

Referenced by jevois::dnn::PostProcessorClassify::onParamChange().

◆ shapestr() [1/5]

std::string jevois::dnn::shapestr ( cv::Mat const & m )

Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional cv::Mat with data type TYPE.

Definition at line 104 of file Utils.C.

References jevois::cvtypestr(), and jevois::to_string().

Referenced by jevois::dnn::dequantize(), jevois::dnn::NetworkTPU::doprocess(), jevois::dnn::NetworkONNX::doprocess(), jevois::dnn::NetworkNPU::doprocess(), jevois::dnn::NetworkHailo::doprocess(), jevois::GUIhelperPython::drawPoly2(), jevois::dnn::NetworkTPU::load(), jevois::dnn::NetworkONNX::load(), jevois::dnn::PostProcessorSegment::process(), jevois::dnn::PostProcessorDetect::process(), jevois::dnn::PreProcessorBlob::process(), jevois::dnn::Pipeline::process(), jevois::dnn::Network::process(), jevois::dnn::quantize(), jevois::dnn::PreProcessor::sendreport(), jevois::dnn::shapestr(), and jevois::dnn::PostProcessorDetectYOLO::yolo().

◆ shapestr() [2/5]

std::string jevois::dnn::shapestr ( hailo_vstream_info_t const & vi )

Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional Hailo tensor with data type TYPE.

Definition at line 251 of file Utils.C.

References jevois::dnn::shapestr(), and jevois::dnn::tensorattr().

◆ shapestr() [3/5]

std::string jevois::dnn::shapestr ( Ort::ConstTensorTypeAndShapeInfo const & ti )

Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional ONNX tensor with data type TYPE.

Definition at line 173 of file Utils.C.

References jevois::to_string().

◆ shapestr() [4/5]

std::string jevois::dnn::shapestr ( TfLiteTensor const * t )

Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional TfLiteTensor with data type TYPE.

Definition at line 114 of file Utils.C.

References jevois::to_string().

◆ shapestr() [5/5]

std::string jevois::dnn::shapestr ( vsi_nn_tensor_attr_t const & attr )

Get a string of the form: "nD AxBxC... TYPE" from an n-dimensional NPU tensor with data type TYPE.

Definition at line 143 of file Utils.C.

References jevois::to_string().

◆ softmax()

size_t jevois::dnn::softmax	(	float const *	input,
		size_t const	n,
		size_t const	stride,
		float const	fac,
		float *	output,
		bool	maxonly
	)

Apply softmax to a float vector.

n is the number of elements to process, stride is the increment in the arrays from one element to the next. So the arrays should have size n * stride. Returns the index in [0..n*stride[ of the highest scoring element. If maxonly is true, only output[returned index] is valid.

Definition at line 667 of file Utils.C.

References LFATAL.

Referenced by jevois::dnn::PostProcessorClassify::process().