Live saliency computation and image filtering using 4-core CPU and OpenGL-ES 2.0 shaders on the Mali-400MP2 GPU.
By Laurent Ittiitti@usc.eduhttp://jevois.orgGPL v3
 Language: C++Supports mappings with USB output: YesSupports mappings with NO USB output: No 
 Video Mapping:   GREY 160 495 60.0 YUYV 160 120 60.0 JeVois DemoCPUGPU

Module Documentation

This module computes saliency and gist over our 4 CPU cores while also computing 4 different image filters over the GPU, finally combining all results into a single grayscale image:

  • saliency: multicore CPU-based detection of the most conspicuous (most attention-grabbing) object in the field of view.
  • GPU filter 1: Sobel edge detector
  • GPU filter 2: Median filter
  • GPU filter 3: Morphological erosion filter
  • GPU filter 4: Morphological dilation filter

For an introduction to visual saliency, see

Also see DemoSaliency, JeVoisIntro, DarknetSaliency for more about saliency.

Video output

The video output is arranged vertically, with, from top to bottom:

  • Sobel filter results (same size as input image)
  • Median filter results (same size as input image)
  • Morphological erosion filter results (same size as input image)
  • Morphological dilation filter results (same size as input image)
  • Saliency results (those are very small): from left to right: saliency map, color map, intensity map, orientation map, flicker map, motion map. Map size is input size divided by 8 horizontally and vertically. Gist vector is appended to the right but note that at 160x120 it is truncated (see source code for details).

Serial Messages

This module can send standardized serial messages as described in Standardized serial messages formatting, where all coordinates and sizes are standardized using Helper functions to convert coordinates from camera resolution to standardized. One message is issued on every video frame at the temporally filtered attended (most salient) location:

  • Serial message type: 2D
  • id: always sm (shorthand for saliency map)
  • x, y: standardized 2D coordinates of temporally-filtered most salient point
  • w, h: always 0, 0
  • extra: none (empty string)

See Standardized serial messages formatting for more on standardized serial messages, and Helper functions to convert coordinates from camera resolution to standardized for more info on standardized coordinates.

ParameterTypeDescriptionDefaultValid Values
(Saliency) cweightbyteColor channel weight255-
(Saliency) iweightbyteIntensity channel weight255-
(Saliency) oweightbyteOrientation channel weight255-
(Saliency) fweightbyteFlicker channel weight255-
(Saliency) mweightbyteMotion channel weight255-
(Saliency) centerminsize_tLowest (finest) of the 3 center scales2-
(Saliency) deltaminsize_tLowest (finest) of the 2 center-surround delta scales3-
(Saliency) smscalesize_tScale of the saliency map4-
(Saliency) mthreshbyteMotion threshold0-
(Saliency) fthreshbyteFlicker threshold0-
(Saliency) msflickboolUse multiscale flicker computationfalse-
(Kalman2D) usevelboolUse velocity tracking, in addition to positionfalse-
(Kalman2D) procnoisefloatProcess noise standard deviation0.003F-
(Kalman2D) measnoisefloatMeasurement noise standard deviation0.05F-
(Kalman2D) postnoisefloatA posteriori error estimate standard deviation0.3F-
Detailed docs:DemoCPUGPU
Copyright:Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
License:GPL v3
Support URL:
Other URL:
Address:University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA