MI SED API

1. API Reference¶

1.1. Overview¶

SED (smart coding)

The intelligent coding module mainly provides the functions of creating and destroying the intelligent coding channel, opening and stopping the detection source image, calculating the result and associating it with the specified coding channel.

The core function of intelligent coding is first to do image recognition (recognition of images of objects in motion, or recognition of objects in the image), and then to set the coding parameters to the venc module (to increase the QP value and improve clarity (sharpness) when movement of objects is detected, or to increase the QP value and reduce transmission bandwidth when the image detected is in stationary state). The detection algorithm will compare the difference between two frames to detect whether there is any movement of objects, extract the ROI and motion information of the image, and set the QP value of the specified ROI area for the encoder to do the encoding. There are many types of detection algorithms. Those utilized in our platform will be introduced below.

Through the intelligent identification of ROI by the SED, as well as the dynamic setting of QP value against the ROI designated by the image, we can effectively reduce bandwidth while ensuring the quality of the image.

1.1.1. Image Detection Algorithms Type¶

1. E_MI_IVEOBJDETECT_ALGOPARAM: detects moving objects based on VDF/MD mode, with support for ROI image tracking.

2. E_MI_CNNOBJDETECT_ALGOPARAM: objects which this algorithm can detect include bicycle, bus, car, motorbike, human beings. ROI image tracking is supported.

3. E_MI_MOTIONDETECT_ALGOPARAM: motion detection based on IVE, supported only for AVBR motion detection.

The above three detection algorithms can be supported simultaneously by creating different detection channel instances.

1.1.2. Role of SED¶

The SED is responsible for identifying ROI and motion information using Sigmastar or third-party intelligent algorithms. ROI information is used for VENC setting, and motion information is used for automatic adjustment of ISP sharpness, etc.

1.1.3. Data Sources for SED¶

SED obtains yuv data through through VPE for intelligent detection (the resolution of yuv input is 352*288).

1.1.4. SED Detection Flow¶

1.1.5. Platform Limitations¶

Human non-vehicle detection is based on the detection of AI training model through IPU. The accuracy of specific detection depends on the accuracy of AI model training. Currently, only the Pudding/Tiramisu platforms support human non-vehicle detection, while other low-order platforms do not.

1.2. Keyword Description¶

1.2.1 SED¶

The main functions of Smart Encoder (SED) are the creation and destruction of smart encoding channels, opening and stopping detection of source images, calculating results and linking to designated encoding channels, etc.

1.2.2. VDF¶

VDF (Video Detection Framework) realizes initialization of MD, OD and VG video channels, channel management, management of video detection results, channel destruction and other functions

1.2.3. IVE¶

IVE (Intelligent Video Engine) module is responsible for the processing and calculation of image data. The SAD value of two images can be obtained through MI_IVE_Sad, and the histogram statistical task data can be executed through MI_IVE_Hist to achieve the dynamic and static detection of images.

1.2.4. IPU¶

IPU (AI Process Unit) is an AI model processing and computing unit. The MI IPU module implements the rapid reasoning function of the network model, independently configuring the network model for each channel, and managing the acquisition and release of input and output data.

1.2.5. ROI¶

ROI (Region Of Interest) means coding against region of interest. With the ROI function enabled, important or mobile area, i.e. the ROI region, will be in high quality lossless coding. On the other ahnd, for those areas which remain motionless or which are not selected as the ROI areas, the bit rate and image quality will be reduced for standard definition video compression, or even omission of this part of video transmission, so as to ultimately save the network bandwidth and video storage space. Users can configure their ROI area to limit the area of image Qp, so as to differentiate the Qp of that area from the Qp of the other area in the image. The system supports both H.264 and H.265 encoding ROI settings and provides 16 ROI regions for simultaneous use by users.

The 16 ROI regions can overlap with one another, and the priority of the overlapped regions is increased in sequence according to the index number of 0-15, that is to say, the QP of the overlapped regions is finally determined to be processed only according to the highest priority regions. The ROI area can be configured with absolute QP and relative QP.

Absolute QP: The QP in ROI area is the QP value set by the user

Relative QP: The QP in ROI area is the sum of the QP generated by rate control and the QP offset value set by the user. In the following example, the encoded image adopts the FixQp mode, with the image QP set to 30, that is, the QP value of all macroblocks of the image is 30. ROI region 0 is set to absolute QP mode, with QP value = 20, index = 0; ROI region 1 is set to relative QP mode, with QP value = -15, index = 1. Because the index of ROI region 0 is smaller than the index of ROI region 1, the QP of ROI region 1 with higher priority is set in the overlapped image region. The QP value of region 1 is 30-15 = 15, whereas the the QP value of region 0 is 20.

1.2.6. Motion¶

SAD information that contains the image.

1.2.7. QP¶

QP (Quantization Parameter) value corresponds to the sequence number of quantized step length. The smaller the value, the smaller the quantized step length, the higher the quantization accuracy, the better the picture quality, and the larger the size encoded.

1.2.8. MD¶

Motion detection is used to detect the movement of objects in films, and is applied to security monitoring in actual application.

1.2.9. OD¶

The occlusion detection is used to detect whether the received movie is blocked and to output the blocked detection result, if any.

1.2.10. VG¶

The virtual line segment is used to detect whether an object has crossed the set alarm line.

A zone intrusion is used to detect if an object is passing through a set alarm area.

1.2.11. SAD¶

SAD means the Sum of Absolute Differences. This algorithm is often used for image block matching – the sum of the absolute value of the difference between the corresponding values of each pixel is used to evaluate the similarity of two image blocks. It can be seen that this algorithm is fast, but not accurate, and is usually used for preliminary screening of multistage processing.

The value range of SAD is [0, 255]. All SAD values are divided into 16 ranges:

[0, 10), [10, 15, 15, 20), [20, 25), [25, 30), 30, 40), [40, 50), 50, 60), [; seven), 60, 70), [80 living), (90100), (100120), (120140), (140160), [160, 255].

In this algorithm, the percentage of all MB (8*8) in a frame that falls into the above range is calculated.

The larger the value, the more static the picture.

1.2.12. CBR¶

CBR refers to Constant Bit Rate in case of Motion. Since the code rate is constant, the size of the code word can only be reduced by increasing QP, which will however cause the image quality to worsen. When the scene is still, the image quality becomes better; therefore, the image quality is generally unstable. This algorithm prioritizes bitrate (bandwidth) over other factors.

1.2.13. VBR¶

VBR (Variable Bit Rate) means the bit rate is dynamic, varying with the complexity of the image, so its coding efficiency is relatively high. When motion occurs, mosaic is rare. The bitrate control algorithm determines the bit rate to be used according to the image content. If the image content is simple, less bitrate will be allocated (it seems that the codeword is more suitable), and if the image content is complex, more codeword will be allocated, which not only guarantees the quality but also takes into account the bandwidth limit. This algorithm gives priority to image quality over other factors.

1.2.14. CVBR¶

CVBR (Constrained VariableBit Rate) is an improvement on VBR. The Maximum bitRate or Average bitRate corresponding to this algorithm is constant. This method maintains the advantages of the above two methods: when the image content is still, the bandwidth is saved; when the image content is in motion, the bandwidth saved in the early stage is used to improve the image quality as much as possible, so as to achieve the purpose of retaining both bandwidth and image quality. This method usually allows the user to input the maximum and minimum bit rate. When still, the bit rate will be kept at the minimum bit rate, and when in motion, the bit rate will be greater than the minimum bit rate, but not more than the maximum bit rate.

1.2.15. ABR¶

The ABR (Average Bit Rate) refers to the average bit rate to reach the set code rate in a certain time range. The local peak code rate can exceed the set code rate, but the average code rate is constant.

1.2.16. AVBR¶

AVBR (Adaptive Variable Bit Rate) variable bit rate, which allows the code rate to fluctuate during the code rate statistical time, so as to ensure the stable quality of the coded image. The code rate control internally detects the motion and static state of the current scene, encodes with a higher code rate when moving, and actively reduces the target code rate when it is stationary.

Taking H.264 encoding as an example, the VENC module provides users with settings for MaxBitrate, ChangePos and MinStillPercent. MaxBitrate represents the maximum bit rate in a sports scene, and MaxBitrate*ChangePos*MinStillPercent represents the minimum bit rate in a static situation.

The target bit rate will be adjusted between the maximum bit rate and the minimum bit rate according to the degree of movement. MaxQp and MinQp are used to control the quality range of the image. The bit rate control takes QP clamping as the highest priority. If it exceeds MinQp, the bit rate control will be invalid in the MaxQp range.

1.2.17. IOU¶

The IOU (Intersection over Union) concept, used in target detection, means the overlapping rate of the generated candidate bound and the original ground truth bound, that is, the ratio of their intersection and union.

The best case is complete overlap, i.e. the ratio is 1.

2. API List¶

This module provides the following APIs:

2.1. MI_SED_CreateChn¶

• Function

  Create an SED channel

• Syntax

  MI_S32 MI_SED_CreateChn(MI_SED_CHN SedChn,
  MI_SED_DetectorAttr_t* pstAttr);
  

• Parameter

  Parameter Name	Description	Input/Output
  SedChn	SED channel number. Range: [0, SED_MAX_CHN_NUM)	Input
  pstAttr	SED detection attribute pointer	Input

• Return Value

  • Zero: Successful

  • Non-zero: Failed, see error code for details

• Dependency

  • Header: mi_common.h, mi_sed.h

  • Library: libmi_sed.a

2.2. MI_SED_DestroyChn¶

• Function

  Destroy an SED channel.

• Syntax

  MI_S32 MI_SED_DestroyChn(MI_SED_CHN SedChn);
  

• Parameter

  Parameter Name	Description	Input/Output
  SedChn	SED channel number. Range：[0, SED_MAX_CHN_NUM)	Input

• Return Value

  • Zero: Successful

  • Non-zero: Failed, see error code for details

• Dependency

  • Header: mi_common.h, mi_sed.h

  • Library: libmi_sed.a

2.3. MI_SED_StartDetector¶

• Function

  Start detection of a designated channel.

• Syntax

  MI_S32 MI_SED_StartDetector(MI_SED_CHN SedChn);
  

• Parameter

  Parameter Name	Description	Input/Output
  SedChn	SED channel number. Range：[0, SED_MAX_CHN_NUM)	Input

• Return Value

  • Zero: Successful

  • Non-zero: Failed, see error code for details

• Dependency

  • Header: mi_common.h, mi_sed.h

  • Library: libmi_sed.a

2.4. MI_SED_StopDetector¶

• Function

  Stop detection of a designated channel.

• Syntax

  MI_S32 MI_SED_StopDetector(MI_SED_CHN SedChn);
  

• Parameter

  Parameter Name	Description	Input/Output
  SedChn	SED channel number. Range：[0, SED_MAX_CHN_NUM)	Input

• Return Value

  • Zero: Successful

  • Non-zero: Failed, see error code for details

• Dependency

  • Header: mi_common.h, mi_sed.h

  • Library: libmi_sed.a

2.5. MI_SED_AttachToVencChn¶

• Description

  Attach module to designated venc channel.

• Syntax

  MI_S32 MI_SED_AttachToChn(MI_SED_CHN SedChn,
  MI_SED_TARGET_CHN TargetChn);
  

• Parameter

  Parameter Name	Description	Input/Output
  SedChn	SED channel number. Range：[0, SED_MAX_CHN_NUM)	Input
  TargetChn	Target encoder channel. Range：[0,VENC_MAX_CHN_NUM)	Input

• Return Value

  • Zero: Successful

  • Non-zero: Failed, see error code for details

• Dependency

  • Header: mi_common.h, mi_sed.h

  • Library: libmi_sed.a

2.6. MI_SED_DetachFromVencChn¶

• Description

  Detach module from designated venc channel.

• Syntax

  MI_S32 MI_SED_DetachFromChn(MI_SED_CHN SedChn,
  MI_SED_TARGET_CHN TargetChn);
  

• Parameter

  Parameter Name	Description	Input/Output
  SedChn	SED channel number. Range：[0, SED_MAX_CHN_NUM)	Input
  TargetChn	Target encoder channel. Range：[0,VENC_MAX_CHN_NUM)	Input

• Return Value

  • Zero: Successful

  • Non-zero: Failed, see error code for details

• Dependency

  • Header: mi_common.h, mi_sed.h

  • Library: libmi_sed.a

2.7. MI_SED_GetRect¶

• Description

  Get the rect info data detected by sed

• Syntax

  MI_S32 MI_SED_GetRect(MI_SED_CHN SedChn, MI_SED_RectInfo_t
  *pstRectInfo)
  

• Parameter

  Parameter Name	Description	Input/Output
  SedChn	SED channel number. Range：[0, SED_MAX_CHN_NUM)	Input
  pstRectInfo	Rect info	Output

• Return Value

  • Zero: Successful

  • Non-zero: Failed, see error code for details

• Dependency

  • Header: mi_common.h, mi_sed.h

  • Library: libmi_sed.a

2.8. MI_SED_SetDbgLevel¶

• Description

  Set SED module debug level, control log output

• Syntax

  MI_S32 MI_SED_SetDbgLevel(MI_DBG_LEVEL_e eLevel)
  

• Parameter

  Parameter Name	Description	Input/Output
  eLevel	MI debug level enum. Range：[0, MI_DBG_ALL]	Input

• Return Value

  • Zero: Successful

  • Non-zero: Failed, see error code for details

• Dependency

  • Header: mi_common.h, mi_sed.h

  • Library: libmi_sed.a

3. SED Data Type¶

The relevant data types and data structures are defined as follows:

3.1. SED_MAX_CHN_NUM¶

• Description

  Define maximum SED channel number.

• Definition

  #define SED_MAX_CHN_NUM (64)
  

3.2. SED_MAX_ROI_NUM_PER_CHN¶

• Description

  Define maximum ROI number per channel.

• Definition

  #define SED_MAX_ROI_NUM_PER_CHN (16)
  

3.3. SED_MAX_TARGET_CHN_NUM_PER_CHN¶

• Description

  Define maximum target VENC channel number per SED channel.

• Definition

  #define SED_MAX_TARGET_CHN_NUM_PER_CHN (8)
  

3.4. SED_MAX_CUS_DEF_ALGOPARAM_NUM¶

• Description

  Define maximum number of supported customer-defined algorithm.

• Definition

  #define SED_MAX_CUS_DEF_ALGOPARAM_NUM (10)
  

3.5. MI_SED_CHN¶

• Description

  Define SED channel number.

• Definition

  typedef MI_S32 MI_SED_CHN;
  

3.6. MI_SED_TARGET_CHN¶

• Description

  Define VENC encoder channel number corresponding to the SED channel number.

• Definition

  typedef MI_S32 MI_SED_TARGET_CHN;
  

3.7. MI_SED_AlgoType_e¶

• Description

  Define SED algorithm type.

• Definition

  typedef enum
  
  {
  
      E_MI_DEFAULT_ALGOPARAM = 0x0,
  
      E_MI_CUSDEF_ALGOPARAM = 0x1,
  
      E_MI_DEFAULT_ALGOPARAM_MAX
  
  } MI_SED_AlgoType_e;
  

• Member

  Member Name	Description
  E_MI_DEFAULT_ALGOPARAM	Use default SDK algorithm
  E_MI_CUSDEF_ALGOPARAM	Use customer-defined algorithm.

3.8. MI_SED_InputAttr_t¶

• Description

  Define SED input attribute.

• Definition

  typedef struct MI_SED_InputAttr_s
  
  {
  
      MI_U32 u32Width;
  
      MI_U32 u32Height;
  
      MI_U32 u32FrameRateNum;
  
      MI_U32 u32FrameRateDen;
  
      MI_SYS_ChnPort_t stInputPort;
  
  } MI_SED_InputAttr_t;
  

• Member

  Member Name	Description
  u32Width	Input YUV width
  u32Height	Input YUV height
  u32FrameRateNum	Input YUV framerate numerator
  u32FrameRateDen	Input YUV framerate denominator
  stInputPort	Input YUV channel attribute

• Related Data Type and Interface

  MI_SED_CreateChn

  MI_SED_DetectorAttr_t

3.9. MI_SED_IveObjDetect_Algo_t¶

• Description

  Do IVE Detector Param

• Definition

  typedef struct MI_SED_IveObjDetect_Algo_s
  
  MI_U32 u32VdfChn;
  
  MI_U8 u8Sensitivity;
  
  } MI_SED_IveObjDetect_Algo_t;
  

• Member

  Member Name	Description
  u32VdfChn	Vdf channel number
  u8Sensitivity	Algorithm sensitivity

3.10. MI_SED_CNNObjDetect_Algo_t¶

• Description

  Do CNN Object Detect Param

• Definition

  typedef struct MI_SED_CNNObjDetect_Algo_s
  
  {
  
      MI_U32 u32MaxVariableBufSize;
  
      MI_U32 u32IPUChnId;
  
      MI_U8 *u8FwImagePath;
  
      MI_U8 *u8ModelImagePath;
  
  } MI_SED_CNNObjDetect_Algo_t;
  

• Member

  Member Name	Description
  u32MaxVariableBufSize	Maximum memory required by IPU algorithm
  u32IPUChnId	IPU Channel Number
  u8FwImagePath	IPU firmeware file path
  u8ModelImagePath	AI Model file path

3.11. MI_SED_MontionObjDetect_Algo_t¶

• Description

  Define SED input attribute.

• Definition

  typedef struct MI_SED_InputAttr_s
  
  {
  
      MI_IVE_HANDLE iveHandle;
  
  } MI_SED_InputAttr_t;
  

• Member

  Member Name	Description
  iveHandle	IVE file handle, defult value is 0

3.12. MI_SED_Rect_t¶

• Description

  Get SED detector rect data

• Definition

  typedef struct MI_SED_Rect_s
  
  {
  
      MI_U32 u32Left;
  
      MI_U32 u32Top;
  
      MI_U32 u32Width;
  
      MI_U32 u32Height;
  
  } MI_SED_Rect_t;
  

• Member

  Member Name	Description
  u32Left	Column coordinates of the upper left corner of the rectangle
  u32Top	Row coordinates of the upper left corner of the rectangle
  u32Width	Width of rectangle
  u32Height	Height of rectangle

3.13. MI_SED_RectInfo_t¶

• Description

  All rectangle info detected by SED

• Definition

  typedef struct MI_SED_RectInfo_s
  
  {
  
      MI_U32 u32RectCount;
  
      MI_SED_Rect_t stRect[SED_MAX_ROI_NUM_PER_CHN];
  
  } MI_SED_RectInfo_t;
  

• Member

  Member Name	Description
  u32RectCount	Number of rectangles
  stRect	Information of rectangular box

3.14. MI_SED_AlgoAttr_t¶

• Description

  Define SED algorithm properties

• Definition

  typedef struct MI_SED_CusAlgoAttr_ s
  
  {
  
      MI_SED_AlgoType_e eType;
  
      Union
  
      {
  
      MI_SED_IveObjDetect_Algo_t stIveObjDetectAlgo;
  
      MI_SED_CNNObjDetect_Algo_t stCNNObjDetectAlgo;
  
      MI_SED_MotionObjDetect_Algo_t stMotionObjDetectAlgo;
  
  };
  
  } MI_SED_AlgoAttr_t;
  

• Member

  Member Name	Description
  eType	Type of algorithm used by SED
  stIveObjDetectAlgo	IVE algorithm parameters
  stCNNObjDetectAlgo	CNN algorithm parameters
  stMotionObjDetectAlgo	Motion algorithm parameters

3.15. MI_SED_TargetAttr_t¶

• Description

  Define the final effective coding target attribute

• Definition

  typedef struct MI_SED_TargetAttr_s
  
  {
  
      MI_S32 s32RltQp;
  
  } MI_SED_TargetAttr_t;
  

• Member

  Member Name	Description
  s32RltQp	ROI relative QP value

3.16. MI_SED_DetectorAttr_t¶

• Description

  Define SED detection properties

• Definition

  typedef struct MI_SED_DetectorAttr_s
  
  {
  
      MI_SED_InputAttr_t stInputAttr;
  
      MI_SED_AlgoAttr_t stAlgoAttr;
  
      MI_SED_TargetAttr_t stTargetAttr;
  
  } MI_SED_DetectorAttr_t;
  

• Member

  Member Name	Description
  stInputAttr	Properties of the SED input section
  stAlgoAttr	Attributes related to SED algorithm
  stTargetAttr	Target control attribute of SED final effect

4. Error Code¶

The SED API error codes are shown in the table below:

5. Demo¶

Build Demo code:

cd sdk/verify/mi_demo/alderaan$
make

Generated executable program in sdk\verify\mi_demo\out\demo\app\prog_sed.