SDK Architecture Introduction

Version 1.0

1. Hardware Diagram

2. Keywords

Abbr Full name Introduction
VIF Video Input Interface BT656/BT1120 /MIPI/DVP signal acquisition unit. Under the same video input signal, the VIF can output the pre-scaled image and the original size image at the same time, and the original signal is output to the DRAM at the same time. Only the VIF of individual chips supports the pre-scaling function to reduce the system bandwidth in certain scenarios
VDEC Video Decoder H264/H265/JPEG video decoder
VPE Video Process Engine The original image data input by VIF is scaled and rotated, which is also the engine of image enhancement
DIVP De-interlace & VideoPost Process Engine DIVP Engine main functions:
1. DeInterlace the Interlace signal output by VIF (*only available for individual chips)
2. Do video post-processing on the video frame image output by VDEC
DISP Display Engine Two implementations of DISP:
1. DISP0/DISP1 does hardware picture splicing on the images output by the VPE/DIVP processing unit, and encodes them together with the AO output audio signal into a unit of HDMI/VGA/CVBS output signals
2. Virtual DISP : Perform software picture splicing on the images output by VPE/DIVP, and then output the spliced image, it is mainly used to support zero-channel encoding
VENC Video Encoder H264/H265/MotionJpeg encoder, the input original image sequence comes from VPE
AI Audio Input Interface I2S audio input acquisition unit, support 16 channels x16bitx8k audio signal input
AO Audio Output Interface Support 1 channel 16bit steroaudio output
RGN Region overlay & Cover Module RGN provides two functions:
①Graphic layer overlay on VPE/DIVP
②Video layer area occlusion on VPE/DIVP
GFX Graphics Engine Graphic Engine provides basic hardware acceleration support for 2D drawing, reducing CPU load
IVE Intelligent Video Engine Provide basic operator support in graphics intelligent recognition algorithms
IAE Intelligent Audio Engine Provide basic operator support for AI algorithms
MD Motion Detection Based on IVE operator library, provide algorithm modules for motion detection
OD Overlap Detection Based on the IVE operator library, provide algorithm modules for occlusion detection
VDF Video Detection Framework Integrate the intermediate architecture of each video algorithm recognition library

3. Basic Concepts

  • Data Stream

    Each mi module can be regarded as a data processing unit, and the data flow is uniformly scheduled by MI SYS. The input data stream represents the data received by the unit, and the output data stream represents the data processed by the unit.

  • Control Stream

    APP controls the parameters of each mi module data processing process, such as setting MI_VENC encoding parameters, starting and stopping MI_VPE channel, setting MI_VPE channel output port resolution and format, etc.

  • Data Stream

    1. A continuous data stream that an instance of the mi module needs to process or output, such as the original video frame sequence output by VIF, the raw ES data sequence input to VDEC, and the decoded frame sequence output by VDEC.

    2. There is no context correlation between graphics frames that need to be processed by MI_GFX.

  • Channel

    1. For an mi module that needs to process or output a stream, a channel represents the time-multiplexed context and related control flow settings that it processes or outputs a stream.

    2. The time-multiplexed module can support multiple channels, such as MI_VDEC, MI_VPE, MI_DIVP, MI_VENC, MI_VDF, virtual DISP.

    3. Modules that are not time-multiplexed can only support one channel, such as MI_DISP (0/1).

    4. Except for VIF, it can be considered to have 32 channels.

  • Port

    1. The input port is the location of the channel's input data stream, and the output port is the location of the channel's output data stream.

    2. A channel can have multiple input ports and output ports. For example, one MI_VENC channel has one input port and one output port, one MI_VPE channel has one input port and four output ports, and one MI_DISP (0/1) has 16 input port but no output port

  • MI module classification

    Data stream that needed to be handle Which has no data stream concept
    Algorithm module (exists in user mode) MI_VDF, MI_IAE MI_IVE，MI_MD,MI_OD
    Driver module (exists in kernel mode) MI_VIF, MI_VDEC, MI_VPE, MI_DIVP,MI_VENC, MI_DISP, MI_AI, MI_AO MI_GFX, MI_IVE

    • The MI module without the concept of data stream is relatively simple to process the data model. The input and output data are both the user module directly calling its private interface to input data and read back. It does not have a unified model.

    • For modules with the concept of data stream, the data input and output are all dispatched through the MI_SYS.

  • Concept of MI module channel/port

    MI module Channel number and concept Input port number and concept Output port number and concept
    MI_VIF 16 channels, each channel represents a BT656/1120 input image signal None 1 or 2 output ports, one original size output and one H/V half scaled output, the ipc chip can only support one output port
    MI_VDEC 16 channels (TBD), each channel represents one channel of video decoding case 1 input port, decode the input RAW ES data stream 1 output port, decode the input frame sequence
    MI_VPE 2 channels (TBD), each channel represents a VPE video processing case 1 input port, input processing image frame sequence 3~4 output ports, after VPE processing, output frame sequences of different resolutions, the output size, format and frame rate of each port are independently controlled
    MI_DIVP 6 channels (TBD), each channel represents a DIVP video processing case 1 input port, input processed image frame sequence 1 output port, DIVP image enhancement processing and output frame sequence after scaling
    MI_VENC 16 channels, each channel represents a video encoding case 1 input port, input the video frame sequence to be encoded 1 output port, compressed ES RAW data data stream
    MI_DISP(0/1) 1 channel, representing a physical output port (HDMI, VGA, etc) 16 input ports, 16-channel video frame sequence that can be displayed at once time None
    MI_DISP(virtual DISP) Max 4 channels, each channel represents a SW splicing image case 16 input ports, 16-channel video frame input sequence of the image to be spliced 1 output port, output 1 video frame sequence after spliced (need cpu resources)
    MI_AI 16 channel, 16 independent audio inputs None 1 output port, output audio PCM stream
    MI_AO 1 channel 1 input port, output audio PCM stream None
    MI_VDF 16 channels (TBD), each channel represents a case for intelligent identification of video streams 1 input port, input video stream 1 output port, recognition result
    MI_IAE 16 channels (TBD), each channel represents an audio intelligent identification case 1 input port, input audio stream 1 output port, recognition result

4. MI_SYS

The following describes the responsibilities of MI_SYS.

  • Manage the channel, input port, and output port of each MI module

    • Each MI module registers the number of channel/input port/output port with the MI_SYS module when it is started, and implements the related callbacks in mi_sys_DevPassOpsInfo_t

  • Provide different MI Module input port and output port bind functions

  • BIND can be called the SW PINPON function, that is, the data output by the serially connected output port, MI_SYS is automatically pushed to the corresponding input port

  • MI_SYS supports the user APP to push data to the input port of the channel

  • MI_SYS supports users to directly obtain the output data of the channel output port

  • MI_SYS supports automatic calculation of the output frequency of the channel output port for sw FRC

  • MI_SYS provides utility function to connect the data stream between kernel mode and user mode

5. Typical Application Scenario

5.1. DVR

5.2. NVR

5.3. IPC