SSD_Sensor移植
1. SENSOR PORTING RELATED HARDWARE INFORMATION¶
This document describes how to porting SigmaStar sensor driver including basic hardware knowledge, software control sensor API (sensor handle), sensor interface API (Sensor-IF) of SigmaStar and software porting flow. In this document, we take the sensor driver of imx415/imx335/sc2315/sc2335 as examples. You can follow the instructions in this document to complete the sensor porting step by step.
First we introduce the relevant knowledge of hardware and corresponding source code to make it easier to understand for you.
1.1. Sensor power sequence¶
Check the power-on sequence spec in the sensor datasheet. Please follow below power sequence 1.1V > 1.8V > 2.9V as the imx415 sensor datasheet required.
Figure 1: the IMX415 sensor power sequence spec
Table 1: the IMX415 sensor power sequence timing table
-
Corresponding source code
You can control sensor power-on and reset pins through “handle->pCus_sensor_poweron” in the sensor driver handle initial function. If you need to modify sensor power sequence timing, please check the characteristics of sensor and hardware design.
static int pCus_poweron(ms_cus_sensor *handle, u32 idx) { ISensorIfAPI *sensor_if = &handle->sensor_if_api; //Sensor power on sequence sensor_if->PowerOff(idx, handle->pwdn_POLARITY); // Powerdn Pull Low sensor_if->Reset(idx, handle->reset_POLARITY); // Rst Pull Low sensor_if->SetIOPad(idx, handle->sif_bus, handle->interface_attr.attr_mipi.mipi_lane_num); sensor_if->SetCSI_Clk(idx, CUS_CSI_CLK_216M); sensor_if->SetCSI_Lane(idx, handle->interface_attr.attr_mipi.mipi_lane_num, ENABLE); sensor_if->SetCSI_LongPacketType(idx, 0, 0x1C00, 0); if (handle->interface_attr.attr_mipi.mipi_hdr_mode == CUS_HDR_MODE_SONY_DOL) sensor_if->SetCSI_hdr_mode(idx, handle->interface_attr.attr_mipi.mipi_hdr_mode, 1); sensor_if->PowerOff(idx, !handle->pwdn_POLARITY); //Sensor board PWDN Enable, 1.8V & 2.9V need 30ms then Pull High SENSOR_MSLEEP(31); sensor_if->Reset(idx, !handle->reset_POLARITY); SENSOR_UDELAY(1); sensor_if->MCLK(idx, 1, handle->mclk); SENSOR_DMSG("Sensor Power On finished\n"); return SUCCESS; }
NOTE: Below is SigmaStar Sensor board schematic of power circuit. 1.1V, 1.8V, 2.9V power will pull high after power-on enable. First 1.1V, then 1.8V and 2.9V, you can use an oscilloscope to measure it on your sensor board.
Figure 2: the IMX415 sensor power circuit
1.2. Sensor IC slave address check¶
SigmaStar SoC needs to communicate with a sensor through I2C, please check the sensor I2C slave address first.
Table 2: the IMX415 sensor slave id table
NOTE: Check the sensor board schematic of I2C slave mode Option
Figure 3: the IMX415 sensor board I2C slave mode
-
Corresponding source code
The sensor driver I2C Setting in Marco define
//////////////////////////////////// // I2C Info // //////////////////////////////////// #define SENSOR_I2C_ADDR 0x34 //I2C slave address #define SENSOR_I2C_SPEED 300000 //I2C speed, 60000~320000 //#define SENSOR_I2C_CHANNEL 1 //I2C Channel //#define SENSOR_I2C_PAD_MODE 2 //Pad/Mode Number #define SENSOR_I2C_LEGACY I2C_NORMAL_MODE //usually set CUS_I2C_NORMAL_MODE, if use old OVT I2C protocol=> set CUS_I2C_LEGACY_MODE #define SENSOR_I2C_FMT I2C_FMT_A16D8 //CUS_I2C_FMT_A8D8, CUS_I2C_FMT_A8D16, CUS_I2C_FMT_A16D8, CUS_I2C_FMT_A16D16
NOTE: Sensor-IF of I2C control Interface in handle initial function
handle->i2c_cfg.mode = SENSOR_I2C_LEGACY; //(CUS_ISP_I2C_MODE) FALSE; handle->i2c_cfg.fmt = SENSOR_I2C_FMT; //CUS_I2C_FMT_A16D16; handle->i2c_cfg.address = SENSOR_I2C_ADDR; //0x34; handle->i2c_cfg.speed = SENSOR_I2C_SPEED; //300000;
1.3. Sensor MCLK support¶
Each sensor can support in different MCLK. That the MCLK list table that we can provide is as follows. You can find the MCLK corresponding to the sensor from the initialization settings provided by the sensor vendor. The sensor MCLK Setting in Marco define
-
Corresponding source code
//////////////////////////////////// // MCLK Info // //////////////////////////////////// /*******I5/MACARON Support MCLK List******* * CUS_CMU_CLK_27MHZ / CUS_CMU_CLK_21P6MHZ / CUS_CMU_CLK_12MHZ, * CUS_CMU_CLK_5P4MHZ / CUS_CMU_CLK_36MHZ / CUS_CMU_CLK_54MHZ, * CUS_CMU_CLK_43P2MHZ / CUS_CMU_CLK_61P7MHZ / CUS_CMU_CLK_72MHZ, * CUS_CMU_CLK_48MHZ / CUS_CMU_CLK_24MHZ / CUS_CMU_CLK_37P125MHZ, ******End of Support MCLK List*******/ #define Preview_MCLK_SPEED CUS_CMU_CLK_27MHZ #define Preview_MCLK_SPEED_HDR_DOL CUS_CMU_CLK_37P125MHZ
NOTE: The sensor MCLK control Interface in handle initial function
//////////////////////////////////// // mclk // //////////////////////////////////// handle->mclk = UseParaMclk(SENSOR_DRV_PARAM_MCLK());
2. Sensor porting relevant software control¶
After introducing the relevant knowledge of hardware and corresponding source code, we will introduce software control API (sensor handle) and sensor interface API (Sensor-IF) of SigmaStar in this part.
2.1. Sensor driver support mode define¶
Register the sensor support behavior mode. For example, the IMX415 sensor supports linear and HDR mode, you can refer to the IMX415 sensor driver and fill it in define of “SENSOR_DRV_ENTRY_IMPL_END_EX”.
SENSOR_DRV_ENTRY_IMPL_END_EX( IMX415_HDR, cus_camsensor_init_handle, cus_camsensor_init_handle_hdr_dol_sef, cus_camsensor_init_handle_hdr_dol_lef, imx415_params );
Table 3: the sensor driver “SENSOR_DRV_ENTRY_IMPL_END_EX” define
Parameter | Describe |
---|---|
Name | Sensor Name and support mode |
LinearEntry | Sensor Linear mode handle initial function |
HdrSefEntry | Sensor HDR Short Exposure handle initial function |
HdrLefEntry | Sensor HDR Long Exposure handle initial function |
PrivateDataType | Sensor Private Data Type |
2.1.1. Introduce “SENSOR_DRV_IMPL_END_EX”¶
You can check macro define “SENSOR_DRV_ENTRY_IMPL_END_EX” more detail in the path “driver/SensorDriver/drv/inc/drv_sensor_common.h”. LinearEntry has a group of private data set, HdrSefEntry and HdrLefEntry have another a group of private data set.
#define SENSOR_DRV_ENTRY_IMPL_END_EX (Name,LinearEntry,HdrSefEntry,HdrLefEntry,PrivateDataType)\ static PrivateDataType* g_pData[2][MAX_CAMID_LEN] = { {0,0,0,0}, {0,0,0,0} };\ int Name##_init_driver(unsigned char chmap)\ {\ int nCamID=0;\ /*To avoid compile warning*/\ void* p0 = LinearEntry;\ void* p1 = HdrSefEntry;\ void* p2 = HdrLefEntry;\ if(DrvSensorHandleVer(CUS_CAMSENSOR_HANDLE_MAJ_VER, CUS_CAMSENSOR_HANDLE_MIN_VER)==FAIL)\ return FAIL;\ if(DrvSensorIFVer(CUS_CAMSENSORIF_MAJ_VER, CUS_CAMSENSORIF_MIN_VER)==FAIL)\ return FAIL;\ if(DrvSensorI2CVer(CUS_CAMSENSOR_I2C_MAJ_VER, CUS_CAMSENSOR_I2C_MIN_VER)==FAIL)\ return FAIL;\ for(nCamID=0;nCamID<MAX_CAMID_LEN;++nCamID)\ {\ if((chmap>>nCamID)&0x1)\ {\ if(p0){\ void* pData = CamOsMemAlloc(sizeof(PrivateDataType));\ memset(pData,0,sizeof(PrivateDataType));\ DrvRegisterSensorDriverEx(nCamID, LinearEntry,pData);\ DrvRegisterSensorI2CSlaveID(nCamID, (i2c_slave_id>>(nCamID*8))&0xFF);\ SENSOR_DMSG("Connect %s linear to sensor pad %d\n",__FUNCTION__, nCamID);\ g_pData[0][nCamID] = pData;\ }\ if(p1||p2){\ void* pData = CamOsMemAlloc(sizeof(PrivateDataType));\ memset(pData,0,sizeof(PrivateDataType));\ if(p1){\ DrvRegisterPlaneDriverEx(nCamID, 1, HdrSefEntry,pData);\ DrvRegisterPlaneI2CSlaveID(nCamID, 1, (i2c_slave_id>>(nCamID*8))&0xFF);\ SENSOR_DMSG("Connect %s SEF to vif sensor pad %d\n",__FUNCTION__, nCamID);\ }\ if(p2){\ DrvRegisterPlaneDriverEx(nCamID, 0, HdrLefEntry,pData);\ DrvRegisterPlaneI2CSlaveID(nCamID, 1, (i2c_slave_id>>(nCamID*8))&0xFF);\ SENSOR_DMSG("Connect %s LEF to sensor pad %d\n",__FUNCTION__, nCamID);\ }\ g_pData[1][nCamID] = pData;\ }\ }\ }\ return 0;\ }\
2.2. Sensor driver handle function¶
Next we will introduce and explain handle API in the linear and HDR handle initial functions. We can control, set or read any state of the sensor through the handle API. Here we take the sensor driver of imx415 as an example. You can step through the sensor porting of linear and HDR mode according to the instructions in this document.
2.2.1. Handle of IC API¶
Set I2C hardware information
Table 4: handle of I2C API define
Parameter | Describe | NOTE |
---|---|---|
handle->i2c_cfg.mode | Register I2C Mode | I2C_NORMAL_MODE /**\< Sensor driver can only use I2C_NORMAL_MODE */ I2C_LEGACY_MODE, /**\< Do not use */ |
handle->i2c_cfg.fmt | Register set I2C format for bit of address & data length. | /**\< I2C_FMT_A16D8 means 16 bits Address, 8 bits Data */ I2C_FMT_A8D8, I2C_FMT_A16D8, I2C_FMT_A8D16, I2C_FMT_A16D16, I2C_FMT_END /**\< Reserved */ |
handle->i2c_cfg.address | Register set sensor I2C slave address, SigmaStar SoC supports 8bits slave address | Check From Sensor Datasheet |
handle->i2c_cfg.speed | Register set sensor max I2C speed (MACARON fixed at 300 KHz) | I2C speed, 60000~320000 |
2.2.2. Handle of MCLK API¶
Set the sensor MCLK information
Table 5: handle of MCLK API define
handle->mclk | Register set VIF module output MCLK to sensor For MI Query!! | Main Chip Support MCLK List as : CUS_CMU_CLK_27MHZ, CUS_CMU_CLK_21P6MHZ, CUS_CMU_CLK_12MHZ, CUS_CMU_CLK_5P4MHZ, CUS_CMU_CLK_36MHZ, CUS_CMU_CLK_54MHZ, CUS_CMU_CLK_43P2MHZ, CUS_CMU_CLK_61P7MHZ, CUS_CMU_CLK_72MHZ, CUS_CMU_CLK_48MHZ, CUS_CMU_CLK_24MHZ, CUS_CMU_CLK_37P125MHZ, CUS_CMU_CLK_LPLL_DIV1, CUS_CMU_CLK_LPLL_DIV2, CUS_CMU_CLK_LPLL_DIV4, CUS_CMU_CLK_LPLL_DIV8, |
---|---|---|
2.2.3. Handle of Sensor model id API¶
Set sensor name and mode information
Table 6: handle of model id API define
Parameter | Describe | NOTE |
---|---|---|
handle->model_id | Register set sensor model for libcamera user by using | MIPI IMX415_MIPI |
PARALLEL IMX323_PARL | ||
HDR IMX415_MIPI_HDR_SEF IMX415_MIPI_HDR_LEF |
2.2.4. Handle of sensor interface and output hardware information¶
Table 7: handle of Sensor interface and output hardware information API define
Parameter | Describe | NOTE |
---|---|---|
handle->pCus_sensor_poweron | Fill Up Sensor IF Info!!! Register set sensor power-on sequence function. | Function ref : pCus_poweron |
handle->pCus_sensor_init | Register load sensor initial setting function. | |
handle->pCus_sensor_post_init | Set and enable SoC mipi csi status. | Special case is like SmartSens product SC4236/SC2315 |
handle->isp_type | Register ISP type (not used in MACARON) | SENSOR_ISP_TYPE ISP_SOC, /**\< Not support */ ISP_EXT, /**\< sensor without built-in ISP */ |
handle->sif_bus | Register set sensor interface bus type | SENSOR_IFBUS_TYPE CUS_SENIF_BUS_PARL = 0, /**\< sensor data bus is parallel */ CUS_SENIF_BUS_MIPI = 1, /**\< sensor data bus is mipi */ CUS_SENIF_BUS_BT601 = 2, CUS_SENIF_BUS_BT656 = 3, CUS_SENIF_BUS_BT1120 = 4, |
handle->data_prec | Register set sensor output raw data precision | SENSOR_DATAPREC CUS_DATAPRECISION_8 = 0, /**\< raw data precision 8bits */ CUS_DATAPRECISION_10 = 1, /**\< raw data precision 10bits */ CUS_DATAPRECISION_16 = 2, /**\< raw data precision 16bits */ CUS_DATAPRECISION_12 = 3, /**\< raw data precision 12bits */ CUS_DATAPRECISION_14 = 4, /**\< raw data precision 14bits */ |
handle->data_mode | Register set sensor bayer raw (8/10-bits) to 12-bits mode (Not used in MACARON) | SENSOR_DATAMODE CUS_SEN_8TO12_7074, CUS_SEN_8TO12_7000, CUS_SEN_8TO12_114118, CUS_SEN_8TO12_11400, CUS_SEN_10TO12_9098, CUS_SEN_10TO12_9000, CUS_SEN_10TO12_1121110, CUS_SEN_10TO12_11200 |
handle->bayer_id | Register set sensor bayer raw start id | SENSOR_BAYERID CUS_BAYER_RG = 0, CUS_BAYER_GR, CUS_BAYER_BG, CUS_BAYER_GB, |
handle->RGBIR_id | Register set sensor RGB-IR start id #If NOT RGB-IR sensor, please set CUS_RGBIR_NONE |
SENSOR_RGBIRID CUS_RGBIR_NONE = 0, CUS_RGBIR_R0 = 1, CUS_RGBIR_G0 = 2, CUS_RGBIR_B0 = 3, CUS_RGBIR_G1 = 4, CUS_RGBIR_G2 = 5, CUS_RGBIR_I0 = 6, CUS_RGBIR_G3 = 7, CUS_RGBIR_I1 = 8 |
handle->orient | Register get sensor image mirror-flip default status | SENSOR_ORIT CUS_ORIT_M0F0, /**\< mirror, flip unchanged */ CUS_ORIT_M1F0, /**\< mirror changed */ CUS_ORIT_M0F1, /**\< flip changed */ CUS_ORIT_M1F1, /**\< mirror and flip changed */ |
handle->snr_pad_group | Register sensor pad group | CUS_SENSOR_PAD_GROUP_A = 0, CUS_SENSOR_PAD_GROUP_B = 1, CUS_SENSOR_PAD_GROUP_A = 0, CUS_SENSOR_PAD_GROUP_B = 1, |
handle->snr_pad_mode | Register sensor master and slave mode | CUS_SENSOR_MASTER_MODE = 0, CUS_SENSOR_SLAVE_MODE = 1, |
handle->ir_only_fmt | Register sensor ir format status | |
For HDR (MIPI) Interface | ||
handle->interface_attr.attr_mipi.mipi_lane_num | Register set MIPI lane number | Default : 4 |
handle->interface_attr.attr_mipi.mipi_data_format | Register set MIPI data format | CUS_SEN_INPUT_FORMAT_YUV422, CUS_SEN_INPUT_FORMAT_RGB, |
handle->interface_attr.attr_mipi.mipi_yuv_order | Register set MIPI yuv order | For yuv sensor only |
handle->interface_attr.attr_mipi.mipi_hsync_mode | Register set HDR virtual channel0 hsync mode | SENSOR_MIPI_HSYNC_MODE PACKET_HEADER_EDGE1 = 0, /**\< packet header edge */ PACKET_HEADER_EDGE2 = 1, /**\< line end edge */ PACKET_HEADER_EDGE3 = 2, /**\< line start edge */ PACKET_FOOTER_EDGE = 3, /**\< packet footer edge */ |
handle->interface_attr.attr_mipi.mipi_hdr_mode | Register set MIPI HDR mode | CUS_HDR_MODE_NONE = 0, CUS_HDR_MODE_SONY_DOL = 1, CUS_HDR_MODE_DCG = 2, CUS_HDR_MODE_EMBEDDED_RAW8 = 3, CUS_HDR_MODE_EMBEDDED_RAW10 = 4, CUS_HDR_MODE_EMBEDDED_RAW12 = 5, CUS_HDR_MODE_EMBEDDED_RAW16 = 6, //Only for OV2718? CUS_HDR_MODE_LI = 7, |
handle->interface_attr.attr_mipi. mipi_sampling_delay | MIPI CLK skip time (ns) (Not used in MACARON) | |
handle->interface_attr.attr_mipi.mipi_hdr_virtual_channel_num | Register set HDR virtual channel number #Based on each sensor VC mode header define | For example imx415: Long Exposure Frame : 0 Short Exposure Frame : 1 |
2.2.5. Handle of signal polarity API¶
Set SoC hardware signal polarity
Table 8: handle of signal polarity API define
Parameter | Describe | NOTE |
---|---|---|
handle->pwdn_POLARITY | Register set output sensor power-down pin (GPIO) polarity | SENSOR_PWDN_POL CUS_CLK_POL_POS = 0, /**\< High active */ CUS_CLK_POL_NEG /**\< Low active */ |
handle->reset_POLARITY | Register set output sensor reset pin (GPIO) polarity | SENSOR_RST_POL CUS_CLK_POL_POS = 0, /**\< High active */ CUS_CLK_POL_NEG /**\< Low active */ |
For Parallel Interface | ||
handle->VSYNC_POLARITY | Register set receive sensor Vsync pin signal polarity | SENSOR_VSYNC_POL CUS_CLK_POL_POS = 0, /**\< High active */ CUS_CLK_POL_NEG /**\< Low active */ |
handle->HSYNC_POLARITY | Register set receive sensor Hsync pin signal polarity | SENSOR_HSYNC_POL CUS_CLK_POL_POS = 0, /**\< High active */ CUS_CLK_POL_NEG /**\< Low active */ |
handle->PCLK_POLARITY | Register set receive sensor pixel clk pin signal polarity | SENSOR_PCLK_POL CUS_CLK_POL_POS = 0, /**\< High active */ CUS_CLK_POL_NEG /**\< Low active */ |
2.2.6. Handle of sensor resolution capability API¶
Set sensor support and implement output image information.
Table 9: handle of sensor resolution capability API define
Parameter | Describe | NOTE |
---|---|---|
handle->video_res_supported.num_res | Register set sensor support resolution number | |
handle->video_res_supported.ulcur_res | Register set sensor current resolution | Default : 0 |
handle->video_res_supported.res[n].width | Register set sensor support width | |
handle->video_res_supported.res[n].height | Register set sensor support height | |
handle->video_res_supported.res[n].max_fps | Register set sensor support preview maximum fps | |
handle->video_res_supported.res[n].min_fps | Register set sensor support preview minimum fps | |
handle->video_res_supported.res[n].crop_start_x | Register set VIF crop sensor output image start of X | |
handle->video_res_supported.res[n].crop_start_y | Register set VIF crop sensor output image start of Y | |
handle->video_res_supported.res[n].nOutputWidth | Register set VIF output active pixel | |
handle->video_res_supported.res[n].nOutputHeight | Register set VIF output active line | |
handle->video_res_supported.res[n].strResDesc | Register set sensor resolution & fps string | EX: 1920x1080\@30fps |
2.2.7. Handle of set/get sensor control API¶
We can load sensor initial setting, switch sensor output resolution, sensor mirror or flip and fps set or get through these functions.
Table 10: handle of set/get sensor control API define
Parameter | Describe | NOTE |
---|---|---|
handle->pCus_sensor_release | Register sensor release function. | Function ref : cus_camsensor_release_handle |
handle->pCus_sensor_init | Register load sensor initial setting function. | Function ref : pCus_init_mipi4lane_linear |
handle->pCus_sensor_poweron | Register set sensor power-on sequence function. | Function ref : pCus_poweron |
handle->pCus_sensor_poweroff | Register set sensor power-off sequence function. | Function ref : pCus_poweroff |
handle->pCus_sensor_GetSensorID | Register get and check sensor ID | Function ref : pCus_GetSensorID |
handle->pCus_sensor_GetVideoResNum | Register get sensor support resolution number function, that’s video output index. | Function ref : pCus_GetVideoResNum |
handle->pCus_sensor_GetVideoRes | Register get sensor support preview info function (width, height, and frame rate). | Function ref : pCus_GetVideoRes |
handle->pCus_sensor_GetCurVideoRes | Register get sensor current preview info function (width, height, and frame rate). | Function ref : pCus_GetCurVideoRes |
handle->pCus_sensor_SetVideoRes | Register switch sensor and load initial setting function. | Function ref : pCus_SetVideoRes |
handle->pCus_sensor_GetOrien | Register get sensor mirror-flip status function. | Function ref : pCus_GetOrien |
handle->pCus_sensor_SetOrien | Register set Sensor mirror-flip status function. | Function ref : pCus_SetOrien |
handle->pCus_sensor_GetFPS | Register get sensor FPS status function. | Function ref : pCus_GetFPS |
handle->pCus_sensor_SetFPS | Register set sensor FPS status function. | Function ref : pCus_SetFPS |
handle->pCus_sensor_SetPatternMode | Set sensor test pattern (Color Bar Pattern). |
2.2.8. Handle of set/get sensor AE control API¶
First we will introduce the sensor driver private data in this part, this purpose is to record the parameters of the sensor without any modification. Here is imx415 as a reference example. The naming can be customized according to each sensor or developer, and there is no mandatory requirement.
Table 11: example for imx415 private data
Parameter | Describe | NOTE |
---|---|---|
params->tGain_reg | Register set each frame sensor AE gain parameter. | For imx415 Linear |
params->tExpo_reg | Register set each frame sensor exposure line parameter. | For imx415 Linear |
params->tVts_reg | Register set each frame sensor total line parameter. | For imx415 Linear, LEF |
params->tSHR0_reg | Register set each frame long exposure frame shutter time. | For imx415 LEF |
params->tRHS1_reg | Register set each frame readout time of short exposure frame. | For imx415 SEF |
params->tSHR1_reg | Register set each frame short exposure frame shutter time. | For imx415 SEF |
params->tGain_hdr_dol_lef_reg | Register set each frame long exposure frame HDR AE Gain. | For imx415 LEF |
params->tGain_hdr_dol_sef_reg | Register set each frame short exposure frame HDR AE Gain. | For imx415 SEF |
params->expo.vts | Register set each frame total line. | For imx415 Linear, LEF |
params->expo.expo_lines | Register set each frame exposure line. | For imx415 Linear, LEF |
params->expo.fps | Register set sensor fps. | For imx415 Linear, LEF |
OTHERS |
We can set or get sensor AE function status through this API, including AE shutter time, gain, limit AE setting value and AE update timing, etc.
Table 12: handle of set/get sensor AE control API define
Parameter | Describe | NOTE |
---|---|---|
handle->ae_gain_delay | Register set sensor AE Gain effective frame after setting frame. | Based On Sensor Characteristic Please check *SUPPLYMENT4.2 |
handle->ae_shutter_delay | Register set sensor AE Shutter effective frame after setting frame. | Based On Sensor Characteristic Please check *SUPPLYMENT4.2 |
handle->ae_gain_ctrl_num | Register set sensor AE Gain table number. | Linear mode set to 1 HDR mode set to 2 |
handle->ae_shutter_ctrl_num | Register set sensor AE Shutter table number. | Linear mode set to 1 HDR mode set to 2 |
handle->sat_mingain | Register get sensor Initial minimum Gain. | Minimum Gain != 1 |
handle->pCus_sensor_AEStatusNotify | Register update AE Gain, Shutter and fps status at frame start or frame end. | Function Ref : pCus_AEStatusNotify Status 0 is frame end update Status 1 is frame start update |
handle->pCus_sensor_GetAEUSecs | Register get sensor output image exposure time. | Function Ref : pCus_GetAEUSecs |
handle->pCus_sensor_SetAEUSecs | Register set sensor output image exposure time. | Function Ref : pCus_SetAEUSecs |
handle->pCus_sensor_GetAEGain | Register get sensor AE Gain. | Function Ref : pCus_GetAEGain |
handle->pCus_sensor_SetAEGain | Register set sensor AE Gain. | Function Ref : pCus_SetAEGain |
handle->pCus_sensor_GetAEMinMaxGain | Register get sensor minimum & maximum Gain information | Function Ref : pCus_GetAEMinMaxGain |
handle->pCus_sensor_GetAEMinMaxUSecs | Register get sensor minimum & maximum exposure time | Function Ref : pCus_GetAEMinMaxUSecs |
handle->pCus_sensor_GetShutterInfo | Register get sensor minimum & maximum exposure in line, and shutter step | Function: |
2.2.9. Sensor-IF API¶
SENSOR_IF API for Connect with the sensor driver and the VIF driver, this callback control in below handle control function.
We will introduce and describe this API, more detail you can reference the sensor driver of IMX415.
Parameter | Describe | NOTE |
---|---|---|
sensor_if->PowerOff | Set Sensor-IF PowerDown pull high or not. | Callback to VIF driver |
sensor_if->Reset | Set Sensor-IF SW RESET pull high or not. | Callback to VIF driver |
sensor_if->MCLK | Set VIF Output MCLK | Callback to VIF driver |
sensor_if->SetIOPad | For SENSOR-IF I/O Bus Mode | Callback to VIF driver |
sensor_if->QueryMCLK | For query sensor master clock | Callback to VIF driver |
sensor_if->QueryLaneNum | For query sensor MIPI lane number | Callback to VIF driver |
sensor_if->SetSkipFrame | Skip vif output frame | Callback to VIF driver |
For MIPI Interface | ||
sensor_if->SetCSI_Clk | Set MIPI Interface MAC CLK | Callback to VIF driver |
sensor_if->SetCSI_Lane | Set MIPI Interface Data Lane | Callback to VIF driver |
sensor_if->SetCSI_hdr_mod | Set HDR Data Format | Callback to VIF driver |
sensor_if->SetCSI_LongPacketType | Set MIPI Long Packet Type | Callback to VIF driver |
sensor_if->SetCSI_clk_data_skip | Skip hs clock when into HSRX mode | Callback to VIF driver |
sensor_if->VCOHsmode | Callback to VIF driver |
3. Software porting flow¶
This chapter introduces the process of software porting sensor driver. First, it will explain how to initialize the sensor. Then describe the information required by the sensor driver 3A and control the functions that 3A will call.
3.1. The sensor driver initial flow introduction¶
Figure 4: porting the sensor driver flow
This document introduces the initial process of transplanting the sensor according to the process in the figure above.
3.1.1. Set the sensor information setting¶
These settings are the hardware information of the sensor, we have listed the necessary hardware information, you can refer to the IMX415 / IMX335 sensor driver,
//////////////////////////////////// // Sensor-If Info // //////////////////////////////////// //MIPI config begin. #define SENSOR_MIPI_LANE_NUM (lane_num) //IMX335 Linear mode supports MIPI 2/4 Lane #define SENSOR_MIPI_LANE_NUM_DOL (4) //(hdr_lane_num)//IMX335 DOL mode supports MIPI 4 Lane //#define SENSOR_MIPI_HDR_MODE (0) //0: Non-HDR mode. 1:Sony DOL mode #define SENSOR_ISP_TYPE ISP_EXT //ISP_EXT, ISP_SOC (Non-used) //#define SENSOR_DATAFMT CUS_DATAFMT_BAYER //CUS_DATAFMT_YUV, CUS_DATAFMT_BAYER #define SENSOR_IFBUS_TYPE CUS_SENIF_BUS_MIPI //CFG //CUS_SENIF_BUS_PARL, CUS_SENIF_BUS_MIPI #define SENSOR_MIPI_HSYNC_MODE PACKET_HEADER_EDGE1 #define SENSOR_MIPI_HSYNC_MODE_HDR_DOL PACKET_FOOTER_EDGE #define SENSOR_DATAPREC CUS_DATAPRECISION_10 #define SENSOR_DATAPREC_DOL CUS_DATAPRECISION_10 #define SENSOR_DATAMODE CUS_SEN_10TO12_9098 //CFG #define SENSOR_BAYERID CUS_BAYER_RG //0h: CUS_BAYER_RG, 1h: CUS_BAYER_GR, 2h: CUS_BAYER_BG, 3h: CUS_BAYER_GB #define SENSOR_BAYERID_HDR_DOL CUS_BAYER_RG #define SENSOR_RGBIRID CUS_RGBIR_NONE #define SENSOR_ORIT CUS_ORIT_M0F0 //CUS_ORIT_M0F0, CUS_ORIT_M1F0, CUS_ORIT_M0F1, CUS_ORIT_M1F1, //#define SENSOR_YCORDER CUS_SEN_YCODR_YC //CUS_SEN_YCODR_YC, CUS_SEN_YCODR_CY //#define long_packet_type_enable 0x00 //UD1~UD8 (user define) //////////////////////////////////// // MCLK Info // //////////////////////////////////// #define Preview_MCLK_SPEED CUS_CMU_CLK_27MHZ//CUS_CMU_CLK_37P125MHZ//CUS_CMU_CLK_27MHZ #define Preview_MCLK_SPEED_HDR_DOL CUS_CMU_CLK_27MHZ
3.1.2. Set the sensor IC setting¶
SoC communicates with sensors mainly through I2C. Therefore, you need to set I2C before sending sensor setting value to the sensor.
Please refer to this document chapter 1.2.
3.2. Set support image resolution information¶
You need to fill in the sensor image information provided by the sensor provider FAE. This information includes preview size, image crop size and site etc. We define this information as a global variable.
//////////////////////////////////// // Image Info // //////////////////////////////////// static struct { // LINEAR // Modify it based on number of support resolution enum {LINEAR_RES_1 = 0, LINEAR_RES_2, LINEAR_RES_3, LINEAR_RES_4, LINEAR_RES_5, LINEAR_RES_6, LINEAR_RES_7, LINEAR_RES_8, LINEAR_RES_END}mode; // Sensor Output Image info struct _senout{ s32 width, height, min_fps, max_fps; }senout; // VIF Get Image Info struct _sensif{ s32 crop_start_X, crop_start_y, preview_w, preview_h; }senif; // Show resolution string struct _senstr{ const char* strResDesc; }senstr; }imx415_mipi_linear[] = { {LINEAR_RES_1, {3860, 2250, 3, 20}, {0, 0, 3840, 2160}, {"3840x2160@20fps"} }, {LINEAR_RES_2, {3096, 2190, 3, 30}, {0, 0, 3072, 2048}, {"3072x2048@30fps"} }, // Modify it {LINEAR_RES_3, {3096, 1758, 3, 30}, {0, 0, 3072, 1728}, {"3072x1728@30fps"} }, // Modify it {LINEAR_RES_4, {2616, 1974, 3, 30}, {0, 0, 2592, 1944}, {"2592x1944@30fps"} }, // Modify it {LINEAR_RES_5, {2976, 1686, 3, 30}, {0, 0, 2944, 1656}, {"2944x1656@30fps"} }, // Modify it {LINEAR_RES_6, {2592, 1470, 3, 30}, {0, 0, 2560, 1440}, {"2560x1440@30fps"} }, // Modify it {LINEAR_RES_7, {1920, 1080, 3, 60}, {0, 0, 1920, 1080}, {"1920x1080@60fps"} }, // Modify it {LINEAR_RES_8, {3864, 2192, 3, 30}, {0, 0, 3840, 2160}, {"3840x2160@30fps"} }, // Modify it //{LINEAR_RES_9, {2616, 1974, 3, 25}, {0, 0, 2592, 1944}, {"2592x1944@25fps"} }, // Modify it };
3.2.1. The sensor power on sequence function¶
Please refer to this document chapter 1.1.
3.2.2. The sensor initial function¶
Here is the initial setting of the sensor through the i2c device, first read the product ID or sensor ID of the sensor device to ensure that the correct sensor is used. Please refer to the function registered by “handle->pCus_sensor_init”.
3.2.3. Select the sensor output resolution function¶
MI will get the image resolution information supported by the sensor driver and select one of the sensor's initial settings.
Please refer to the function registered by “handle->pCus_sensor_SetVideoRes”.
3.3. Implement the sensor control function¶
The functions that the sensor driver needs to support include fps adjustment, AE shutter adjustment, AE gain adjustment, and mirror flip function settings, which we will not describe more detail, because this needs to be implemented according to the characteristics of each sensor.
The following is our control the sensor implement function flow. Relatively, we also need to get the sensor information, please refer to the figure below:
Figure 5: control the sensor implement function flow
Figure 6: get the sensor implement function information or status
3.3.1. AE shutter control and get AE shutter information¶
How to control AE shutter, please refer to the function registered by “handle->pCus_sensor_SetAEUSecs”.
Also, ISP algo or MI need to get AE shutter information, please refer to the function registered by “handle->pCus_sensor_GetAEUSecs” and “handle->pCus_sensor_GetShutterInfo”.
3.3.2. AE gain control and get AE gain information¶
How to control AE gain, please refer to the function registered by “handle->pCus_sensor_SetAEGain”.
Also, ISP algo or MI need to get AE gain information, please refer to the function registered by “handle->pCus_sensor_GetAEGain” and “handle->pCus_sensor_GetAEMinMaxGain”.
3.3.3. Sensor frame rate control and get sensor frame rate information¶
How to control the sensor output frame rate, please refer to the function registered by “handle->pCus_sensor_SetFPS”.
Also, MI needs to get the sensor output frame rate information, please refer to the function registered by “handle->pCus_sensor_GetFPS”.
3.3.4. Sensor mirror-flip control and get sensor mirror-flip information¶
How to control the sensor image mirror-flip, please refer to the function registered by “handle->pCus_sensor_SetOrien”.
Also, MI needs to get the sensor image mirror-flip information, please refer to the function registered by “handle->pCus_sensor_GetOrien”.
3.4. IC execution time control¶
The above AE shutter, AE gain, sensor frame rate and mirror control, we hope everyone can send commands to the sensor through I2C at the same time
So we will implement this using the function "handle->pCus_sensor_AEStatusNotify"
Therefore, we can control the time of I2C send commands to the sensor through two parameters "CUS_FRAME_ACTIVE" and "CUS_FRAME_INACTIVE". Most sensors can be set at the frame start, so the parameter "CUS_FRAME_ACTIVE" can be selected.
Figure 7: I2C execution time
NOTE:
If the porting sensor cannot find out what caused the image abnormality, you can call “return success” directly in the function.
3.4.1. Sensor AE shutter/gain delay count¶
How to set “handle->ae_shutter_delay” and “handle->ae_gain_delay”, please refer to this document chapter 4.2.
3.5. Sensor AE shutter/gain delay count¶
How to set “handle->ae_gain_ctrl_num” and “handle->ae_shutter_ctrl_num”, please refer to this document chapter 2.2.8.
4. Supplyment¶
4.1. Install the Sensor Driver and parameter introduce¶
SigmaStar sensor driver provides several installation parameters that can be used when installing the sensor driver. Here are the parameters and instructions as below:
Table 13: sensor driver install parameter
Parameter | Describe |
---|---|
chmap | Specify SoC Channel pad for sensor |
lane_num | Specify Sensor linear mode MIPI lane number |
hdr_lane_num | Specify Sensor hdr mode MIPI lane number |
i2c_slave_id | Specify Sensor i2c slave id |
For example:
insmod imx335_MIPI.ko chmap=1 lane_num=4 hdr_lane_num=4 i2c_slave_id=0x20 |
---|
If the above parameters are not used when installing the sensor driver, the default values in the sensor driver are used.
4.2. Verify and debug video stream¶
The next step is to verify that the sensor driver can be used normally, so, you need to copy app from “/sdk/verify/mi_demo/out/demo/app” to EVB, I take “prog_cus3a” as an example:
/customer # ./prog_cus3a use default sensor pad 0, vif dev 0 Use HDR ? 0 not use, 1 use DOL 0 You select not use HDR client [548] connected, module:sensor index 0, Crop(0,0,3840,2160), outputsize(3860,2250), maxfps 20, minfps 3, ResDesc 3840x2160@20fps index 1, Crop(0,0,3072,2048), outputsize(3096,2190), maxfps 30, minfps 3, ResDesc 3072x2048@30fps index 2, Crop(0,0,3072,1728), outputsize(3096,1758), maxfps 30, minfps 3, ResDesc 3072x1728@30fps index 3, Crop(0,0,2592,1944), outputsize(2616,1974), maxfps 30, minfps 3, ResDesc 2592x1944@30fps index 4, Crop(0,0,2944,1656), outputsize(2976,1686), maxfps 30, minfps 3, ResDesc 2944x1656@30fps index 5, Crop(0,0,2560,1440), outputsize(2592,1470), maxfps 30, minfps 3, ResDesc 2560x1440@30fps index 6, Crop(0,0,1920,1080), outputsize(1920,1080), maxfps 60, minfps 3, ResDesc 1920x1080@60fps index 7, Crop(0,0,3840,2160), outputsize(3864,2192), maxfps 30, minfps 3, ResDesc 3840x2160@30fps choice which resolution use, cnt 8
4.2.1. Check frame interval¶
At the same time you can open telnet and enter the following cmd to confirm the VIF and ISP status.
Usually 30fps means that a frame period is 33.3ms.
/ # cat /sys/class/mstar/vif0/vif_ints == VIF CH-0 == Interval(ns) : 33309000 VREF_RISING : 6147 VREF_FALLING : 6149 LINE_CNT_0 : 2 LINE_CNT_1 : 2
4.2.2. Check the sensor output stream status¶
Confirm the information of the sensor signal received by VIF device
/ # cat /sys/class/mstar/vif0/vif_info == VIF CH-0 == INTF_MODE = 4 CH_EN: 1 SRC: MIPI0 INPUT_FMT: RGB PIX_FMT: 12 bits HDR_EN: 0 HDR_CH: VC0 CROP_EN: 1 CROP_START_X: 0 - 1919 CROP_START_Y: 0 - 1079 PIXEL_CNT: 1920 LINE_CNT: 1080
4.2.3. Check the ISP status¶
If the transmission speed of one line or pixel per frame is too fast, even if the VIF can receive the image data normally, there is no guarantee that the ISP can process it immediately, it causes isp fifo full.
/ # cat /sys/class/mstar/isp0/isp_ints frame_count : 1089 u32FrameIntervalNanoSec : 33332 us ISP_FIFO_FULL : 0 INT_PIPE1_FIFO_FULL : 0 ISP_BUSY : 1089, 06.952374 ISP_IDLE : 1090, 06.950575 RDMA0_DONE : 0 RDMA1_DONE : 0 WDMA0_DONE : 0 WDMA1_DONE : 0 WDMA6_LINE_HIT : 0 WDMA6_DONE : 0 ISPIF_VSYNC : 1089, 06.952046, 0 AE_DONE : 1088, 06.950279, 1934 AE_DONE period : 31565 us
4.3. Verify shutter and gain delay¶
If you want to verify that the sensor AE gain/shutter delay, you can use apitool, also follow the steps below to control the sensor AE gain/shutter function step by step.
STEP 1. You need to execute process of mixer first
/customer # ./mixer -q -n 1 (linear mode) /customer # ./mixer -q -n 1 -N 3_2 (hdr mode)
STEP 2. Open IQ tool in sdk path “project/tools/apitool”
STEP 3. Host Name is EVB’s IP address, check it then connect to EVB
STEP 4. Select Macaron AE option, and select “ExposureMode” menu table to M_Mode
Step 5. Adjust “DebugLev” menu and set AE Debug Level from 0 to 64, then you can check you console log
Check the AE message shown on the console
Step 6. Adjust AE sensor gain or shutter manually. At the same time, you can observe the AE message shown on the console.
Observer the value of “CurYx10” change from 308 to 1436
After we set the gain of AE through ISP tool, it will take effect in the next frame. The CurYx10 represents the brightness information of a previous frame. The following figure is an example. The AE gain delay needs to set 2.
Figure 8: AE Y value response time
4.4. Verify shutter and gain linearity¶
If you want to verify that the sensor AE gain/shutter function is normal after porting the sensor AE gain/shutter function finish, you can use apitool. You can follow the steps below to control the sensor AE gain/shutter function step by step.
STEP 1. You need to execute process of mixer
/customer # ./mixer -q -n 1 (linear mode) /customer # ./mixer -q -n 1 -N 3_2 (hdr mode)
STEP 2. Open IQ tool in sdk path “project/tools/apitool”
STEP 3. Host Name is EVB’s IP address, check it then connect to EVB
STEP 4. Select Macaron AE option, and select “ExposureMode” menu table to M_Mode
Step 5. Adjust AE sensor gain and shutter manually. At the same time, you can observe image changes through VLC or your debug message.
The IQ Tool has defined upper and lower limit for the AE settings, you can adjust it according to your needs. You can reference Exposure limit table.
4.5. Skip frame feature¶
The sensor driver that has been implemented has imx307. You can refer to the sensor driver of imx307. Here I will briefly explain how to use skip frame function.
Step 1. Add a variable "skip _cnt" inside Params structure of sensor driver
u32 max_rhs1; u32 lef_shs2; u32 skip_cnt; bool dirty; bool change; I2C_ARRAY tVts_reg[3]; I2C_ARRAY tGain_reg[3]; I2C_ARRAY tExpo_reg[3]; I2C_ARRAY tShs2_reg[3]; I2C_ARRAY tRhs1_reg[3]; I2C_ARRAY tGain_hdr_dol_lef_reg[1]; I2C_ARRAY tGain_hdr_dol_sef_reg[1]; bool orien_dirty; } imx307_params;
Step 2. Set the number of skip frames in the function that needs to skip frames.
For an example of imx307, in HDR mode, it may flicker when changing fps (from 10 fps to 30 fps) in high brightness environment, so skip 1 frame to avoid flicker.
if(params->expo.expo_lines > 2 * params->expo.vts - params->max_rhs1 -3){ vts = (params->expo.expo_lines + params->max_rhs1 + 3 + 1) / 2; }else{ vts = params->expo.vts; } params->expo.vts = vts; pCus_SetAEUSecsHDR_DOL_SEF1(handle, params->expo.expo_sef_us); params->skip_cnt = 1; return SUCCESS;
Step 3. The actual time to skip frame in function of “AESStatusNotify”. Please add the following features. SensorIF will calculate the time to set the VIF mask disable based on the number of skip frames and the current fps time period.
if(params->skip_cnt){ sensor_if->SetSkipFrame(handle->snr_pad_group, params->expo.fps, params->skip_cnt); params->skip_cnt = 0; } break; default : break; } return SUCCESS;
4.6. Sensor post initial feature (Special case for SmartSens)¶
Since the Smartsens sensor first transmit i2c cmd after power-on, the probability of the sensor mipi csi and clock will be pulled down, it leads to SoC mipi csi abnormal.
Figure 9: Smartsense sensor signal
Currently, the solution is to change the sequence of the sensor power-on and the sensor initialization. After the sensor initialize completed, then enable SoC mipi csi to resolve this problem. Please refer to the following process:
Figure 10: power on sequence flow change