MI BF API
1. Overview¶
1.1. Algorithm Description¶
BF (short for Beamforming) or spatial filtering is a directional signal processing technology for sensor array transmission or reception. Strengthen the signal to achieve a specific angle combination of components in the microphone array in such a way, while others will be attenuated. Beamforming can be used at the transmitter and receiver to achieve spatial selectivity.
1.2. Keyword¶
1.2.1. Noise Gate¶
Noise Gate (dBFS), below this value, the frame will be treated as noise.
If the setting is too high, it will easily cause speech distortion. If the setting is too low, it will cause Beamforming result not good enough. Please be careful to use this parameter. This parameter must be referred to the actual recording volume of microphones. If the volume is too low and this parameter is set to a large value, the frame will be treated as noise. The value range is [-80, 0], the recommend value is -20, and the step size is 1.
1.2.2. Diagonal loading¶
By adding a diagonal loading to the diagonal elements of the matrix for regularization, it can be ensured that the inverse matrix must exist and hence improve the robustness of Beamforming. Please be careful to use this parameter. If the setting is too high, it will cause Beamforming result not good enough. If the setting is too low, it will cause speech distortion. The value range is [100, 1], the recommend value is 10, and the step size is 1.
1.3. Note¶
1.3.1. Implementation¶
In order to facilitate debugging and confirm the effect of the algorithm, the user application needs to implement replacement algorithm parameter and grab audio data.
1.3.2. Cooperation¶
With the help of Audio Process Chain, Beamforming will have better signal enhancement and interference suppression effect.
1.3.3. Correspondence¶
Different channel numbers will correspond to different reference libraries. The user needs to confirm whether to use the correct number of channels and the corresponding reference library.
2. SPECIFICATION¶
-
The number of microphone is two or more.
-
The microphones should be the same type.
-
The microphones gain should be the same.
-
For best performance, background environment should be quiet.
-
If you are using audio files as the sound source , you should make sure that there is no signal clipping in audio files, see Figure 1.
Figure 1: Audio example of clipping
3. Coordinate system of Microphone Array¶
3.1. Multichannel Microphone array¶
Compared to the system consisting of a dual microphone array, multichannel microphone array system is capable of reaching better spatial selectivity at a specific angle because it has better bandwidth resolution and more microphone weights could be controlled. We denotes the specific angle (the angle we want to improve) microphone_doa that represents the direction of sound. There are two main types of multichannel array system used in practice, one is the uniform linear array, and the other is uniform circular array.
3.1.1. Uniform Linear Array¶
Uniform linear array is a straight line array that is spaced evenly. Because of its DOA symmetry, we only consider the microphone_doa locates at the upper plane (from -90 degree to 90 degree). Figure 2 illustrates the uniform linear array and its coordinate system. The microphone_doa is defined as the angle between the array center and the x-axis where counterclockwise is positive. We recommend the distance of adjacent microphones should be bigger than 5cm or 6cm.
Figure 2: Uniform linear array and its coordinate system
3.1.2. Uniform Circular Array¶
Uniform circular array is a circular array that the angle between adjacent microphones and array center are the same. Because of its DOA asymmetry, we consider the microphone_doa locates at the whole plane (from -90 degree to 270 degree). Figure 3 illustrates the uniform circular array and its coordinate system. The microphone_doa is defined as the angle between the array center and the x-axis where counterclockwise is
positive. The distance is the diameter of the circle. We recommend the distance should be bigger than 6cm.
Figure 3: Uniform circular array and its coordinate system
4. API Reference¶
4.1. API List¶
API name | Features |
---|---|
IaaBf_GetBufferSize | Get the memory size required for Bf algorithm running |
IaaBf_Init | Initialize Bf algorithm |
IaaBf_SetConfig | Configure Bf algorithm |
IaaBf_GetConfig | Get the current configuration parameter information of the Bf algorithm |
IaaBf_SetShape | Assign the array shape of BF. |
IaaBf_Run | Bf algorithm processing |
IaaBf_Reset | Reinitialize Bf algorithm |
IaaBf_Free | Release Bf algorithm resources |
4.2. IaaBf_GetBufferSize¶
-
Features
Get the memory size required for Bf algorithm running.
-
Syntax
unsigned int IaaBf_GetBufferSize(void);
-
Return value
Return value is the memory size required for Bf algorithm running.
-
Dependency
-
Header: AudioBfProcess.h
-
Library: libBF_LINUX.so/ libBF_LINUX.a
-
-
Note
The interface only returns the required memory size, and the application and release of memory need to be processed by the application.
-
Example
Please refer to IaaBf_Run example.
4.3. IaaBf_Init¶
-
Features
Initialize the Bf algorithm.
-
Syntax
F_HANDLE IaaBf_Init(char* working_buffer,AudioBfInit* bf_init);
-
Parameters
Parameter Name Description Input/Output working_buffer Memory address used by Bf algorithm Input bf_init Bf algorithm initialization structure pointer Input -
Return value
Return value Result Not NULL Successful NULL Failed -
Dependency
-
Header: AudioBfProcess.h
-
Library: libBF_LINUX.so/ libBF_LINUX.a
-
-
Example
Please refer to IaaBf_Run example.
4.4. IaaBf_SetConfig¶
-
Features
Configure Bf algorithm.
-
Syntax
ALGO_BF_RET IaaBf_SetConfig(BF_HANDLE handle,AudioBfConfig* bf_config);
-
Parameters
Parameter Name Description Input/Output handle Bf algorithm handle Input bf_config Bf algorithm configuration structure pointer Input -
Return value
Return value Result 0 Successful Non-zero Failed, refer to Error code -
Dependency
-
Header: AudioBfProcess.h
-
Library: libBF_LINUX.so/ libBF_LINUX.a
-
-
Example
Please refer to IaaBf_Run example.
4.5. IaaBf_GetConfig¶
-
Features
Get the current configuration parameter information of the Bf algorithm.
-
Syntax
ALGO_BF_RET IaaBf_GetConfig(BF_HANDLE handle,AudioBfConfig* bf_config);
-
Parameters
Parameter Name Description Input/Output handle Bf algorithm handle Input bf_config Bf algorithm configuration structure pointer Output -
Return value
Return value Result 0 Successful Non-zero Failed, refer to Error code -
Dependency
-
Header: AudioBfProcess.h
-
Library: libBF_LINUX.so/ libBF_LINUX.a
-
-
Example
Please refer to IaaBf_Run example.
4.6. IaaBf_SetShape¶
-
Features
Assign the array shape of BF.
-
Syntax
ALGO_BF_RET IaaBf_Setshape(BF_HANDLE handle,int shape);
-
Parameters
Parameter Name Description Input/Output handle Bf algorithm handle Input shape The integer to decide array shape of BF. 0: Uniform Linear Array. 1: Uniform Circular Array. Input -
Return value
Return value Result 0 Successful Non-zero Failed, refer to Error code -
Dependency
-
Header: AudioBfProcess.h
-
Library: libBF_LINUX.so/ libBF_LINUX.a
-
-
Note
-
Our lib only supports uniform linear array and uniform circular array (Figure 2 and Figure 3). User has to inform the modification if he/she requires special array geometry.
-
Only uniform linear array exists when there are only two microphones.
-
The settings of array position will impact largely on Beamforming performance. Therefore, the settings of array position must be matched to the microphone array being used.
-
-
Example
Please refer to IaaBf_Run example.
4.7. IaaBf_Run¶
-
Features
Bf algorithm processing.
-
Syntax
ALGO_BF_RET IaaBf_Run(BF_HANDLE handle, short* microphone_input, short* microphone_output,float* microphone_doa);
-
Parameters
Parameter Name Description Input/Output handle Bf algorithm handle Input microphone_input The microphone raw data. Input microphone_output Output data after Beamforming. Output microphone_doa Beamforming processing angle. This value could be set by user or calculated from SSL (Need one more transformation). Set 0 for sound coming from forward,90 for sound coming from left, and -90 for sound coming from right. Input -
Return value
Return value Result 0 Successful Non-zero Failed, refer to Error code -
Dependency
-
Header: AudioBfProcess.h
-
Library: libBF_LINUX.so/ libBF_LINUX.a
-
-
Note
-
For dual microphone array (the microphone input is binaural data), the data pointed to microphone_input should use the sampling point as the smallest unit and be placed in the format of L,R,L,R … .The length must correspond to the point_number (the number of sampling points once beamforming process) set in IaaBf_Init.
-
For multichannel microphone array (microphone number is bigger than two where binaural data is not enough for processing), the input data of each microphone must be mono-channel. The data pointed to microphone_input should use the sampling point as the smallest unit and be placed in the format of [Left → Right], according to the relative position. The length must correspond to the point_number (the number of sampling points once beamforming process) set in IaaBf_Init.
-
Take uniform linear array in Figure 2 as example, the microphone_input should be placed in the format of [MIC1→MIC2→MIC3→MIC4].
-
The data length of microphone_output must correspond to the point_number set in IaaBf_init.
For 4MIC array, If point_number =128points→ microphone_input = 128x4 = 512points, and microphone_output = 128points.
-
-
Example
Example I: Dual Microphone array(The microphone input is a binaural audio)
1. #include <stdio.h> 2. #include <unistd.h> 3. #include <fcntl.h> 4. #include <string.h> 5. #include <sys/time.h> 6. #include <sys/ioctl.h> 7. #include <stdlib.h> 8. #include "AudioBfProcess.h" 9. 10. #define USE_MALLOC (1) 11. typedef unsigned char uint8; 12. typedef unsigned short uint16; 13. typedef unsigned long uint32; 14. 15. float AVERAGE_RUN(int a) 16. { 17. static unsigned int num = 0; 18. static float avg = 0; 19. if(num == 0) avg = 0; 20. num++; 21. avg = avg + ((float)a - avg) / ((float)num); 22. return avg; 23. } 24. unsigned int _OsCounterGetMs(void) 25. { 26. struct timeval t1; 27. gettimeofday(&t1,NULL); 28. unsigned int T = ( (1000000 * t1.tv_sec)+ t1.tv_usec ) / 1000; 29. return T; 30. } 31. 32. int main(int argc, char *argv[]) 33. { 34. short input[256]; 35. short output[128]; 36. char infileName[512]; 37. char outfileName[512]; 38. 39. FILE * fin, * fout; 40. int ret; 41. int shape = 0; 42. float direction = 0.0; 43. int delay_sample = 0; 44. float avg = 0; 45. int counter = 0; 46. unsigned int T0,T1; 47. AudioBfInit bf_init; 48. AudioBfConfig bf_config; 49. BF_HANDLE handle; 50. bf_init.mic_distance = 8.0; 51. bf_init.point_number = 128; 52. bf_init.sample_rate = 16000; 53. bf_init.channel = 2; 54. bf_config.noise_gate_dbfs = -20; 55. bf_config.temperature = 20; 56. bf_config.noise_estimation = 0; 57. bf_config.output_gain = 0.7; 58. bf_config.vad_enable = 0; 59. bf_config.diagonal_loading = 10; 60. #if USE_MALLOC 61. char *WorkingBuffer2; 62. WorkingBuffer2 = (char*)malloc(IaaBf_GetBufferSize()); 63. #endif 64. handle = IaaBf_Init((char*)WorkingBuffer2, &bf_init); 65. if (handle==NULL) 66. { 67. printf("BF init error\r\n"); 68. return -1; 69. } 70. else 71. { 72. printf("BF init succeed\r\n"); 73. } 74. ret = IaaBf_SetConfig(handle,&bf_config); 75. if (ret) 76. { 77. printf("Error occured in Config\n"); 78. return -1; 79. } 80. ret = IaaBf_SetShape(handle,shape); 81. if (ret) 82. { 83. printf("Error occured in Array shape\n"); 84. return -1; 85. } 86. sprintf(infileName,"%s","./Chn-14.wav"); 87. sprintf(outfileName,"%s","./BFOut.pcm"); 88. 89. fin = fopen(infileName, "rb"); 90. if(!fin) 91. { printf("the input file could not be open\n"); 92. return -1; 93. } 94. 95. fout = fopen(outfileName, "wb"); 96. if(!fout) 97. { 98. printf("the output file could not be open\n"); 99. return -1; 100. } 101. 102. fread(input, sizeof(char), 44, fin); // read header 44 bytes 103. //fwrite(input, sizeof(char),44, fout); // write 44 bytes output 104. // int delay_sample = 0; 105. while(fread(input, sizeof(short), bf_init.point_number*bf_init.channel, fin)) 106. { 107. counter++; 108. T0 = (long)_OsCounterGetMs(); 109. ret = IaaBf_Run(handle,input,output,&direction); 110. T1 = (long)_OsCounterGetMs(); 111. avg += (T1 - T0); 112. if(ret) 113. { 114. printf("Error occured in Run\n"); 115. return -1; 116. } 117. fwrite(output, sizeof(short), bf_init.point_number, fout); 118. } 119. avg /= counter; 120. printf("AVG is %.2f ms\n",avg); 121. IaaBf_Free(handle); 122. fclose(fin); 123. fclose(fout); 124. free(WorkingBuffer2); 125. printf("Done\n"); 126. return 0; 127. }
Example II: Multichannel Microphone array (Microphone input are 4x mono-channel audio)
128. #include <stdio.h> 129. #include <unistd.h> 130. #include <fcntl.h> 131. #include <string.h> 132. #include <sys/time.h> 133. #include <sys/ioctl.h> 134. #include <stdlib.h> 135. #include "AudioBfProcess.h" 136. 137. #define MIC_NUM (4) 138. #define USE_MALLOC (1) 139. typedef unsigned char uint8; 140. typedef unsigned short uint16; 141. typedef unsigned long uint32; 142. 143. float AVERAGE_RUN(int a) 144. { 145. static unsigned int num = 0; 146. static float avg = 0; 147. if(num == 0) avg = 0; 148. num++; 149. avg = avg + ((float)a - avg) / ((float)num); 150. return avg; 151. } 152. unsigned int _OsCounterGetMs(void) 153. { 154. struct timeval t1; 155. gettimeofday(&t1,NULL); 156. unsigned int T = ( (1000000 * t1.tv_sec)+ t1.tv_usec ) / 1000; 157. return T; 158. } 159. 160. int main(int argc, char *argv[]) 161. { 162. /*********Input file init*******/ 163. short input[512]; 164. short output[128]; 165. short input_tmp1[128],input_tmp2[128],input_tmp3[128],input_tmp4[128]; 166. FILE * fin0, * fin1, * fin2, * fin3, * fout; 167. 168. char infileName[MIC_NUM][512]; 169. char outfileName[512]; 170. int ret,k; 171. float avg = 0; 172. int counter = 0; 173. unsigned int T0, T1; 174. /********common setting between SSL and BF********/ 175. int point_number = 128; 176. float microphone_distance = 4.0; 177. int temperature = 20; 178. int sample_rate = 16000; 179. int shape = 0; 180. // int delay_sample[MIC_NUM-1] = {0,0,0}; //channel-1 181. float direction = 0.0; 182. /*******BF data init*********/ 183. #if USE_MALLOC 184. char *WorkingBuffer_BF; 185. WorkingBuffer_BF = (char*)malloc(IaaBf_GetBufferSize()); 186. #endif 187. 188. AudioBfInit bf_init; 189. AudioBfConfig bf_config; 190. BF_HANDLE bf_handle; 191. bf_init.mic_distance = microphone_distance; 192. bf_init.point_number = point_number; 193. bf_init.sample_rate = sample_rate; 194. bf_init.channel = MIC_NUM; 195. bf_config.noise_gate_dbfs = -20; 196. bf_config.temperature = temperature; 197. bf_config.noise_estimation = 0; 198. bf_config.output_gain = 0.7; 199. bf_config.vad_enable = 0; 200. bf_config.diagonal_loading = 10; 201. bf_handle = IaaBf_Init((char*)WorkingBuffer_BF, &bf_init); 202. if (bf_handle==NULL) 203. { 204. printf("BF init error\r\n"); 205. return -1; 206. } 207. else 208. { 209. printf("BF init succeed\r\n"); 210. } 211. ret = IaaBf_SetConfig(bf_handle,&bf_config); 212. if (ret) 213. { 214. printf("Error occured in Config\n"); 215. return -1; 216. } 217. ret = IaaBf_SetShape(bf_handle,shape); 218. if (ret) 219. { 220. printf("Error occured in Array shape\n"); 221. return -1; 222. } 223. /********open input file and output file*****/ 224. sprintf(infileName[0],"%s","./Chn-01.wav"); 225. sprintf(infileName[1],"%s","./Chn-02.wav"); 226. sprintf(infileName[2],"%s","./Chn-03.wav"); 227. sprintf(infileName[3],"%s","./Chn-04.wav"); 228. sprintf(outfileName,"%s","./BFOut.pcm"); 229. 230. fin0 = fopen(infileName[0], "rb"); 231. if(!fin0) 232. { 233. printf("the input file 0 could not be open\n"); 234. return -1; 235. } 236. 237. fin1 = fopen(infileName[1], "rb"); 238. if(!fin1) 239. { 240. printf("the input file 1 could not be open\n"); 241. return -1; 242. } 243. fin2 = fopen(infileName[2], "rb"); 244. if(!fin2) 245. { 246. printf("the input file 2 could not be open\n"); 247. return -1; 248. } 249. fin3 = fopen(infileName[3], "rb"); 250. if(!fin3) 251. { 252. printf("the input file 3 could not be open\n"); 253. return -1; 254. } 255. fout = fopen(outfileName, "wb"); 256. if(!fout) 257. { 258. printf("the output file could not be open\n"); 259. return -1; 260. } 261. fread(input, sizeof(char), 44, fin0); 262. fread(input, sizeof(char), 44, fin1); 263. fread(input, sizeof(char), 44, fin2); 264. fread(input, sizeof(char), 44, fin3); 265. short * input_ptr; 266. 267. while(fread(input_tmp1, sizeof(short), bf_init.point_number, fin0)) 268. { 269. fread(input_tmp2, sizeof(short), bf_init.point_number, fin1); 270. fread(input_tmp3, sizeof(short), bf_init.point_number, fin2); 271. fread(input_tmp4, sizeof(short), bf_init.point_number, fin3); 272. input_ptr = input; 273. for(k=0;k<point_number;k++) 274. { 275. *input_ptr = input_tmp1[k]; 276. input_ptr++; 277. *input_ptr = input_tmp2[k]; 278. input_ptr++; 279. *input_ptr = input_tmp3[k]; 280. input_ptr++; 281. *input_ptr = input_tmp4[k]; 282. input_ptr++; 283. } 284. counter++; 285. T0 = (long)_OsCounterGetMs(); 286. ret = IaaBf_Run(bf_handle,input,output,&direction); 287. T1 = (long)_OsCounterGetMs(); 288. avg += (T1 - T0); 289. if(ret) 290. { 291. printf("Error occured in Run\n"); 292. return -1; 293. } 294. fwrite(output, sizeof(short),point_number, fout); 295. } 296. avg /= counter; 297. printf("AVG is %.2f ms\n",avg); 298. IaaBf_Free(bf_handle); 299. fclose(fin0); 300. fclose(fin1); 301. fclose(fin2); 302. fclose(fin3); 303. fclose(fout); 304. free(WorkingBuffer_BF); 305. printf("Done\n"); 306. 307. return 0; 308. }
4.8. IaaBf_Reset¶
-
Features
Reinitialize Bf algorithm.
-
Syntax
BF_HANDLE IaaBf_Reset(BF_HANDLE working_buffer,AudioBfInit* bf_init);
-
Parameters
Parameter Name Description Input/Output working_buffer Bf algorithm running memory address Input bf_init Bf algorithm initialization structure pointer Input -
Return value
Return value Result Not NULL Successful NULL Failed -
Dependency
-
Header: AudioBfProcess.h
-
Library: libBF_LINUX.so/ libBF_LINUX.a
-
4.9. IaaBf_Free¶
-
Features
Release Bf algorithm resources.
-
Syntax
ALGO_BF_RET IaaBf_Free(BF_HANDLE handle);
-
Parameters
Parameter Name Description Input/Output handle Bf algorithm handle Input -
Return value
Return value Result 0 Successful Non-zero Failed, refer to Error code -
Dependency
-
Header: AudioBfProcess.h
-
Library: libBF_LINUX.so/ libBF_LINUX.a
-
-
Example
Please refer to IaaBf_Run example.
5. BF Data Type¶
5.1. BF data type list¶
DATA TYPE | Description |
---|---|
AudioBfInit | Bf algorithm initialization parameter structure type |
AudioBfConfig | Bf algorithm configuration parameter structure type |
BF_HANDLE | Bf algorithm handle type |
5.2. AudioBfInit¶
-
Description
Define Bf algorithm initialization parameter structure type.
-
Definition
typedef struct { unsigned int point_number; unsigned int sample_rate; float mic_distance; unsigned int channel; }AudioBfInit;
-
Member
Member name Description point_number The sampling points that Bf algorithm processed once sample_rate Sampling rate, currently supports 8k/16k. mic_distance The distance between adjacent mics, unit: cm, recommend: 5cm or 6cm channel Number of channels -
Related data types and interfaces
5.3. AudioBfConfig¶
-
Description
Define Bf algorithm configuration parameter structure type
-
Definition
typedef struct { unsigned int temperature; int noise_gate_dbfs; int noise_estimation; float output_gain; int vad_enable; int diagonal_loading; }AudioBfConfig;
-
Member
Member name Description temperature Ambient temperature (Celsius) Celsius = (5/9) * (Fahrenheit-32) Step size is 1 noise_gate_dbfs If the power of signal is smaller than this setting, this frame will take this frame as noise part, and use this frame to calculate some estimation. Please be careful to use this parameter. This parameter must be referred to the actual recording volume of microphones. If setting too high, it will cause the speech distortion. If setting too small, it will cause BF result not good enough. Recommend: -20, step size: 1. noise_estimation Noise estimation mode, Input: 0 or 1. 0 for adaptation method. 1 for average method. Recommend to use 0 for normal application. output_gain Output signal gain control (range : 0 ~ 1). 0.7 is the normal case. Setting as 1 will add 4db. Recommend: 0.7 (nothing change). vad_enable Whether voice alive detection (VAD) mode is enable. Input: 0 or 1. If this setting is enable, VAD is used to calculate whether each frame enters the noise estimation process. If this setting is disable, the noise_gate_dbfs set above is used to determine whether it is treated as noise. diagonal_loading Regularization term for inverse matrix. Please be careful to use this parameter. If setting too high, it will cause BF result not good enough. If setting too low, it will cause the speech distortion. Recommend: 10. -
Related data types and interfaces
5.4. BF_HANDLE¶
-
Description
Bf algorithm handle type.
-
Definition
typedef void* BF_HANDLE;
-
Related data types and interfaces
6. Error code¶
BF API error codes are shown as follow:
Error code | Definition | Description |
---|---|---|
0x00000000 | ALGO_BF_RET_SUCCESS | BF runs successfully |
0x10000301 | ALGO_BF_RET_INIT_ERROR | BF initialization error |
0x10000302 | ALGO_BF_RET_INVALID_CONFIG | BF Config is invalid |
0x10000303 | ALGO_BF_RET_INVALID_HANDLE | BF Handle is invalid |
0x10000304 | ALGO_BF_RET_INVALID_NOISE_ESTIMATION | BF noise estimation mode setting is invalid |
0x10000305 | ALGO_BF_RET_INVALID_VAD_ENABLE | BF VAD mode setting is invalid |
0x10000306 | ALGO_BF_RET_INVALID_OUTPUT_GAIN | BF output gain setting is invalid |
0x10000307 | ALGO_BF_RET_INVALID_INPUT_POINTER | BF input indicator is invalid |
0x100003089 | ALGO_BF_RET_INVALID_SAMPLERATE | BF sampling rate is invalid |
0x10000309 | ALGO_BF_RET_INVALID_POINTNUMBER | BF sampling point is invalid |
0x10000310 | ALGO_BF_RET_INVALID_CHANNEL | BF channel number is invalid |
0x10000311 | ALGO_BF_RET_INVALID_CALLING | BF call API sequence error |
0x10000312 | ALGO_BF_RET_API_CONFLICT | Other APIs are running |
0x10000313 | ALGO_BF_RET_INVALID_GEOMETRY_TYPE | BF array shape is invalid. |
0x10000314 | ALGO_BF_RET_INVALID_MIC_DISTANCE | BF mic distance is invalid |
7. Flow¶
Figure 4: Beamforming working flow