Image fusion in a man-portable multi-sensor optoelectronic device based on an embedded system on module
9 viewsKeywords:
Multi-sensor optoelectronic system; Image fusion; Embedded System-on-Module (SoM).Abstract
Man-portable optoelectronic devices combining visible (or/and low-light) and infrared (IR) sensors require real-time image fusion to enhance situational awareness under degraded visual conditions, while operating under strict size, weight, power consumption, and cost (SWaP-C) constraints. This paper presents the design and implementation of a real-time EO/IR image fusion system based on an embedded System-on-Module (SoM) architecture for a handheld multi-sensor platform. The proposed system supports synchronized acquisition from multiple visible and IR cameras and executes parallel fusion processing using a multi-core CPU with GPU/NPU acceleration. Experimental results demonstrate real-time performance with end-to-end processing latency below 40 ms, sustained frame rates of 25 fps for fused video output. The fused image exhibits better contrast, clearer details, and improved target differentiation compared to the original input images. The experiments indicate that SoM-based architectures can provide a balanced and practical solution for real-time multi-channel EO/IR image fusion in man-portable applications.
References
[1]. Zhang, J., "Multi-Source Remote Sensing Data Fusion: Status and Trends", International Journal of Image and Data Fusion, Vol. 1, pp. 5–24, (2010).
[2]. Li, H., Qi, X., and Xie, W., "Fast Infrared and Visible Image Fusion with Structural Decomposition", Knowledge-Based Systems, Vol. 204, p. 106182, (2020).
[3]. Zhang, S., Huang, F., Liu, B., Li, G., Chen, Y., Chen, Y., Zhou, B., and Wu, D., "A Multi-Modal Image Fusion Framework Based on Guided Filter and Sparse Representation", Optics and Lasers in Engineering, Vol. 137, p. 106354, (2021).
[4]. N. A. Tuấn, "Ứng dụng thuật toán tách biên hình ảnh Canny cho thiết bị quan sát trộn ảnh quang học", Tạp chí Nghiên cứu KH&CN quân sự, Số Đặc san FEE, tr. 273–280, (2019) (in Vietnamese).
[5]. T. V. Ha et al., "Block Design of Edge Detection and Color Blending of Thermal Video on the Pynq-Z1 Kit Using Vivado HLS for Image Fusion Applications", Journal of Military Science and Technology, No. 77, pp. 128–136, (2022).
[6]. V. N. Le et al., "A Night Vision Color Image Fusion Method Based on Statistical Color Transform Technology in YUV Color Space", Journal of Military Science and Technology, Vol. 93, No. 93, pp. 114–120, (2024).
[7]. A. Boutros and V. Betz, "FPGA Architecture: Principles and Progression", IEEE Circuits and Systems Magazine, Vol. 21, No. 2, pp. 4–29, (2021).
[8]. Ling Zhang, Xuefei Yang, Zhenlong Wan, Dingxin Cao, and Yingcheng Lin, "A Real-Time FPGA Implementation of Infrared and Visible Image Fusion Using Guided Filter and Saliency Detection", Sensors, Vol. 22, p. 8487, (2022).
[9]. Seung-Hun Yoo et al., "Accelerating Multi-Scale Image Fusion Algorithms Using CUDA", IEEE International Conference of Soft Computing and Pattern Recognition, (2009).
[10]. Changqi Sun, Cong Zhang, and Naixue Xiong, "Infrared and Visible Image Fusion Techniques Based on Deep Learning: A Review", Electronics, Vol. 9, p. 2162, (2020).
[11]. Tao Yang, Xinyu Zhang, Qingbo Xu, Shuangxi Zhang, and Tong Wang, "An Embedded-GPU-Based Scheme for Real-Time Imaging Processing of Unmanned Aerial Vehicle Borne Video Synthetic Aperture Radar", Remote Sensing, Vol. 16, No. 1, p. 191, (2024).
[12]. “Rockchip RK3588 Datasheet”, Rockchip Electronics Co., Ltd., (2023).
[13]. Song, Y., Gao, K., Ni, G., and Lu, R., "Implementation of Real-Time Laplacian Pyramid Image Fusion Processing Based on FPGA", Proceedings of the Electronic Imaging and Multimedia Technology V, Beijing, China, (2007).
[14]. Liangliang Li, Yan Shi, Ming Lv, Zhenhong Jia, Minqin Liu, Xiaobin Zhao, Xueyu Zhang, and Hongbing Ma, "Infrared and Visible Image Fusion via Sparse Representation and Guided Filtering in Laplacian Pyramid Domain", Remote Sensing, Vol. 16, p. 3804, (2024).
[15]. Jiayi Ma, Yong Ma, and Chang Li, "Infrared and Visible Image Fusion Methods and Applications: A Survey", Information Fusion, (2018).
