Machine learning algorithm partially reconfigured on FPGA for an image edge detection system

Gracieth Cavalcanti Batista; Johnny Öberg; Osamu Saotome; Haroldo F. de Campos Velho; Elcio Hideiti Shiguemori; Ingemar Söderquist

doi:10.1016/j.jnlest.2024.100248

Journal of Electronic Science and Technology ›› 2024, Vol. 22 ›› Issue (2) :100248 DOI: 10.1016/j.jnlest.2024.100248

research-article

Machine learning algorithm partially reconfigured on FPGA for an image edge detection system

Author information +

^a Division of Electronic and Embedded Systems, KTH Royal Institute of Technology, Stockholm, 164 40, Sweden

^b Electronic Engineering Division, Aeronautics Institute of Technology, São José dos Campos, SP, 12228-900, Brazil

^c Laboratory of Applied Computing and Mathematics, National Institute for Space Research, São José dos Campos, SP 12227-900, Brazil

^d Department of C4ISR, Institute of Advanced Studies, São José dos Campos, SP 12228-001, Brazil

^e Saab AB, Linköping, 581 88, Sweden

^* E-mail addresses: gracieth@kth.se (G.C. Batista),

johnnyob@kth.se (J. Öberg),

osaotome@ita.br (O. Saotome),

haroldo.camposvelho@inpe.br (H.F. de Campos Velho),

elciohs@gmail.com (E.H. Shiguemori),

ingemar.soderquist@saabgroup.com (I. Söderquist).

Gracieth Cavalcanti Batista obtained the B.S. degree in electrical engineering from Federal Institute of Maranhão, São Luís, Brazil in 2017 with a sandwich period at Lakehead University, Thunder Bay, Canada in 2015 through the Science without Borders Program. She obtained her M.S. degree in electronic engineering and computing from Aeronautics Institute of Technology (ITA), São José dos Campos, Brazil in 2020. Currently, she is pursuing her double-degree Ph.D. degrees in information and communications technology with KTH Royal Institute of Technology, Stockholm, Sweden and in electronic systems with ITA. Her research interests include machine learning, optimization of synchronous and asynchronous architectures, and discrete signal processing.

Johnny Öberg received his M.S. and Ph.D. degrees from KTH Royal Institute of Technology in 1993 and 1999, respectively. He is currently working as an associate professor with the School of Electrical Engineering and Computer Science, Division of Electronics and Embedded Systems, KTH Royal Institute of Technology. His current research interests include design methods for network-on-chips, application-specific integrated circuit (ASIC)/field programmable gate arrays (FPGAs), multi-/many-core architectures, hardware accelerators, and brain-scale computing.

Osamu Saotome got his B.S. degree in electronic engineering from ITA in 1974. He obtained his M.S. and Ph.D. degrees from Tokyo Institute of Technology, Tokyo, Japan in 1984 and 1987, respectively, both in electrical, electronic, and information engineering focusing on the field of digital signal processing. Currently, he is a professor with the Electronic Engineering Division, ITA. His main research interest focuses on artificial intelligence in microelectronic and reconfigurable field programmable logic devices.

Haroldo F. de Campos Velho received his B.S. degree in chemical engineering from Pontifical Catholic University of Rio Grande do Sul, Porto Alegre, Brazil in 1982. He obtained the M.S. and Ph.D. degrees in mechanical engineering from Federal University of Rio Grande do Sul, Porto Alegre, Brazil in 1988 and 1992, respectively. He was a visiting scientist with Istituto di Cosmo-Geofisica, Turim, Italy in 1999 and the Department of Atmospheric Science, Colorado State University, Fort Collins, USA in 1998. Currently, he is a senior researcher with National Institute for Space Research (INPE), São José dos Campos, Brazil. His research interests include inverse problems, mathematical modeling for atmospheric turbulence, data assimilation, scientific computing, and machine learning applications.

Elcio Hideiti Shiguemori received the M.S. and Ph.D. degrees in applied computing from INPE in 2002 and 2007, respectively. He has been a senior researcher with Institute of Advanced Studies, São José dos Campos, Brazil since 2005. He has also been working with Universidade Paulista, São Paulo, Brazil, as a resident professor since 2008. Currently, he is also a professor with ITA. His research interests include autonomous aerial navigation, computer vision, machine learning, and real-time systems.

Ingemar Söderquist received the M.S. degree in electronic and electrical engineering and the Ph.D. degree in electronic devices from University of Linköping, Linköping, Sweden in 1985 and 2002, respectively. From 1985 to 2000, he was with Saab Dynamics AB, where he focused on the analysis and development of high-performance signal processing, radar systems, and microwave products. From 2000 to 2003, he was with Saab Avionics AB, where he worked as a manager of the Digital RF Systems Group, Electronic Warfare Division. In 2003, he moved to Saab AB, Business Area Aeronautics, working as a project manager in the product area recording and monitoring systems. Currently, he is working with a broad-based view of electronics and software in avionic computers. In 2008, he was appointed to the position of Technical Fellow within the strategic area of electronics design in avionics. He is also working with the Department of Electrical Engineering, Linköping University. Since 2022, he has also been an adjunct professor in embedded systems in avionics with KTH Royal Institute of Technology. His research interests include high-performance circuits and systems in complementary metal-oxide-semiconductor (CMOS), failure modes and mechanisms in electronic systems, reliable computer architectures, and high-performance embedded real-time systems.

Show less

History +

PDF (3226KB)

Abstract

Unmanned aerial vehicles (UAVs) have been widely used in military, medical, wireless communications, aerial surveillance, etc. One key topic involving UAVs is pose estimation in autonomous navigation. A standard procedure for this process is to combine inertial navigation system sensor information with the global navigation satellite system (GNSS) signal. However, some factors can interfere with the GNSS signal, such as ionospheric scintillation, jamming, or spoofing. One alternative method to avoid using the GNSS signal is to apply an image processing approach by matching UAV images with georeferenced images. But a high effort is required for image edge extraction. In this paper, a support vector regression (SVR) model is proposed to reduce this computational load and processing time. The dynamic partial reconfiguration (DPR) of part of the SVR datapath is implemented to accelerate the process, reduce the area, and analyze its granularity by increasing the grain size of the reconfigurable region. Results show that the implementation in hardware is 68 times faster than that in software. This architecture with DPR also facilitates the low power consumption of 4 mW, leading to a reduction of 57% than that without DPR. This is also the lowest power consumption in current machine learning hardware implementations. Besides, the circuitry area is 41 times smaller. SVR with Gaussian kernel shows a success rate of 99.18% and minimum square error of 0.0146 for testing with the planning trajectory. This system is useful for adaptive applications where the user/designer can modify/reconfigure the hardware layout during its application, thus contributing to lower power consumption, smaller hardware area, and shorter execution time.

Keywords

Dynamic partial reconfiguration (DPR) / Field programmable gate array (FPGA) implementation / Image edge detection / Support vector regression (SVR) / Unmanned aerial vehicle (UAV) pose estimation

Cite this article

Download citation ▾

Gracieth Cavalcanti Batista, Johnny Öberg, Osamu Saotome, Haroldo F. de Campos Velho, Elcio Hideiti Shiguemori, Ingemar Söderquist. Machine learning algorithm partially reconfigured on FPGA for an image edge detection system. Journal of Electronic Science and Technology, 2024, 22(2): 100248 DOI:10.1016/j.jnlest.2024.100248

登录浏览全文

4963

注册一个新账户忘记密码

Funding

This work was financially supported by the National Council for Scientific and Technological Development (CNPq, Brazil), Swedish-Brazilian Research and Innovation Centre (CISB), and Saab AB under Grant No. CNPq: 200053/2022-1; the National Council for Scientific and Technological Development (CNPq, Brazil) under Grants No. CNPq: 312924/2017-8 and No. CNPq: 314660/2020-8.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Ingemar Söderquist is currently employed by Saab AB, Linköping, Sweden and the research project is funded by Saab AB. Other authors declare that there are no competing interests.

References

Publishing order | Descend order by publishing year | Descend order by cited within

[1]	J.F. Keane, S.S. Carr, A brief history of early unmanned aircraft, J. Hopkins APL Tech. D. 32 (3) (2013) 558-571.

[2]	R. Yanushevsky, Guidance of Unmanned Aerial Vehicles, CRC Press, Boca Raton, (2011).

[3]	M.S. Grewal, A.P. Andrews, C.G. Bartone, Global Navigation Satellite Systems, Inertial Navigation, and Integration, (fourth ed.), Wiley, Hoboken, (2020).

[4]

M. Jun, S.I. Roumeliotis, G.S. Sukhatme, State estimation of an autonomous helicopter using Kalman filtering, in: Proc. of IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems. Human and Environment Friendly Robots with High Intelligence and Emotional Quotients, Kyongju, South Korea, (1999), 1346-1353.

[5]	Y. Oshman, I. Shaviv, Optimal tuning of a Kalman filter using genetic algorithms, in: Proc. of AIAA Guidance, Navigation, and Control Conf. and Exhibit, Dever, USA, (2000), pp. 1-11.

[6]	J. Sasiadek, Q. Wang, R. Johnson, L. Sun, J. Zalewski, UAV navigation based on parallel extended Kalman filter, in: Proc. of AIAA Guidance, Navigation, and Control Conf. and Exhibit, Dever, USA, (2000), pp. 1-8.

[7]	G. Conte, P. Doherty, Vision-based unmanned aerial vehicle navigation using geo-referenced information, EURASIP J. Adv. Sig. Pr. 2009 (1) (2009) 387308:1-18.

[8]	A. Mohammadi, F. Sheikholeslam, M. Emami, Metaheuristic algorithms for integrated navigation systems, M. Ouaissa, I.U. Khan, M. Ouaissa, Z. Boulouard, S.B.H. Shah (Eds.), Computational Intelligence for Unmanned Aerial Vehicles Communication Networks, Springer, Cham, (2022), pp. 45-72.

[9]	B. Zolesi, L.R. Cander, Ionospheric Prediction and Forecasting, Springer, Berlin, (2014).

[10]	R.A. Steenburgh, C.G. Smithtro, K.M. Groves, Ionospheric scintillation effects on single frequency GPS, Space Weather 6 (4) (2008) S04D02:1-12.

[11]	R.W. Eastes, S.C. Solomon, R.E. Daniell, et al., Global-scale observations of the equatorial ionization anomaly, Geophys. Res. Lett. 46 (16) (2019) 9318-9326.

[12]	M.H. Mokhtar, N.A. Rahim, M.Y. Ismail, S.M. Buhari, Ionospheric perturbation: A review of equatorial plasma bubble in the ionosphere, in: Proc. of 6th Intl. Conf. on Space Science and Communication, Johor Bahru, Malaysia, (2019), pp. 23-28.

[13]	D. Silva, H. Takahashi, C. Wrasse, C. Figueireido, Characteristics of ionospheric bubbles observed by TEC maps in Brazilian sector, in: Proc. of 15th Intl. Congress of the Brazilian Geophysical Society, Rio de Janeiro, Brazil, (2017), pp. 23-28.

[14]	H. Takahashi, M.J. Taylor, J.H.A. Sobral, A.F. Medeiros, D. Gobbi, D.C. Santana, Fine structure of the ionospheric plasma bubbles observed by the OI 6300 and 5577 airglow images, Adv. Space Res. 27 (6–7) (2001) 1189-1194.

[15]	F.D. Chu, J.Y. Liu, H. Takahashi, J.H.A. Sobral, M.J. Taylor, A.F. Medeiros, The climatology of ionospheric plasma bubbles and irregularities over Brazil, Ann. Geophys.-Germany 23 (2) (2005) 379-384.

[16]	A. Couturier, M.A. Akhloufi, A review on absolute visual localization for UAV, Robot. Auton. Syst. 135 (2021) 103666:1-17.

[17]	A. Couturier, M.A. Akhloufi, Convolutional neural networks and particle filter for UAV localization, in: Proc. of SPIE 11758, Unmanned Systems Technology XXIII, Online, (2021), pp. 117580D:1-13.

[18]	V. Walter, M. Vrba, M. Saska, On training datasets for machine learning-based visual relative localization of micro-scale UAVs, in: Proc. of IEEE Intl. Conf. on Robotics and Automation, Paris, France, (2020), pp. 10674-10680.

[19]	V.A.M.F. Torres, B.R.A. Jaimes, E.S. Ribeiro, et al., Combined weightless neural network FPGA architecture for deforestation surveillance and visual navigation of UAVs, Eng. Appl. Artif. Intel. 87 (2020) 103227:1-9.

[20]	G. da Penha Neto, H.F. de Campos Velho, E.H. Shiguemori, J.R.G. Braga, Image processing for UAV autonomous navigation applying self-configuring neural network, C. Constanda, P. Harris (Eds.), Integral Methods in Science and Engineering, Springer, Cham, (2019), pp. 321-332.

[21]	J.R.G. Braga, H.F. de Campos Velho, E.H. Shiguemori, P. Doherty, Drone autonomous navigation by hardware image processing, A. Cardoan, L. Garelli, J.M Gimenez, P.A Kler, S.M. Damián, M.A. Storti (Eds.), Mecânica Computacional Vol XXXVII, Santa Fe, (2019), pp. 2033-2043.

[22]	C.A.O. Silva, G.A.M. Goltz, E.H. Shiguemori, et al., Image matching applied to autonomous navigation of unmanned aerial vehicles, Intl. Journal of High Performance Systems Architecture 6 (4) (2017) 205-212.

[23]	A. Shetty, G.X. Gao, UAV pose estimation using cross-view geolocalization with satellite imagery, in: Proc. of Intl. Conf. on Robotics and Automation, Montreal, Canada, (2019), pp. 1827-1833.

[24]	T. Hinzmann, R. Siegwart, Deep UAV localization with reference view rendering [Online]. Available, https://arxiv.org/abs/2008.04619, August 2020.

[25]	T.Y. Lin, S. Belongie, J. Hays, Cross-view image geolocalization, in: Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, Portland, USA, (2013), pp. 891-898.

[26]	T.Y. Lin, Y. Cui, S. Belongie, J. Hays, Learning deep representations for ground-to-aerial geolocalization, in: Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, Boston, USA, (2015), pp. 5007-5015.

[27]	J.R.G. Braga, Navegação Autônoma de VANT por Imagens LiDAR, Ph.D. dissertation, Instituto Nacional de Pesquisas Espaciais, São José dos Campos, Brazil, (2018).

[28]	W. da Silva, N.L. Vijaykumar, S.A. Sandri, et al., Image edge extraction by artificial intelligence schemes for UAV autonomous navigation, Proc. Series of the Brazilian Society of Computational and Applied Mathematics 7 (1) (2020) 1-7.

[29]	A. Al-Amaren, M.O. Ahmad, M.N.S. Swamy, RHN: a residual holistic neural network for edge detection, IEEE Access 9 (2021) 74646-74658.

[30]	D.-P. Yang, B. Peng, Z. Al-Huda, A. Malik, D.-H. Zhai, An overview of edge and object contour detection, Neurocomputing 488 (2022) 470-493.

[31]	K. Li, Y.-J. Tian, B. Wang, Z.-Q. Qi, Q. Wang, Bi-directional pyramid network for edge detection, Electronics 10 (3) (2021) 329:1-15.

[32]	J.-Y. Zhan, A.-T. Yu, W. Jiang, et al., FPGA-based acceleration for binary neural networks in edge computing, Journal of Electronic Science and Technology 21 (2) (2023) 100204:1-13.

[33]	M.D. Ciletti, Advanced Digital Design with the Verilog HDL, (second ed.), Prentice Hall Press, Upper Saddle River, (2010).

[34]	G. Valente, T.D. Mascio, L. Pomante, G. D’Andrea, Dynamic partial reconfiguration profitability for real-time systems, IEEE Embed. Syst. Lett. 13 (3) (2021) 102-105.

[35]	X.-D. Sun, Z.-Q. Wang, W. Jing, Image filtering and edge detection system based on FPGA, in: Proc. of 10th Intl. Conf. on Communications, Signal Processing, and Systems, Singapore, (2022), pp. 148-156.

[36]	Q. Qin, L.-Y. Sun, Y. Feng, Real-time image filtering and edge detection method based on FPGA, in: Proc. of IEEE 5th Intl. Conf. on Electronics Technology, Chengdu, China, (2022), pp. 1168-1173.

[37]	F. De Vivo, M. Battipede, E. Johnson, Infrared line camera data-driven edge detector in UAV forest fire monitoring, Aero. Sci. Technol. 111 (2021) 106574:1-8.

[38]	K. Kaur, N. Jindal, K. Singh, Fractional Fourier transform based Riesz fractional derivative approach for edge detection and its application in image enhancement, Signal Process. 180 (2021) 107852:1-19.

[39]	J.-M. Zhang, Z.-Y. Zi, T. Jiang, C. Zhang, Y.-H. Tai, Implementation and optimization of FPGA-based edge detection algorithm, Z.T. Yu, S. Patnaik, J. Wang, N. Dey (Eds.), Advancements in Mechatronics and Intelligent Robotics, Springer, Singapore, (2021), pp. 611-617.

[40]	D.-J. Liu, G.-L. Pu, X.-Y. Wu, Quaternion-based improved cuckoo algorithm for colour UAV image edge detection, IET Image Process. 16 (3) (2022) 926-935.

[41]	G. Conte, P. Doherty, An integrated UAV navigation system based on aerial image matching, in: Proc. of IEEE Aerospace Conf., Big Sky, USA, (2008), pp. 1-10.

[42]	J.R.G. Braga, H.F.C. Velho, G. Conte, P. Doherty, É.H. Shiguemori, An image matching system for autonomous UAV navigation based on neural network, in: Proc. of 14th Intl. Conf. on Control, Automation, Robotics and Vision, Phuket, Thailand, (2016), pp. 1-6.

[43]	E.F.P. da Luz, J.C. Becceneri, H.F. de Campos Velho, A new multi-particle collision algorithm for optimization in a high performance environment, Journal of Computational Interdisciplinary Sciences 1 (1) (2008) 3-10.

[44]	H. Goforth, S. Lucey, GPS-denied UAV localization using pre-existing satellite imagery, in: Proc. of Intl. Conf. on Robotics and Automation, Montreal, Canada, (2019), pp. 2974-2980.

[45]	J.F. Hughes, A. van Dam, M. McGuire, et al., Computer Graphics: Principles and Practice, (third ed.), Addison-Wesley, Reading, (2013).

[46]	N. Otsu, A threshold selection method from gray-level histograms, IEEE T. Systems, Man, and Cybernetics 9 (1) (1979) 62-66.

[47]	C. Cortes, V. Vapnik, Support-vector networks, Mach. Learn. 20 (3) (1995) 273-297.

[48]	B. Schölkopf, P. Simard, A. Smola, V. Vapnik, Prior knowledge in support vector kernels, in: Proc. of 10th Intl. Conf. on Neural Information Processing Systems, Cambridge, UK, (1997), pp. 640-646.

[49]	V.N. Vapnik, The Nature of Statistical Learning Theory, Springer, New York, (1995).

[50]	V.N. Vapnik, A.Y. Chervonenkis, On the uniform convergence of relative frequencies of events to their probabilities, V. Vovk, H. Papadopoulos, A. Gammerman (Eds.), Measures of Complexity, Springer, Cham, (2015), pp. 11-30.

[51]	H. Drucker, C.J.C. Burges, L. Kaufman, A. Smola, V. Vapnik, Support vector regression machines, in: Proc. of 9th Intl. Conf. on Neural Information Processing Systems, Denver, USA, (1996), pp. 155-161.

[52]	M. Awad, R. Khanna, Efficient Learning Machines: Theories, Concepts, and Applications for Engineers and System Designers, Apress, Berkeley, (2015).

[53]	G.C. Batista, D.L. Oliveira, O. Saotome, W.L.S. Silva, A low-power asynchronous hardware implementation of a novel SVM classifier, with an application in a speech recognition system, Microelectron. J. 105 (2020) 104907:1-17.

[54]	L.-G. Guo, S.-T. Wu, FPGA implementation of a real-time edge detection system based on an improved Canny algorithm, Appl. Sci. 13 (2) (2023) 870:1-17.

[55]	V. Conti, C. Militello, L. Rundo, S. Vitabile, A novel bio-inspired approach for high-performance management in service-oriented networks, IEEE T. Emerg. Top. Com. 9 (4) (2021) 1709-1722.