Machine learning algorithm partially reconfigured on FPGA for an image edge detection system

Abstract

Unmanned aerial vehicles (UAVs) have been widely used in military, medical, wireless communications, aerial surveillance, etc. One key topic involving UAVs is pose estimation in autonomous navigation. A standard procedure for this process is to combine inertial navigation system sensor information with the global navigation satellite system (GNSS) signal. However, some factors can interfere with the GNSS signal, such as ionospheric scintillation, jamming, or spoofing. One alternative method to avoid using the GNSS signal is to apply an image processing approach by matching UAV images with georeferenced images. But a high effort is required for image edge extraction. In this paper, a support vector regression (SVR) model is proposed to reduce this computational load and processing time. The dynamic partial reconfiguration (DPR) of part of the SVR datapath is implementated to accelerate the process, reduce the area, and analyze its granularity by increasing the grain size of the reconfigurable region. Results show that the implementation in hardware is 68 times faster than that in software. This architecure with DPR also facilitates the low power consumption of 4 ​mW, leading to a reduction of 57% than that without DPR. This is also the lowest power consumption in current machine learning hardware implementations. Besides, the circuitry area is 41 times smaller. SVR with Gaussian kernel shows a success rate of 99.18% and minimum square error of 0.0146 for testing with the planning trajectory. This system is useful for adaptive applications where the user/designer can modify/reconfigure the hardware layout during its application, thus contributing to lower power consumption, smaller hardware area, and shorter execution time.

Keywords

Dynamic partial reconfiguration (DPR) / Field programmable gate array (FPGA) implementation / Image edge detection / Support vector regression (SVR) / Unmanned aerial vehicle (UAV) pose estimation / Dynamic partial reconfiguration (DPR) / Field programmable gate array (FPGA) implementation / Image edge detection / Support vector regression (SVR) / Unmanned aerial vehicle (UAV) pose estimation

References

[1]	J.F. Keane, S.S. Carr. A brief history of early unmanned aircraft. J. Hopkins APL Tech. D., 32 (3) (2013), pp. 558-571.
[2]	R. Yanushevsky.Guidance of Unmanned Aerial Vehicles. CRC Press, Boca Raton (2011).
[3]	M.S. Grewal, A.P. Andrews, C.G. Bartone. Global Navigation Satellite Systems, Inertial Navigation,Integration. (fourth ed.), Wiley, Hoboken (2020).
[4]	M. Jun, S.I. Roumeliotis, G.S. Sukhatme. State estimation of an autonomous helicopter using Kalman filtering. Proc. of IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems. Human and Environment Friendly Robots with High Intelligence and Emotional Quotients (Kyongju, South Korea, 1999), pp. 1346-1353.
[5]	Y. Oshman, I. Shaviv.Optimal tuning of a Kalman filter using genetic algorithms. Proc. of AIAA Guidance, Navigation, and Control Conf. and Exhibit, Dever, USA(2000), pp. 1-11.
[6]	J. Sasiadek, Q. Wang, R. Johnson, L. Sun, J. Zalewski.UAV navigation based on parallel extended Kalman filter. Proc. of AIAA Guidance, Navigation, and Control Conf. and Exhibit, Dever, USA(2000), pp. 1-8.
[7]	G. Conte, P. Doherty. Vision-based unmanned aerial vehicle navigation using geo-referenced information. EURASIP J. Adv. Sig. Pr., 2009 (1)(2009), pp. 387308:1-18.
[8]	A. Mohammadi, F. Sheikholeslam, M. Emami. Metaheuristic algorithms for integrated navigation systems. M. Ouaissa, I.U. Khan, M. Ouaissa, Z. Boulouard, S.B.H. Shah (Eds.), Computational Intelligence for Unmanned Aerial Vehicles Communication Networks, Springer, Cham (2022), pp. 45-72.
[9]	B. Zolesi, L.R. Cander. Ionospheric Prediction and Forecasting. Springer, Berlin (2014).
[10]	R.A. Steenburgh, C.G. Smithtro, K.M. Groves. Ionospheric scintillation effects on single frequency GPS. Space Weather, 6 (4) (2008), pp. S04D02:1-12.
[11]	R.W. Eastes, S.C. Solomon, R.E. Daniell, et al. Global-scale observations of the equatorial ionization anomaly. Geophys. Res. Lett., 46 (16) (2019), pp. 9318-9326.
[12]	M.H. Mokhtar, N.A. Rahim, M.Y. Ismail, S.M. Buhari. Ionospheric perturbation: A review of equatorial plasma bubble in the ionosphere. Proc. of 6th Intl. Conf. on Space Science and Communication, Johor Bahru, Malaysia (2019), pp. 23-28.
[13]	D. Silva, H. Takahashi, C. Wrasse, C. Figueireido.Characteristics of ionospheric bubbles observed by TEC maps in Brazilian sector. Proc. of 15th Intl. Congress of the Brazilian Geophysical Society, Rio de Janeiro, Brazil(2017), pp. 23-28.
[14]	H. Takahashi, M.J. Taylor, J.H.A. Sobral, A.F. Medeiros, D. Gobbi, D.C. Santana. Fine structure of the ionospheric plasma bubbles observed by the OI6300 and 5577 airglow images. Adv. Space Res., 27 (6-7)(2001), pp. 1189-1194. View articleGoogle Scholar.
[15]	F.D. Chu, J.Y. Liu, H. Takahashi, J.H.A. Sobral, M.J. Taylor, A.F. Medeiros. The climatology of ionospheric plasma bubbles and irregularities over Brazil. Ann. Geophys.-Germany, 23 (2) (2005), pp. 379-384.
[16]	A. Couturier, M.A. Akhloufi. A review on absolute visual localization for UAV. Robot. Auton. Syst., 135 (2021), pp. 103666:1-17.
[17]	A. Couturier, M.A. Akhloufi. Convolutional neural networks and particle filter for UAV localization. Proc. of SPIE 11758, Unmanned Systems Technology XXIII, Online (2021), pp. 117580D:1-13.
[18]	V. Walter, M. Vrba, M. Saska. On training datasets for machine learning-based visual relative localization of micro-scale UAVs. Proc. of IEEE Intl. Conf. on Robotics and Automation, Paris, France (2020), pp. 10674-10680.
[19]	V.A.M.F. Torres, B.R.A. Jaimes, E.S. Ribeiro, et al. Combined weightless neural network FPGA architecture for deforestation surveillance and visual navigation of UAVs. Eng. Appl. Artif. Intel., 87 (2020), pp. 103227:1-9.
[20]	G. da Penha Neto, H.F. de Campos Velho, E.H. Shiguemori, J.R.G. Braga. Image processing for UAV autonomous navigation applying self-configuring neural network. C. Constanda, P. Harris (Eds.), Integral Methods in Science and Engineering, Springer, Cham (2019), pp. 321-332.
[21]	J.R.G. Braga, H.F. de Campos Velho, E.H. Shiguemori, P. Doherty. Drone autonomous navigation by hardware image processing. Mecanica Computacional, XXXVII (2019), pp. 2033-2043. Santa Fe.
[22]	C.A.O. Silva, G.A.M. Goltz, E.H. Shiguemori, et al. Image matching applied to autonomous navigation of unmanned aerial vehicles. Intl. Journal of High Performance Systems Architecture, 6 (4) (2017), pp. 205-212.
[23]	A. Shetty, G.X. Gao. UAV pose estimation using cross-view geolocalization with satellite imagery. Proc. of Intl. Conf. on Robotics and Automation, Montreal, Canada (2019), pp. 1827-1833.
[24]	T. Hinzmann, R. Siegwart, Deep UAV localization with reference view rendering [Online]. Available, https://arxiv.org/abs/2008.04619, August 2020.
[25]	T.Y. Lin, S. Belongie, J. Hays. Cross-view image geolocalization. Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, Portland, USA (2013), pp. 891-898.
[26]	T.Y. Lin, Y. Cui, S. Belongie, J. Hays. Learning deep representations for ground-to-aerial geolocalization. Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, Boston, USA (2015), pp. 5007-5015.
[27]	J.R.G. Braga. Navega??o Aut?noma de VANT por Imagens LiDAR. Ph.D. dissertation, Instituto Nacional de Pesquisas Espaciais, S?o José dos Campos, Brazil (2018).
[28]	W. da Silva, N.L. Vijaykumar, S.A. Sandri, et al. Image edge extraction by artificial intelligence schemes for UAV autonomous navigation. Proc. Series of the Brazilian Society of Computational and Applied Mathematics, 7 (1) (2020), pp. 1-7.
[29]	A. Al-Amaren, M.O. Ahmad, M.N.S. Swamy. RHN: a residual holistic neural network for edge detection. IEEE Access, 9 (2021), pp. 74646-74658.
[30]	D.-P. Yang, B. Peng, Z. Al-Huda, A. Malik, D.-H. Zhai. An overview of edge and object contour detection. Neurocomputing, 488 (2022), pp. 470-493. View articleGoogle Scholar.
[31]	K. Li, Y.-J. Tian, B. Wang, Z.-Q. Qi, Q. Wang. Bi-directional pyramid network for edge detection. Electronics, 10 (3) (2021), pp. 329:1-15.
[32]	J.-Y. Zhan, A.-T. Yu, W. Jiang, et al. FPGA-based acceleration for binary neural networks in edge computing. Journal of Electronic Science and Technology, 21 (2) (2023), pp. 100204:1-13.
[33]	M.D. Ciletti.Advanced Digital Design with the Verilog HDL. (second ed.), Prentice Hall Press, Upper Saddle River (2010).
[34]	G. Valente, T.D. Mascio, L. Pomante, G. D’Andrea. Dynamic partial reconfiguration profitability for real-time systems. IEEE Embed. Syst. Lett., 13 (3) (2021), pp. 102-105.
[35]	X.-D. Sun, Z.-Q. Wang, W. Jing. Image filtering and edge detection system based on FPGA. Proc. of 10th Intl. Conf. on Communications, Signal Processing,Systems, Singapore (2022), pp. 148-156.
[36]	Q. Qin, L.-Y. Sun, Y. Feng. Real-time image filtering and edge detection method based on FPGA. Proc. of IEEE 5th Intl. Conf. on Electronics Technology, Chengdu, China (2022), pp. 1168-1173.
[37]	F. De Vivo, M. Battipede, E. Johnson. Infrared line camera data-driven edge detector in UAV forest fire monitoring. Aero. Sci. Technol., 111 (2021), pp. 106574:1-8.
[38]	K. Kaur, N. Jindal, K. Singh. Fractional Fourier transform based Riesz fractional derivative approach for edge detection and its application in image enhancement. Signal Process., 180 (2021), pp. 107852:1-19.
[39]	J.-M. Zhang, Z.-Y. Zi, T. Jiang, C. Zhang, Y.-H. Tai. Implementation and optimization of FPGA-based edge detection algorithm. Z.T. Yu, S. Patnaik, J. Wang, N. Dey (Eds.), Advancements in Mechatronics and Intelligent Robotics, Springer, Singapore(2021), pp. 611-617.
[40]	D.-J. Liu, G.-L. Pu, X.-Y. Wu. Quaternion-based improved cuckoo algorithm for colour UAV image edge detection. IET Image Process., 16 (3) (2022), pp. 926-935.
[41]	G. Conte, P. Doherty. An integrated UAV navigation system based on aerial image matching. Proc. of IEEE Aerospace Conf., Big Sky, USA (2008), pp. 1-10.
[42]	J.R.G. Braga, H.F.C. Velho, G. Conte, P. Doherty, é.H. Shiguemori. An image matching system for autonomous UAV navigation based on neural network. Proc. of 14th Intl. Conf. on Control, Automation, Robotics and Vision, Phuket, Thailand (2016), pp. 1-6.
[43]	E.F.P. da Luz, J.C. Becceneri, H.F. de Campos Velho. A new multi-particle collision algorithm for optimization in a high performance environment. Journal of Computational Interdisciplinary Sciences, 1 (1) (2008), pp. 3-10.
[44]	H. Goforth, S. Lucey. GPS-denied UAV localization using pre-existing satellite imagery. Proc. of Intl. Conf. on Robotics and Automation, Montreal, Canada (2019), pp. 2974-2980.
[45]	J.F. Hughes, A. van Dam, M. McGuire, et al. Computer Graphics: Principles and Practice. (third ed.), Addison-Wesley, Reading (2013).
[46]	N. Otsu. A threshold selection method from gray-level histograms. IEEE T. Systems, Man,Cybernetics, 9 (1) (1979), pp. 62-66.
[47]	C. Cortes, V. Vapnik. Support-vector networks. Mach. Learn., 20 (3) (1995), pp. 273-297.
[48]	B. Sch?lkopf, P. Simard, A. Smola, V. Vapnik. Prior knowledge in support vector kernels. Proc. of 10th Intl. Conf. on Neural Information Processing Systems, Cambridge, UK(1997), pp. 640-646.
[49]	V.N. Vapnik. The Nature of Statistical Learning Theory. Springer, New York (1995).
[50]	V.N. Vapnik, A.Y. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. V. Vovk, H. Papadopoulos, A. Gammerman (Eds.), Measures of Complexity, Springer, Cham (2015), pp. 11-30.
[51]	H. Drucker, C.J.C.Burges, L. Kaufman, A. Smola, V. Vapnik. Support vector regression machines. Proc. of 9th Intl. Conf. on Neural Information Processing Systems, Denver, USA(1996), pp. 155-161.
[52]	M. Awad, R. Khanna. Efficient Learning Machines: Theories, Concepts,Applications for Engineers and System Designers. Apress, Berkeley (2015).
[53]	G.C. Batista, D.L. Oliveira, O. Saotome, W.L.S. Silva. A low-power asynchronous hardware implementation of a novel SVM classifier, with an application in a speech recognition system. Microelectron. J., 105 (2020), pp. 104907:1-17.
[54]	L.-G. Guo, S.-T. Wu. FPGA implementation of a real-time edge detection system based on an improved Canny algorithm. Appl. Sci., 13 (2) (2023), pp. 870:1-17.
[55]	V. Conti, C. Militello, L. Rundo, S. Vitabile. A novel bio-inspired approach for high-performance management in service-oriented networks. IEEE T. Emerg. Top. Com., 9 (4) (2021), pp. 1709-1722.