Autonomous vehicles for micro-mobility

Abstract

Autonomous vehicles have been envisioned for more than 100 years. One of the first suggestions was a front cover of Scientific America back in 1916. Today, it is possible to get cars that drive autonomously for extended distances. We are also starting to see micro-mobility solutions, such as the Nuro vehicles for pizza delivery. Building autonomous cars that can operate in urban environments with a diverse set of road-users is far from trivial. Early 2018 the Contextual Robotics Institute at UC San Diego launched an effort to build a full stack autonomous vehicle for micro-mobility. The motivations were diverse: i) development of a system for operation in an environment with many pedestrians, ii) design of a system that does not rely on dense maps (or HD-maps as they are sometimes named), iii) design strategies to build truly robust systems, and iv) a framework to educate next-generation engineers. In this paper, we present the research effort of design, prototyping, and evaluation of such a vehicle. From the evaluation, several research directions are explored to account for shortcomings. Lessons and issues for future work are additionally drawn from this work.

Keywords

Autonomous vehicles / Intelligent systems / Micro-mobility

References

[1] E. D. Dickmanns, A. Zapp, in Mobile Robots, SPIE, 727, ed. by Wolfe W.J., Marquina N.A Curvature-based Scheme for Improving Road Vehicle Guidance by Computer Vision (Bellingham, 1987), pp. 161–168.

[2] DickmannsE.. Dynamic Vision for Perception and Control of Motion, 2007 Heidelberg Springer Verlag

[3] TurkM. A., MorgenthalerD. G., GrembanK. D., MarraM.. VITS - A Vision System for Autonomous Land Vehicle Navigation. IEEE Trans. Pattern Anal. Mach. Intell., 1988, 10(3):342-361

[4] B. Marr, Key milestones of Waymo - Google’s self-driving cars (2018). https://forbes.com/sites/bernardmarr/2018/09/21/key-milestones-of-waymo-googles-self-driving-cars. Accessed 02 Oct 2021.

[5] I. Bonifacic, Toyota is developing autonomous taxis with help from Aurora (2021). Engadget. https://www.engadget.com/toyota-aurora-denso-autonomous-vehicle-partnership-191500404.html. Accessed 02 Oct 2021.

[6] A. Palmer, Amazon Zoox unveils self-driving robotaxi (2020). CNBC, https://cnbc.com/2020/12/14/amazons-self-driving-company-zoox-unveils-autonomous-robotaxi.html. Accessed 02 Oct 2021.

[7] A. Adler, Aurora closes in on production version of self-driving truck technology (2021). FreightWaves, https://www.freightwaves.com/?p=358829. Accessed 02 Oct 2021.

[8] Baidu-Apollo-team, Apollo: Open source autonomous driving (2017). https://github.com/ApolloAuto/apollo. Accessed 02 Oct 2021.

[9] KatoS., TakeuchiE., IshiguroY., NinomiyaY., TakedaK., HamadaT.. An open approach to autonomous vehicles. IEEE Micro, 2015, 35(6):60-68 https://doi.org/10.1109/MM.2015.133

[10] D. Paz, P. -J. Lai, S. Harish, H. Zhang, N. Chan, C. Hu, S. Binnani, H. Christensen, in Field and Service Robotics. Lessons learned from deploying autonomous vehicles at UC San Diego (Tokyo, JP, 2019).

[11] ZhangZ.. A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell., 2000, 22(11):1330-1334

[12] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, A. Ng, in ICRA Workshop on Open Source Software, 3. ROS: an open-source Robot Operating System, (2009).

[13] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, O. Beijbom, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). PointPillars: Fast encoders for object detection from point clouds, (2019), pp. 12689–12697. https://doi.org/10.1109/CVPR.2019.01298.

[14] FischlerM., BollesR.. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 1981, 24: 381-395

[15] LepetitV., Moreno-NoguerF., FuaP.. EPnP: An accurate O(n) solution to the PnP problem. Int. J. Comput. Vis., 2009, 81: 155 https://doi.org/10.1007/s11263-008-0152-6

[16] J. Redmon, A. Farhadi, YOLOv3: An Incremental Improvement. ArXiv abs/1804.02767 (2018).

[17] C. Campos, R. Elvira, J. Rodríguez, J. Montiel, J. D. Tardós, ORB-SLAM3: An accurate open-source library for visual, visual-inertial and multi-map SLAM. ArXiv abs/2007.11898 (2020).

[18] J. Zhang, S. Singh, in Robotics: Science and Systems. LOAM: Lidar odometry and mapping in real-time, (2014).

[19] M. Magnusson, The three-dimensional normal-distributions transform — an efficient representation for registration, surface analysis, and loop detection. PhD dissertation, Örebro universitet (2009).

[20] R. B. Rusu, S. Cousins, in IEEE International Conference on Robotics and Automation (ICRA). 3D is here: Point Cloud Library (PCL) (Shanghai, China, 2011).

[21] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, O. Beijbom, in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). nuScenes: A multimodal dataset for autonomous driving, (2020). https://doi.org/10.1109/CVPR42600.2020.01164.

[22] C. Ming-Fang, L. John, S. Patsorn, S. Jagjeet, B. Slawomir, H. Andrew, D. Wang, C. Peter, L. Simon, R. Deva, H. James, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Argoverse: 3D tracking and forecasting with rich maps, (2019), pp. 8740–8749.

[23] DarweeshH., TakeuchiE., TakedaK., NinomiyaY., SujiwoA., MoralesY., AkaiN., TomizawaT., KatoS.. Open source integrated planner for autonomous navigation in highly dynamic environments. J. Robot. Mechatron., 2017, 29: 668-684 https://doi.org/10.20965/jrm.2017.p0668

[24] R. C. Coulter, Implementation of the pure pursuit path tracking algorithm. Tech. rep. Carnegie-Mellon UNIV Pittsburgh PA Robotics INST (1992).

[25] PazD., LaiP. -J., ChanN., JianfY., ChristensenH. I.. Autonomous vehicle benchmarking using unbiased metrics. International Conference on Intelligent Robots and Systems (IROS), 2020 Las Vegas, NV IEEE/RSJ

[26] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, A. Alahi, 2018, in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks, pp. 2255–2264. https://doi.org/10.1109/CVPR.2018.00240.

[27] FernandoT., DenmanS., SridharanS., FookesC.. JawaharC. V., LiH., MoriG., SchindlerK.. GD-GAN: Generative Adversarial Networks for Trajectory Prediction and Group Detection in Crowds. Computer Vision – ACCV 2018, 2019 Cham Springer International Publishing 314-330

[28] J. Amirian, J. Hayet, Pettré J, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Social Ways: Learning multi-modal distributions of pedestrian trajectories with GANs, (2019), pp. 2964–2972.

[29] A. Sadeghian, V. Kosaraju, A. Sadeghian, N. Hirose, H. Rezatofighi, S. Savarese, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). SoPhie: An attentive GAN for predicting paths compliant to social and physical constraints, (2019), pp. 1349–1358. https://doi.org/10.1109/CVPR.2019.00144.

[30] V. Kosaraju, A. Sadeghian, R. Martín-Martín, I. Reid, H. Rezatofighi, S. Savarese, in Advances in Neural Information Processing Systems, Curran Associates, Inc., 32, ed. by H. Wallach, H. Larochelle, A. Beygelzimer, F. d’ Alché-Buc, E. Fox, and R. Garnett. Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks, (2019). https://proceedings.neurips.cc/paper/2019/file/d09bf41544a3365a46c9077ebb5e35c3-Paper.pdf.

[31] T. Salzmann, B. Ivanovic, P. Chakravarty, M. Pavone, Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data (2020). https://arxiv.org/pdf/2001.03093.pdf.

[32] X. Feng, Z. Cen, J. Hu, Y. Zhang, in 2019 IEEE Intelligent Transportation Systems Conference (ITSC). Vehicle trajectory prediction using intention-based conditional variational autoencoder, (2019), pp. 3514–3519. https://doi.org/10.1109/ITSC.2019.8917482.

[33] A. Mohamed, K. Qian, M. Elhoseiny, C. Claudel, in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Social-STGCNN: A social spatio-temporal graph convolutional neural network for human trajectory prediction, (2020), pp. 14412–14420. https://doi.org/10.1109/CVPR42600.2020.01443.

[34] J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, C. Schmid, in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). VectorNet: Encoding hd maps and agent dynamics from vectorized representation, (2020), pp. 11522–11530. https://doi.org/10.1109/CVPR42600.2020.01154.

[35] A. Ribeiro, L. Dihl, C. Jung, in International Conference on Systems, Signals and Image Processing. Automatic camera calibration for driver assistance systems, (2006), pp. 173–176.

[36] LuL., LuX., JiS., TongC.. A traffic camera calibration method based on multi-rectangle. Intelligent Information Processing VII, 2014 Berlin, Heidelberg Springer 230-238

[37] WangH., CaiY., LinG., ZhangW.. A novel method for camera external parameters online calibration using dotted road line. Adv. Robot., 2014, 28: 1033-1042 https://doi.org/10.1080/01X00000.6918642014.902329

[38] GeigerA., LenzP., StillerC., UrtasunR.. Vision meets robotics: The KITTI dataset. Int. J. Robot. Res. (IJRR), 2013, 32(11):1231-1237

[39] Y. Wu, A. Kirillov, F. Massa, W. -Y. Lo, R. Girshick, Detectron2 (2019). https://github.com/facebookresearch/detectron2. Accessed 02 Oct 2019.

[40] MadsenC. B., ChristensenH. I.. Chapter 1. Modelling and testing the stability of edge segments: Length and orientation, 1995 Singapore World Scientific Press

[41] K. He, G. Gkioxari, P. Dollár, R. Girshick, in 2017 IEEE International Conference on Computer Vision (ICCV). Mask R-CNN, (2017), pp. 2980–2988. https://doi.org/10.1109/ICCV.2017.322.

[42] Grompone von GioiR., RandallG.. A Sub-Pixel Edge Detector: an Implementation of the Canny/Devernay Algorithm. Image Process. On Line, 2017, 7: 347-372 https://doi.org/10.5201/ipol.2017.216

[43] Y. Han, Y. Liu, D. Paz, H. Christensen, Auto-calibration method using stop signs for urban autonomous driving applications. ArXiv abs/2010.07441 (2021). https://doi.org/2010.07441.

[44] ParkC., MoghadamP., KimS., SridharanS., FookesC.. Spatiotemporal Camera-LiDAR calibration: A targetless and structureless approach. IEEE Robot. Autom. Lett., 2020, 5: 1556-1563

[45] J. Kümmerle, T. Kühner, in 2020 IEEE International Conference on Robotics and Automation (ICRA). Unified intrinsic and extrinsic camera and LiDAR calibration under uncertainties, (2020), pp. 6028–6034. https://doi.org/10.1109/ICRA40945.2020.9197496.

[46] A. Dhall, K. Chelani, V. Radhakrishnan, K. M. Krishna, LiDAR-Camera Calibration using 3D-3D Point correspondences. ArXiv e-prints 1705.09785 (2017).

[47] S. Mishra, G. Pandey, S. Saripalli, in 2020 IEEE Intelligent Vehicles Symposium (IV). Extrinsic calibration of a 3D-LIDAR and a camera, (2020), pp. 1765–1770.

[48] TorrP., ZissermanA.. MLESAC: A new robust estimator with application to estimating image geometry. Comput. Vis. Image Underst., 2000, 78: 138-156

[49] R. B. Rusu, Semantic 3D object maps for everyday manipulation in human living environments. PhD thesis, Computer Science department, Technische Universitaet Muenchen, Germany (2009).

[50] T. Yin, X. Zhou, P. Krähenbühl, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Center-based 3D object detection and tracking, (2021), pp. 11784–11793.

[51] B. Zhu, Z. Jiang, X. Zhou, Z. Li, G. Yu, Class-balanced grouping and sampling for point cloud 3D object detection. ArXiv abs/1908.09492 (2019).

[52] BrummelenJ. V., O’BrienM., GruyerD. Najjaran H. Autonomous vehicle perception: The technology of today and tomorrow. Transp. Res. C Emerg. Technol., 2018, 89: 384-406 https://doi.org/10.1016/j.trc.2018.02.012

[53] ElfringJ., AppeldoornR., DriesS., KwakkernaatM.. Effective world modeling: Multisensor data fusion methodology for automated driving. Sensors (Basel, Switzerland), 2016, 16: 1668

[54] RangeshA., TrivediM. M.. No blind spots: Full-surround multi-object tracking for autonomous vehicles using cameras and lidars. IEEE Trans. Intell. Veh., 2019, 4(4):588-599 https://doi.org/10.1109/TIV.2019.2938110

[55] Z. Ding, Y. Hu, R. Ge, L. Huang, S. Chen, Y. Wang, J. Liao, 1st place solution for Waymo open dataset challenge - 3D detection and domain adaptation. ArXiv abs/2006.15505 (2020).

[56] H. Cho, Y. Seo, B. V. K. V. Kumar, R. R. Rajkumar, in 2014 IEEE International Conference on Robotics and Automation (ICRA). A multi-sensor fusion system for moving object detection and tracking in urban driving environments, (2014), pp. 1836–1843. https://doi.org/10.1109/ICRA.2014.6907100.

[57] C. R. Qi, W. Liu, C. Wu, H. Su, L. J. Guibas, in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Frustum PointNets for 3D object detection from RGB-D data, (2018), pp. 918–927. https://doi.org/10.1109/CVPR.2018.00102.

[58] D. Xu, D. Anguelov, A. Jain, in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Pointfusion: Deep sensor fusion for 3D bounding box estimation, (2018), pp. 244–253.

[59] X. Weng, J. Wang, D. Held, K. Kitani, 3D Multi-Object Tracking: A Baseline and New Evaluation Metrics, (2020).

[60] H. -k Chiu, A. Prioletti, J. Li, J. Bohg, Probabilistic 3d multi-object tracking for autonomous driving. ArXiv abs/2001.05673 (2020).

[61] D. Paz, P. J. Lai, N. Chan, Y. Jianf, H. I. Christensen, Probabilistic semantic mapping for urban autonomous driving applications (IEEE, 2020).

[62] N. Homayounfar, W. C. Ma, J. Liang, X. Wu, J. Fan, R. Urtasun, in Proceedings of the IEEE/CVF International Conference on Computer Vision. Dagmapper: Learning to map by discovering lane topology, (2019), pp. 2911–2920.

[63] N. Homayounfar, W. -C. Ma, S. K. Lakshmikanth, R. Urtasun, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Hierarchical recurrent attention networks for structured online maps, (2018), pp. 3417–3426.

[64] S. Sengupta, P. Sturgess, L. Ladickỳ, P. H. Torr, in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. Automatic dense visual semantic mapping from street-level imagery (IEEE, 2012), pp. 857–862.

[65] S. Sengupta, E. Greveson, A. Shahrokni, P. H. Torr, in 2013 IEEE International Conference on robotics and Automation. Urban 3D semantic modelling using stereo vision (IEEE, 2013), pp. 580–585.

[66] T. Westfechtel, K. Ohno, R. P. B. Neto, S. Kojima, S. Tadokoro, in 2019 IEEE Intelligent Transportation Systems Conference (ITSC). Fusion of camera and lidar data for large scale semantic mapping (IEEE, 2019), pp. 257–264.

[67] L. -C. Chen, Y. Zhu, G. Papandreou, F. Schroff, H. Adam, in Computer Vision - ECCV 2018. Lecture Notes in Computer Science, 11211, ed. by V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss. Encoder-decoder with atrous separable convolution for semantic image segmentation (Springer, 2018).

[68] S. Hecker, D. Dai, L. Van Gool, in Proceedings of the european conference on computer vision (ECCV). End-to-end learning of driving models with surround-view cameras and route planners, (2018), pp. 435–453.

[69] A. Amini, G. Rosman, S. Karaman, D. Rus, in 2019 International Conference on Robotics and Automation (ICRA). Variational end-to-end navigation and localization (IEEE, 2019), pp. 8958–8964.

[70] D. Paz, H. Zhang, H. I. Christensen, TridentNet: A conditional generative model for dynamic trajectory generation, (Singapore, 2021).

[71] HaklayM., WeberP.. Openstreetmap: User-generated street maps. IEEE Pervasive Comput., 2008, 7(4):12-18

[72] SohnK., YanX., LeeH.. Learning structured output representation using deep conditional generative models. Proceedings of the 28th International Conference on Neural Information Processing Systems, 2015 Cambridge MIT Press 3483-3491

[73] T. Yang, Z. Nan, H. Zhang, S. Chen, N. Zheng, in 2020 IEEE Intelligent Vehicles Symposium (IV). Traffic agent trajectory prediction using social convolution and attention mechanism (IEEE, 2020), pp. 278–283.

[74] AnguelovD., DulongC., FilipD., FruehC., LafonS., LyonR., OgaleA., VincentL., WeaverJ.. Google street view: Capturing the world at street level. Computer, 2010, 43(6):32-38

[75] ZhangP., ZhangM., LiuJ.. Real-time HD map change detection for crowdsourcing update based on mid-to-high-end sensors. Sensors, 2021, 21(7):2477

[76] ZhangY., YuanX., FangY., ChenS.. UAV low altitude photogrammetry for power line inspection. ISPRS Int. J. GEO-Inf., 2017, 6(1):14

[77] RemziE., AlkanE., AydinA.. A comparative analysis of UAV-RTK and UAV-PPK methods in mapping different surface types. Eur. J. For. Eng., 2020, 7(1):12-25

[78] BarazzettiL., RoncoroniF., BrumanaR., PrevitaliM.. Georeferencing accuracy analysis of a single worldview-3 image collected over milan. XXIII ISPRS Congress, 2016, 38: 429-434

[79] P. Daruthep, N. Sutthisangiam, in 2020 22nd International Conference on Advanced Communication Technology (ICACT). Development of automated processing for high-definition mapping system (IEEE, 2020), pp. 507–510.

[80] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, V. Koltun, in Proceedings of the 1st Annual Conference on Robot Learning. CARLA: An open urban driving simulator, (2017), pp. 1–16.

[81] G. Rong, B. Shin, H. Tabatabaee, Q. Lu, S. Lemke, M. Mozeiko, E. Boise, G. Uhm, M. Gerow, S. Mehta, E. Agafonov, T. H. Kim, E. Sterner, K. Ushiroda, M. Reyes, D. Zelenkovsky, S. Kim, in 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC). LGSVL Simulator: A high fidelity simulator for autonomous driving, (2020), pp. 1–6.

Funding

qualcomm