Performance comparison of deep reinforcement robot-arm learning in sequential fabrication of rule-based building design form

Abstract

Deep reinforcement learning (DRL) remains underexplored within architectural robotics, particularly in relation to self-learning of architectural design principles and design-aware robotic fabrication. To address this gap, we applied established DRL methods to enable robot arms to autonomously learn design rules in a pilot block wall assembly-design scenario. Recognizing the complexity inherent in such learning tasks, the problem was strategically decomposed into two sub-tasks: (i) target reaching (T1), modeled within a continuous action space, and (ii) sequential planning (T2), formulated within a discrete action space. For T1, we evaluated major DRL algorithms—Proximal Policy Optimization (PPO), Advantage Actor-Critic (A2C), Deep Deterministic Policy Gradient, Twin Delayed Deep Deterministic Policy Gradient, and Soft Actor-Critic (SAC), and PPO, A2C, and Double Deep Q-Network (DDQN) were tested for T2. Performance was assessed based on training efficacy, reliability, and two novel metrics: degree index and variation index. Our results revealed that SAC was the best for T1, whereas DDQN excelled in T2. Notably, DDQN exhibited strong learning adaptability, yielding diverse final layouts in response to varying initial conditions.

Keywords

Robotic architecture / Robot learning / Reinforcement learning / Robotic construction / Robot arm

References

[1] Aditra, R.F. , Tedjawinata, G.K. , Widyowijatnoko, A. , 2025. Jute tower: experience in natural fibre winding fabrication with collaborative robot. In: Reinhardt, Dagmar, Rogeau, Nicolas, Herr, Christiane M., Globa, Anastasia, Chen, Jielin, Narahara, Taro (Eds.), ARCHITECTURAL INFORMATICS-Proceedings of the 30th CAADRIA Conference, vol. 2, pp. 255-264. Tokyo, 22-29 March 2025.

[2] Amtsberg, F. , Yang, X. , Skoury, L. , Calepso, A.S. , Wortmann, T. , Sedlmair, M. , Menges, A. , 2023. Multi-actor fabrication for digital timber construction. In: Dokonal, W., Hirschberg, U., Wurzer, G. (Eds.), Digital Design Reconsidered-Proceedings of the 41st Conference on Education and Research in Computer Aided Architectural Design in Europe (eCAADe 2023), vol. 1, pp. 417-426. Graz, 20-22 September 2023.

[3] Apolinarska, A.A. , Pacher, M. , Li, H. , Cote, N. , Pastrana, R. , Gramazio, F. , Kohler, M. , 2021. Robotic assembly of timber joints using reinforcement learning. Autom. Constr. 125, 103569.

[4] Belousov, B. , Wibranek, B. , Schneider, J. , Schneider, T. , Chalvatzaki, G. , Peters, J. , Tessmann, O. , 2022. Robotic architectural assembly with tactile skills: simulation and optimization. Autom. Constr. 133, 104006.

[5] Breyer, M. , Furrer, F. , Novkovic, T. , Siegwart, R. , Nieto, J. , 2019. Comparing task simplifications to learn closed-loop object picking using deep reinforcement learning. IEEE Rob. Autom. Lett. 4, 1549- 1556.

[6] Bruun, E.P.G. , Oval, R. , Al Asali, W. , Gáspár, O. , Paris, V. , Adriaenssens, S. , 2024. Automating historical centeringminimizing masonry vaulting strategies: applications to cooperative robotic construction. Dev. Built Environ. 20, 100516.

[7] Chan, S.C.Y. , Fishman, S. , Canny, J. , Korattikara, A. , Guadarrama, S. , 2020. Measuring the reliability of reinforcement learning algorithms. arXiv: 1912.05663.

[8] de La Bourdonnaye, F.L. , Teulière, C. , Triesch, J. , Chateau, T. , 2018. Stage-wise learning of reaching using little prior knowledge. Front. Robot. AI 5, 5.

[9] Dong, R. , Du, J. , Liu, Y. , Heidari, A.A. , Chen, H. , 2023. An enhanced deep deterministic policy gradient algorithm for intelligent control of robotic arms. Front. Neuroinf. 17.

[10] Duan, K. , Suen, C.W.K. , Zou, Z. , 2023. MARC: a multi-agent robots control framework for enhancing reinforcement learning in construction tasks. arXiv: 2305.14586.

[11] Duan, K. , Zou, Z. , 2025. Enhancing construction robot collaboration via multiagent reinforcement learning. J. Intell. Constr. 3, 1- 16.

[12] Duan, K. , Zou, Z. , 2023. Learning from demonstrations: an intuitive VR environment for imitation learning of construction robots.arXiv: 2305.14584.

[13] Esfangareh, A.M. , 2022. Adopting Ai Techniques in Robotic Fabrication in Architecture: Intelligent Robotic Bricklaying Using Reinforcement Learning Algorithms. Middle East Technical University. Master Thesis.

[14] Franceschetti, A. , Tosello, E. , Castaman, N. , Ghidoni, S. , 2022. Robotic arm control and task training through deep reinforcement learning. In: Ang, M.H., Asama, H., Lin, W., Foong, S.(Eds.), Intelligent Autonomous Systems 16, Ias-16. Springer International Publishing Ag, Cham, pp. 532-550.

[15] Fuente, N.D.L. , Guerra, D.A.V. , 2024. A comparative study of deep reinforcement learning models: DQN vs PPO vs A2C. arXiv: 2407.14151.

[16] Ghiyasi, T. , Zargar, S.H. , Baghi, A. , 2023. Layer-by-Layer pick and place collaboration between human and robot using optimization. In: Dokonal, W., Hirschberg, U., Wurzer, G. (Eds.), Digital Design Reconsidered-Proceedings of the 41st Conference on Education and Research in Computer Aided Architectural Design in Europe (Ecaade 2023), vol. 1, pp. 769-778. Graz, 20-22 September 2023.

[17] Groenewolt, A. , Krieg, O. , Menges, A. , 2023. Collaborative humanrobot timber construction. In: Dokonal, W., Hirschberg, U., Wurzer, G. (Eds.), Digital Design Reconsidered-Proceedings of the 41st Conference on Education and Research in Computer Aided Architectural Design in Europe (eCAADe 2023), vol. 1, pp. 407-416. Graz, 20-22 September 2023.

[18] He, M. , Cheng, H. , Duan, F. , Sun, Z. , Li, S.N. , Yokoi, H. , Zhu, C. , 2022. Deep reinforcement learning for the improvement of robot manipulation skills under sparse reward. In: Presented at the Proceedings of the 34th Chinese Control and Decision Conference. CCDC, pp. 5508-5513, 2022.

[19] Huang, L. , Zhu, Z. , Zou, Z. , 2023. To imitate or not to imitate: boosting reinforcement learning-based construction robotic control for long-horizon tasks using virtual demonstrations. Autom. Constr. 146, 104691.

[20] Huang, L. , Cai, W. , Zhu, Z. , Zou, Z. , 2023a. Dexterous manipulation of construction tools using anthropomorphic robotic hand. Autom. Constr. 156, 105133.

[21] Huang, L.-S. , Spaw, G.T. , 2023a. InterLoop. In: Crawford, A., Diniz, N., Beckett, R., Vanucchi, J., Swackhamer, M. (Eds.), ACADIA 2023: Habits of the Anthropocene: Scarcity and Abundance in a Post-Material Economy, vol. 1, pp. 180-187. Denver, 26-28 October 2023.

[22] Huang, L.-S. , Spaw, G.T. , 2023b. Augmented reality assisted robotic: tube bending. In: Crawford, A., Diniz, N., Beckett, R., Vanucchi, J., Swackhamer, M. (Eds.), ACADIA 2023: Habits of the Anthropocene: Scarcity and Abundance in a Post-Material Economy, vol. 2, pp. 340-349. Denver, 26-28 October 2023.

[23] Ibrahim, S. , Mostafa, M. , Jnadi, A. , Salloum, H. , Osinenko, P. , 2024. Comprehensive overview of reward engineering and shaping in advancing reinforcement learning applications. IEEE Access 12, 175473- 175500.

[24] Iriondo, A. , Lazkano, E. , Ansuategi, A. , Rivera, A. , Lluvia, I. , Tubío, C. , 2023. Learning positioning policies for Mobile manipulation operations with deep reinforcement learning. Int. J. Mach. Learn. Cybern. 14, 3003- 3023.

[25] Iriondo, A. , Lazkano, E. , Susperregi, L. , Urain, J. , Fernandez, A. , Molina, J. , 2019. Pick and place operations in logistics using a mobile manipulator controlled with deep reinforcement learning. Appl. Sci. Switz. 9 (2), 348.

[26] Jiang, X.G. , Gao, T. , Zhang, C. , Yuan, P.F. , 2024. Mortise and tenon beam-to-beam Joints Solver for discrete timber structures: a structural performance-driven tool based on finite element analysis. In: Kontovourkis, O., Phocas, M.C., Wurzer, G. (Eds.), Data-Driven Intelligence-Proceedings of the 42nd Conference on Education and Research in Computer Aided Architectural Design in Europe (eCAADe 2024), vol. 1, pp. 157-166. Nicosia, 11-13 September 2024.

[27] Karimi, S. , Asadi, S. , Payberah, A.H. , 2024. BaziGooshi: a hybrid model of reinforcement learning for generalization in gameplay. IEEE Trans. Games 16, 722- 734.

[28] Karsan, Z. , Dillenburger, B. , De Wolf, C. , 2025. Multi-robotic system for welded steel disassembly. In: Reinhardt, Dagmar, Rogeau, Nicolas, Herr, Christiane M., Globa, Anastasia, Chen, Jielin, Narahara, Taro (Eds.), ARCHITECTURAL INFORMATICS-Proceedings of the 30th CAADRIA Conference, vol. 3, pp. 467-476. Tokyo, 22-29 March 2025.

[29] Kim, K. , 2022. Enhancing reinforcement learning performance in delayed reward system using DQN and heuristics. IEEE Access 10, 50641- 50650.

[30] Leder, S. , Kannenberg, F. , Siriwardena, L. , Schwinn, T. , Menges, A. , 2024. Approaching the augmentation of heuristic behaviors with reinforcement learning in collective robotic construction. In: Herrera, Pablo C., Gómez, Paula, Estevez, Alberto T., Torreblanca-Díaz, David A. (Eds.), Biodigital Intelligent Systems-Proceedings of the XXVIII Conference of the Iberoamerican Society of Digital Graphics (SIGraDi 2024), pp. 1283-1294 iBAG-UIC Barcelona, Spain, 13-15 November 2024.

[31] Lee, D. , Lee, S. , Masoud, N. , Krishnan, M.S. , Li, V.C. , 2022. Digital twin-driven deep reinforcement learning for adaptive task allocation in robotic construction. Adv. Eng. Inform. 53, 101710.

[32] Li, F. , Jiang, Q. , Quan, W. , Cai, S. , Song, R. , Li, Y. , 2020. Manipulation skill acquisition for robotic assembly based on multimodal information description. IEEE Access 8, 6282- 6294.

[33] Liu, Y. , Shamsi, S.M. , Fang, L. , Chen, C. , Napp, N. , 2018. Deep QLearning for dry stacking irregular objects. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS). Presented at the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1569-1576.

[34] Menezes, A. , Vicente, P. , Bernardino, A. , Ventura, R. , 2021. From rocks to walls: a model-free reinforcement learning approach to dry stacking with irregular rocks. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). Presented at the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2057-2065.

[35] Minooee Sabery, S. , Soana, V. , Bhaskar, C. , Bayraktaoglu, Y. , 2023. Elastic choreographies: a robotic bending active structure interacting with humans. In: Dokonal, W., Hirschberg, U., Wurzer, G. (Eds.), Digital Design Reconsidered-Proceedings of the 41st Conference on Education and Research in Computer Aided Architectural Design in Europe (eCAADe 2023), vol. 1, pp. 253-262. Graz, 20-22 September 2023.

[36] Mostafavi, S. , Kemper, B.N. , Kretzer, M. , Etemadi, A. , Mahmoud, H.A. , Yaseen, A. , Nabizadeh, M. , Balazadeh, T. , Chatterjee, S. , 2023. Cobotic matters-collaborative robots and discrete assembly design: from stacking to self-interlocking of reciprocal components. In: Koh, Immanuel, Reinhardt, Dagmar, Makki, Mohammed, Khakhar, Mona, Bao, Nic(Eds.), HUMAN-CENTRIC-Proceedings of the 28th CAADRIA Conference, pp. 241-250. Ahmedabad, 18-24 March 2023.

[37] Pittiglio, A. , Yang, X. , Kladeftira, M. , Amstberg, F. , Menges, A. , Parascho, S. , 2025. Augmented human-robot collaborative bending: human-Robot collaboration for biogenic material bending-active assemblies. In: Reinhardt, Dagmar, Rogeau, Nicolas, Herr, Christiane M., Globa, Anastasia, Chen, Jielin, Narahara, Taro (Eds.), ARCHITECTURAL INFORMATICS-Proceedings of the 30th CAADRIA Conference, vol. 2, pp. 315-324. Tokyo, 22-29 March 2025.

[38] Ruan, D. , Adel, A. , 2023. Robotic fabrication of nail laminated timber: a case study exhibition. In: Crawford, A., Diniz, N., Beckett, R., Vanucchi, J., Swackhamer, M. (Eds.), ACADIA 2023: Habits of the Anthropocene: Scarcity and Abundance in a Postmaterial Economy, vol. 1, pp. 220-225. Denver, 26-28 October 2023.

[39] Sardari, S. , Sevim, S. , Zhang, P. , Ron, G. , Leder, S. , Menges, A. , Wortmann, T. , 2024. Deep agency: towards human guided robotic training for assembly tasks in timber construction. In: Kontovourkis, O., Phocas, M.C., Wurzer, G. (Eds.), Data-Driven Intelligence-Proceedings of the 42nd Conference on Education and Research in Computer Aided Architectural Design in Europe(eCAADe 2024), Nicosia, 11-13 September 2024, vol. 1. CUMINCAD, pp. 193-202.

[40] Skevaki, E. , Kladeftira, M. , Pittiglio, A.N. , Parascho, S. , 2024. Assembly of spaceframes in hybrid teams: adaptive digital fabrication workflows for human-robot collaboration. In: Gardner, Nicole, Herr, Christiane M., Wang, Likai, Toshiki, Hirano, Ahmad Khan, Sumbul (Eds.), ACCELERATED DESIGN-Proceedings of the 29th CAADRIA Conference, vol. 3, pp. 291-300. Singapore, 20-26 April 2024.

[41] Song, Y. , Hahm, S. , 2023. Augmented robotic bricklaying. In: Koh, Immanuel, Reinhardt, Dagmar, Makki, Mohammed, Khakhar, Mona, Bao, Nic (Eds.), HUMAN-CENTRIC-Proceedings of the 28th CAADRIA Conference, pp. 323-332. Ahmedabad, 18-24 March 2023.

[42] Tala, C. , König, J. , Benavides, P. , Véliz, F. , 2023. Robotic incremental sheet metal forming: waste management through digital fabrication. In: García Amen, F., Goni Fitipaldo, A.L., Armagno Gentile, Á. (Eds.), Accelerated Landscapes-Proceedings of the XXVII International Conference of the IberoAmerican Society of Digital Graphics (SIGraDi 2023), Punta Del Este, Maldonado, Uruguay, 29 November-1 December 2023, pp. 657-668.

[43] Wang, S. , Zhang, Y. , Liu, Y. , Zheng, Y. , Shrestha, N. , Shi, B. , Zhuo, X. , Yuan, P.F. , 2025. Digital bamboo weaving: a study on bamboo pavilion based on generative design and robotic fabrication. In: Reinhardt, Dagmar, Rogeau, Nicolas, Herr, Christiane M., Globa, Anastasia, Chen, Jielin, Narahara, Taro (Eds.), ARCHITECTURAL INFORMATICS-Proceedings of the 30th CAADRIA Conference, vol. 2, pp. 521-530. Tokyo, 22-29 March 2025.

[44] Wei, J. , Llach, D.C. , Bard, J. , 2023. Responsive robotic assembly system with heterogeneous materials: a case study with unprocessed wood. In: Crawford, A., Diniz, N., Beckett, R., Vanucchi, J., Swackhamer, M. (Eds.), ACADIA 2023: Habits of the Anthropocene: Scarcity and Abundance in a Post-material Economy, vol. 2, pp. 352-360. Denver, 26-28 October 2023.

[45] Xiang, Y. , Wang, X. , Hu, S. , Zhu, B. , Huang, X. , Wu, X. , Lyu, S. , 2022. RMBench: benchmarking deep reinforcement learning for robotic manipulator control. arXiv: 2210.11262.

[46] Xiong, Y. , Wang, Y. , 2025. Placing nature: interactive form-finding using computer vision and robotic arm collaboration for natural timber structure. In: Reinhardt, Dagmar, Rogeau, Nicolas, Herr, Christiane M., Globa, Anastasia, Chen, Jielin, Narahara, Taro (Eds.), ARCHITECTURAL INFORMATICS-Proceedings of the 30th CAADRIA Conference, vol. 2, pp. 295-304. Tokyo, 22-29 March 2025.

[47] Yen, C.-C. , 2025. Computational design for single robot timber frame assembly: optimized workflow and robotic toolpath computation for customizable timber frame structures. In: Reinhardt, Dagmar, Rogeau, Nicolas, Herr, Christiane M., Globa, Anastasia, Chen, Jielin, Narahara, Taro (Eds.), ARCHITECTURAL INFORMATICS-Proceedings of the 30th CAADRIA Conference, vol. 2, pp. 213-222. Tokyo, 22-29 March 2025.

[48] Yuji Yasutomi, A. , Mori, H. , Ogata, T. , 2021. A peg-in-hole task strategy for holes in concrete. In: 2021 IEEE International Conference on Robotics and Automation (ICRA). Presented at the 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 2205-2211.

[49] Zhang, P. , Sevim, S. , Leder, S. , Maierhofer, M. , Schwinn, T. , Menges, A. , 2024. Multi-robotic maypole braiding. In: Kontovourkis, O., Phocas, M.C., Wurzer, G. (Eds.), Data-Driven Intelligence-Proceedings of the 42nd Conference on Education and Research in Computer Aided Architectural Design in Europe(eCAADe 2024), vol. 1, pp. 137-146. Nicosia, 11-13 September 2024.

[50] Zhu, A. , Dai, T. , Xu, G. , Pauwels, P. , De Vries, B. , Fang, M. , 2023. Deep reinforcement learning for real-time assembly planning in robot-based prefabricated construction. IEEE Trans. Autom. Sci. Eng. 20, 1515- 1526.

[51] Zhu, A. , Xu, G. , Pauwels, P. , de Vries, B. , Fang, M. , 2021. Deep reinforcement learning for prefab assembly planning in robotbased prefabricated construction. In: 2021 IEEE 17th International Conference on Automation Science and Engineering(CASE). Presented at the 2021 IEEE 17th International Conference on Automation Science and Engineering (CASE), pp. 1282-1288.

RIGHTS & PERMISSIONS

The Author(s). Publishing services by Elsevier B.V. on behalf of Higher Education Press and KeAi.