Creative interior design matching the indoor structure generated through diffusion model with an improved control network

Abstract

AI-driven interior design generation offers promising applications. However, current AI-based diffusion models struggle to generate indoor layouts in pixel-level alignment with the indoor structure. This study proposes a new stable diffusion-based interior design workflow with an Interior Design Control Network (IDCN). IDCN ensures that the batch-generated creative interior designs based on an input image of an unfurnished room match the indoor structure. Generating innovative designs and rendering images directly with the proposed method eliminates the tedious creative design and drawing work in traditional design practices. The results indicate that the proposed method with the new design approach achieves nearly real-time design generation and modification and significantly enhances design creativity and efficiency. Moreover, the proposed method can be generalized to other design generation tasks, thereby promoting the transformation toward intelligent design.

Keywords

Interior design / Structural matching / AI design / Creativity / Control network / Design workflow

References

[1] Alaçam, S. , Karadag, I. , Güzelci, O.Z. , 2022. Reciprocal style and information transfer between historical istanbul pervititch maps and satellite views using machine learning. Estoa. Revista de la Facultad de Arquitectura y Urbanismo de la Universidad de Cuenca 11, 7.

[2] Ashour, M. , Mahdiyar, A. , Haron, S.H. , Hanafi, M.H. , 2022. Barriers to the practice of sustainable interior architecture and design for interior renovations: a parsimonious-cybernetic fuzzy ahp approach. J. Clean. Prod. 366, 132958.

[3] Baqutayan, S.M. , Raji, F. , Ariffin, A.S. , 2015. The impact of housing conditions on the emotion, behaviour, and psychological wellbeing of middle-income groups. J. Sustain. Dev. 8, 262.

[4] Borji, A. , 2022. Generated faces in the wild: quantitative comparison of stable diffusion, midjourney and dall-e 2. ArXiv: 2210.00586.

[5] Brown, N.C. , Mueller, C.T. , 2019. Design variable analysis and generation for performance-based parametric modeling in architecture. Int. J. Architect. Comput. 17, 36- 52.

[6] Caetano, I. , Leitão, A. , 2019. Integration of an algorithmic bim approach in a traditional architecture studio. Journal of Computational Design and Engineering 6, 327- 336.

[7] Chen, J. , Shao, Z. , Cen, C. , Li, J. , 2024a. Hynet: a novel hybrid deep learning approach for efficient interior design texture retrieval. Multimed. Tool. Appl. 83, 28125- 28145.

[8] Chen, J. , Shao, Z. , Zheng, X. , Zhang, K. , Yin, J. , 2024b. Integrating aesthetics and efficiency: ai-driven diffusion models for visually pleasing interior design generation. Sci. Rep. 14, 3496.

[9] Chen, J. , Shao, Z. , Hu, B. , 2023a. Generating interior design from text: a new diffusion model-based method for efficient creative design. Buildings 13, 1861.

[10] Chen, J. , Shao, Z. , Zhu, H. , Chen, Y. , Li, Y. , Zeng, Z. , Yang, Y. , Wu, J. , Hu, B. , 2023b. Sustainable interior design: a new approach to intelligent design and automated manufacturing based on grasshopper. Comput. Ind. Eng. 183, 109509.

[11] Chen, J. , Wang, D. , Shao, Z. , Zhang, X. , Ruan, M. , Li, H. , Li, J. , 2023c. Using artificial intelligence to generate master-quality architectural designs from text descriptions. Buildings 13, 2285.

[12] Cheng, S.I. , Chen, Y.J. , Chiu, W.C. , Tseng, H.Y. , Lee, H.Y. , 2023a. Adaptively-realistic image generation from stroke and sketch with diffusion model. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4054-4062.

[13] Cheng, W. , Chu, Y. , Xia, C. , Zhang, B. , Chen, J. , Jia, M. , Wang, W. , 2023b. Urbangenogan: pioneering urban spatial planning using the synergistic integration of gan, ga, and gis. Front. Environ. Sci. 11, 1287858.

[14] Colenberg, S. , Jylhä, T. , 2021. Identifying interior design strategies for healthy workplaces—a literature review. J. Corp. R. Estate 24, 173- 189.

[15] Croitoru, F.A. , Hondru, V. , Ionescu, R.T. , Shah, M. , 2023. Diffusion models in vision: a survey. IEEE Trans. Pattern Anal. Mach. Intell.

[16] Ding, M. , Zheng, W. , Hong, W. , Tang, J. , 2022. Cogview2: faster and better text-to-image generation via hierarchical transformers. Adv. Neural Inf. Process. Syst. 35, 16890- 16902.

[17] Dolatabadi, H.M. , Erfani, S. , Leckie, C. , 2023. The devil's advocate: shattering the illusion of unexploitable data using diffusion models. ArXiv: 2303.08500.

[18] Gane, V. , Haymaker, J. , 2012. Design scenarios: enabling transparent parametric design spaces. Adv. Eng. Inf. 26, 618- 640.

[19] Gao, J. , Foong, P.S. , Yang, Y. , Jiang, W. , Chen, Y. , Ying, X. , Perrault, S. , 2022. From 996 to 007: challenges of working from home during the epidemic in China. ArXiv: 2201.09045.

[20] Hill, C. , Hegde, A.L. , Matthews, C. , 2014. Throwing in the towel: burnout among practicing interior designers. J. Interior Des. 39, 41- 60.

[21] Ho, J. , Jain, A. , Abbeel, P. , 2020. Denoising diffusion probabilistic models. Adv. Neural Inf. Process. Syst. 33, 6840- 6851.

[22] Hu, E.J. , Shen, Y. , Wallis, P. , Allen-Zhu, Z. , Li, Y. , Wang, S. , Wang, L. , Chen, W. , 2021. Lora: low-rank adaptation of large language models. ArXiv: 2106.09685.

[23] Huang, N. , Tang, F. , Dong, W. , Lee, T.Y. , Xu, C. , 2023. Region-aware diffusion for zero-shot text-driven image editing. ArXiv: 2302.11797.

[24] Ibadullaev, I. , Atoshov, S. , 2019. The effects of colors on the human mind in the interior design. Indonesian Journal of Innovation Studies 7.

[25] Jiang, Z. , Wen, H. , Han, F. , Tang, Y. , Xiong, Y. , 2022. Data-driven generative design for mass customization: a case study. Adv. Eng. Inf. 54, 101786.

[26] Karadag, I. , Güzelci, O.Z. , Alaçam, S. , 2023. Edu-ai: a twofold machine learning model to support classroom layout generation. Construct. Innovat. 23, 898- 914.

[27] Kawar, B. , Zada, S. , Lang, O. , Tov, O. , Chang, H. , Dekel, T. , Mosseri, I. , Irani, M. , 2023. Imagic: text-based real image editing with diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6007-6017.

[28] Khalil, A. , Tolba, O. , Ezzeldin, S. , 2023. Optimization of an office building form using a lattice incubate boxes method. Adv. Eng. Inf. 55, 101847.

[29] Kim, J. , Lee, J.K. , 2020. Stochastic detection of interior design styles using a deep-learning model for reference images. Appl. Sci. 10, 7299.

[30] Lee, K. , 2022. The interior experience of architecture: an emotional connection between space and the body. Buildings 12, 326.

[31] Lee, S. , Alzoubi, H.H. , Kim, S. , 2017. The effect of interior design elements and lighting layouts on prospective occupants' perceptions of amenity and efficiency in living rooms. Sustainability 9, 1119.

[32] Li, Y. , Liu, H. , Wu, Q. , Mu, F. , Yang, J. , Gao, J. , Li, C. , Lee, Y.J. , 2023. Gligen: open-set grounded text-to-image generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22511-22521.

[33] Liu, X. , Park, D.H. , Azadi, S. , Zhang, G. , Chopikyan, A. , Hu, Y. , Shi, H. , Rohrbach, A. , Darrell, T. , 2023. More control for free! image synthesis with semantic diffusion guidance. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 289-299.

[34] Longhurst, N. , Hargreaves, T. , 2019. Emotions and fuel poverty: the lived experience of social housing tenants in the United Kingdom. Energy Res. Social Sci. 56, 101207.

[35] Mou, C. , Wang, X. , Xie, L. , Zhang, J. , Qi, Z. , Shan, Y. , Qie, X. , 2023. T2i-adapter: learning adapters to dig out more controllable ability for text-to-image diffusion models. ArXiv: 2302.08453.

[36] Nasir, S. , Zahid, M.N. , Khan, T.A. , Kadir, K. , Khan, S. , 2018. Augmented reality application for architects and interior designers: interno a cost effective solution. In: 2018 IEEE 5th International Conference on Smart Instrumentation, Measurement and Application (ICSIMA). IEEE, pp. 1-6.

[37] Obukhov, A. , Krasnyanskiy, M. , 2020. Quality assessment method for gan based on modified metrics inception score and fréchet inception distance. In: Software Engineering Perspectives in Intelligent Systems: Proceedings of 4th Computational Methods in Systems and Software 2020, 1 4. Springer, pp. 102-114.

[38] Ploennigs, J. , Berger, M. , 2023. AI art in architecture. AI in Civil Engineering 2 (1), 8.

[39] Pober, E. , Cook, M. , 2019. Thinking in virtual spaces: impacts of virtual reality on the undergraduate interior design process. International Journal of Virtual and Augmented Reality (IJVAR) 3, 23- 40.

[40] Radford, A. , Kim, J.W. , Hallacy, C. , Ramesh, A. , Goh, G. , Agarwal, S. , Sastry, G. , Askell, A. , Mishkin, P. , Clark, J. , et al., 2021. Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. PMLR, pp. 8748-8763.

[41] Rashdan, W. , Ashour, A.F. , 2017. Criteria for sustainable interior design solutions. WIT Trans. Ecol. Environ. 223, 311- 322.

[42] Rombach, R. , Blattmann, A. , Lorenz, D. , Esser, P. , Ommer, B. , 2022. High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695.

[43] Ruiz, N. , Li, Y. , Jampani, V. , Pritch, Y. , Rubinstein, M. , Aberman, K. , 2023. Dreambooth: fine tuning text-to-image diffusion models for subject-driven generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22510.

[44] Schuhmann, C. , Vencu, R. , Beaumont, R. , Kaczmarczyk, R. , Mullis, C. , Katta, A. , Coombes, T. , Jitsev, J. , Komatsuzaki, A. , 2021. Laion-400m: open dataset of clip-filtered 400 million image-text pairs. ArXiv: 2111.02114.

[45] Shao, Z. , Chen, J. , Zeng, H. , Hu, W. , Xu, Q. , Zhang, Y. , 2024. A new approach to interior design: generating creative interior design videos of various design styles from indoor texture-free 3d models. Buildings 14, 1528.

[46] Tabadkani, A. , Shoubi, M.V. , Soflaei, F. , Banihashemi, S. , 2019. Integrated parametric design of adaptive facades for user's visual comfort. Autom. ConStruct. 106, 102857.

[47] Turrin, M. , von Buelow, P. , Stouffs, R. , 2011. Design explorations of performance driven geometry in architectural design using parametric modeling and genetic algorithms. Adv. Eng. Inf. 25, 656- 675.

[48] Van Le, T. , Phung, H. , Nguyen, T.H. , Dao, Q. , Tran, N.N. , Tran, A. , 2023. Anti-dreambooth: protecting users from personalized text-to-image synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2116-2127.

[49] Von Oswald, J. , Henning, C. , Grewe, B.F. , Sacramento, J. , 2019. Continual learning with hypernetworks. ArXiv: 1906.00695.

[50] Voynov, A. , Aberman, K. , Cohen-Or, D. , 2023. Sketch-guided textto-image diffusion models. In: ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11.

[51] Wang, Z. , Bovik, A.C. , Sheikh, H.R. , Simoncelli, E.P. , 2004. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600- 612.

[52] Xiao, Y. , Becerik-Gerber, B. , Lucas, G. , Roll, S.C. , 2021. Impacts of working from home during covid-19 pan demic on physical and mental well-being of office workstation users. J. Occup. Environ. Med. 63, 181.

[53] Yang, C. , Liu, F. , Ye, J. , 2023. A product form design method integrating kansei engineering and diffusion model. Adv. Eng. Inf. 57, 102058.

[54] Yang, L. , Zhang, Z. , Song, Y. , Hong, S. , Xu, R. , Zhao, Y. , Zhang, W. , Cui, B. , Yang, M.H. , 2022. Diffusion models: a comprehensive survey of methods and applications. ACM Comput. Surv. 56 (4), 1- 39.

[55] Zhang, J. , Liu, N. , Wang, S. , 2021. Generative design and performance optimization of residential buildings based on parametric algorithm. Energy Build. 244, 111033.

[56] Zhang, L. , Rao, A. , Agrawala, M. , 2023. Adding conditional control to text-to-image diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847.

[57] Zhu, A. , 2021. Application of artificial intelligence technology and embedded digital image in interior design. Microprocess. Microsyst. 81, 103782.

RIGHTS & PERMISSIONS

The Author(s). Publishing services by Elsevier B.V. on behalf of Higher Education Press and KeAi.