RepColor: deep coloring algorithm combining semantic categories

Abstract

Image coloring is an inherently uncertain and multimodal problem. By inputting a grayscale image into a coloring network, visually plausible colored photos can be generated. Conventional methods primarily rely on semantic information for image colorization. These methods still suffer from color contamination and semantic confusion. This is largely due to the limited capacity of convolutional neural networks to learn deep semantic information inherent in images effectively. In this paper, we propose a network structure that addresses these limitations by leveraging multi-level semantic information classification and fusion. Additionally, we introduce a global semantic fusion network to combat the issues of color contamination. The proposed coloring encoder accurately extracts object-level semantic information from images. To further enhance visual plausibility, we employ a self-supervised adversarial training method. We train the network structure on various datasets with varying amounts of data and evaluate its performance using the ImageNet validation set and COCO validation set. Experimental results demonstrate that our proposed algorithm can generate more realistic images compared to previous approaches, showcasing its high generalization ability.

Keywords

A

References

[1] Ma M, Hu X, Zhang R, et al.. Rendered image denoising method with filtering guided by lighting information. Optoelectronics letters, 2025, 21(4): 242-248 J]

[2] Nazeri K, Ng E, Ebrahimi M. Image colorization using generative adversarial networks. 10th International Conference on Articulated Motion and Deformable Objects, July 12–13, 2018, Palma de Mallorca, Spain, 2018, Heidelberg, Springer International Publishing8594[C]

[3] Chen J, Shen Y, Gao J, et al.. Language-based image editing with recurrent attentive models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 18–22, 2018, Salt Lake City, USA, 2018, New York, IEEE87218729[C]

[4] Amemiya T, Leow C S. Appropriate grape color estimation based on metric learning for judging harvest timing. Optoelectronics letters, 2022, 38: 4083-4094[J]

[5] Zhao Y, Po L M, Cheung K W. SCGAN: saliency map-guided colorization with generative adversarial network. IEEE transactions on circuits and systems for video technology, 2021, 31(8): 3062-3077 J]

[6] Xian W, Sangkloy P, Agrawal V. TextureGAN: controlling deep image synthesis with texture patches. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18–22, 2018, Salt Lake City, USA, 2018, New York, IEEE84568465[C]

[7] Zhu J Y, Park T, Isola P, et al.. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, October 22–29, 2017, Venice, Italy, 2017, New York, IEEE22232232[C]

[8] Song Y, Ren S, Lu Y. Deep learning-based automatic segmentation of images in cardiac radiography: a promising challenge. Computer methods and programs in biomedicine, 2022, 220: 106821 J]

[9] Ding X, Zhang X, Ma N, et al.. RepVGG: making VGG-style convnets great again. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 19–25, 2021, virtual, 2021, New York, IEEE13733-13742[C]

[10] Berman D, Levy D, Avidan S, et al.. Underwater single image color restoration using haze-lines and a new quantitative dataset. IEEE transactions on pattern analysis and machine intelligence, 2020, 43(8): 2822-2837[J]

[11] Zhang R, Isola P, Efros A A. Colorful image colorization. 14th European Conference on Computer Vision, October 11–14, 2016, Amsterdam, The Netherlands, 2016, Heidelberg, Springer International Publishing649666[C]

[12] Luan F, Paris S, Shechtman E. Deep painterly harmonization. Computer graphics forum, 2018, 37(4): 95-106 J]

[13] Zhang L, Ji Y, Lin X. Style transfer for anime sketches with enhanced residual U-net and auxiliary classifier gan. 4th IAPR Asian Conference on Pattern Recognition, May 8–12, Hamar, Norwa, 2017, New York, IEEE506511[C]

[14] Cheng Z, Yang Q, Sheng B. Deep colorization. 2015 IEEE International Conference on Computer Vision, December 7–13, 2015, Santiago, Chile, 2015, New York, IEEE415423[C]

[15] Iizuka S, Simo-Serra E, Ishikawa H. Let there be color! joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM transactions on graphics (ToG), 2016, 35(4): 1-11 J]

[16] Larsson G, Maire M, Shakhnarovich G. Learning representations for automatic colorization. 14th European Conference on Computer Vision, October 11–14, 2016, Amsterdam, The Netherlands, 2016, Heidelberg, Springer International Publishing577593[C]

[17] Zhang R, Zhu J Y, Isola P, et al.. Real-time user-guided image colorization with learned deep priors. IEEE Conference on Computer Vision and Pattern Recognition, July 21–26, 2017, Honolulu, USA, 2017, New York, IEEE111[C]

[18] Su J, Chu H, Huang J. Instance-aware image colorization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 14–19, 2020, Virtual, 2020, New York, IEEE79687977[C]

[19] Vitoria P, Raad L, Ballester C. ChromaGAN: adversarial picture colorization with semantic class distribution. 2020 IEEE Winter Conference on Applications of Computer Vision, March 2–5, 2020, Snowmass Village, CO, USA, 2020, New York, IEEE24342443[C]

[20] Kumar M, Weissenborn D, Kalchbrenner S N. Colorization transformer. 2021 International Conference on Learning Representations, 2021[C]

[21] Weng S, Sun J, Li Y, et al.. CT2: colorization transformer via color tokens. European Conference on Computer Vision, October 23–27, 2022, Tel-Aviv, Israel, 2022, Cham, Springer Nature Switzerland116[C]

[22] Woo S, Park J, Lee J Y. CBAM: convolutional block attention module. European Conference on Computer Vision, July 17, 2018, Munich, Germany, 2018, Heidelberg, Springer319[C]

[23] Li C, Zhou A, Yao A. Omni-dimensional dynamic convolution. 2022 International Conference on Learning Representations, April 25, 2022[C]

[24] Selvaraju R R, Cogswell M, Das A. Grad-CAM: visual explanations from deep networks via gradient-based localization. International journal of computer vision, 2020, 128(2): 336-359 J]

[25] GULRAJANI I, AHMED F, ARJOVSKY M, et al. Improved training of Wasserstein GANs[J]. Advances in neural information processing systems, 2017, 30.

[26] JASON A, DANA K. DeOldify: a deep learning based project for colorizing and restoring old images[EB/OL]. (2024-10-19) [2025-02-23]. https://github.com/jantic/Deoldify.

[27] Li B, Lu Y, Pang W, et al.. Image colorization using CycleGAN with semantic and spatial rationality. Multimedia tools and applications, 2023, 82(14): 21641-21655 J]

RIGHTS & PERMISSIONS

Tianjin University of Technology