Line drawing via saliency map and ETF

Abstract

Line drawing is a means of superior visual communication which can effectively convey useful information to viewers. Artists usually draw what they see rather than what exists, which means the most attractive object is emphasized while the rest are inhibited. Moreover, artists draw the whole object with coherent lines instead of fractured lines, which also contribute to the outstanding visual effect. From these perspectives, generating line drawings with saliency and coherence remains a great challenge. Existing line drawing generation methods ignore these important properties and cannot generate coherent lines in some cases since they do not take into account how artists draw a picture. To this end, a novel saliency-aware line drawing method was proposed to better grasp the viewer’s attention on an image. First, a saliency enhanced line extraction method combining saliency map and edge tangent flow was proposed to ensure the saliency and coherence of lines, especially in salient but low contrast areas. Then, the salient information was highlighted while irrelevant details were eliminated by inhibiting lines in less salient areas to enhance the saliency of the line drawing. Finally, the transparency of lines was adjusted to further emphasize important information. Various results showed that our method can generate line drawings with better visual saliency and coherence than the state-of-the-art methods.

Graphical abstract

Keywords

line drawing / saliency / coherence

References

[1] Son M, Kang H, Lee Y, Lee S. Abstract line drawings from 2D images. In: Proceedings of IEEE Pacific Conference on Computer Graphics and Applications. 2007, 333– 342

[2] Zhou J, Li B. Automatic generation of pencil-sketch like drawings from personal photos. In: Proceedings of IEEE International Conference on Multimedia and Expo. 2005, 1026−1029

[3] He J, Wang S, Zhang Y, Zhang J. A computational fresco sketch generation framework. In: Proceedings of IEEE International Conference on Multimedia and Expo. 2013, 1– 6

[4] Choi J , Cho H , Song J , Yoon S M . Sketchhelper: real-time stroke guidance for freehand sketch retrieval. IEEE Transactions on Multimedia, 2019, 21( 8): 2083– 2092

[5] Hata M , Toyoura M , Mao X . Automatic generation of accentuated pencil drawing with saliency map and LIC. Visual Computer, 2012, 28( 6−8): 657– 668

[6] Doug D , Anthony S . Stylization and abstraction of photographs. ACM Transactions on Graphics, 2002, 21( 3): 769– 776

[7] Winnemoeller H, Xie J, Hertzmann A, Li W. Portraitsketch: face sketching assistance for novices. In: Proceedings of Annual ACM Symposium on User Interface Software and Technology. 2014, 407– 417

[8] Wu Y T , Yeh J S , Wu F C , Chuang Y Y . Tangent-based binary image abstraction. Journal of Imaging, 2017, 3( 2): 16– 30

[9] Kang H , Lee S , Chui C K . Flow-based image abstraction. IEEE Transactions on Visualization and Computer Graphics, 2009, 15( 1): 62– 76

[10] Orzan A, Bousseau A, Barla P, Thollot J. Structurepreserving manipulation of photographs. In: Proceeding of International Symposium on Non-Photorealistic Animation and Rendering. 2007, 103– 110

[11] Rosin P L , Lai Y K . Artistic minimal rendering with lines and blocks. Graphical Models, 2013, 75( 4): 208– 229

[12] Kang H, Lee S, Chui C K. Coherent line drawing. In: Proceedings of International Symposium on NonPhotorealistic Animation and Rendering. 2007, 43– 50

[13] Wang S , Wu E , Liu Y , Liu X , Chen Y . Abstract line drawings from photographs using flow-based filters. Computers and Graphics, 2012, 36( 4): 224– 231

[14] Li M, Lin Z, Mech R, Yumer E, Ramanan D. Photosketching: inferring contour drawings from images. In: Proceedings of Winter Conference on Applications of Computer Vision. 2019, 1– 10

[15] Liu Y, Cheng M M, Hu X, Wang K, Bai X. Richer convolutional features for edge detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2017, 3000−3009

[16] Xie S , Tu Z . Holistically-nested edge detection. International Journal of Computer Vision, 2017, 125( 1): 3– 18

[17] Sheng B , Li P , Gao C , Ma K L . Deep neural representation guided face sketch synthesis. IEEE Transactions on Visualization and Computer Graphics, 2019, 25( 12): 3216– 3230

[18] Kang H W , Chui C K , Chakraborty U K . A unified scheme for adaptive stroke-based rendering. Visual Computer, 2006, 22 : 9– 11

[19] Gooch B , Reinhard E , Gooch A . Human facial illustrations: creation and psychophysical evaluation. ACM Transactions on Graphics, 2004, 23( 1): 27– 44

[20] Nie Y , Cao X , Li P , Zhang Q , Zhang Z , Li G , Sun H . Interactive contour extraction via sketch-alike densevalidation optimization. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30( 4): 906– 916

[21] Wang S , Ao Z , Yao S , Chen Y , Wang G . Adaptive line drawing. Journal of Image and Graphics, 2018, 23( 5): 730– 739

[22] Zhang J, Wani R Z, Xu D. Automatic generation of sketch-like pencil drawing from image. In: Proceedings of IEEE International Conference on Multimedia and Expo. 2017, 261– 266

[23] Li R , Sun S , Yang X , Hu W . Automatic generation of pencil drawings based on multi-level saliency maps. Journal of Computer-Aided Design and Computer Graphics, 2018, 30( 1): 108– 116

[24] Alexe B , Deselaers T , Ferrari V . Measuring the objectness of image windows. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34( 11): 2189– 2202

[25] Yan Q, Xu L, Shi J, Jia J. Hierarchical saliency detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2013, 2013−1162

[26] Gu K , Wang S , Yang H , Lin W , Zhai G , Yang X , Member S , Zhang W . Saliency-guided quality assessment of screen content images. IEEE Transactions on Multimedia, 2016, 18( 6): 1098– 1100

[27] Huang Z, Liu S. Perceptual hashing with visual content understanding for reduced-reference screen content image quality assessment. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(7): 2808−2823

[28] Gao R, Huang Z, Liu S. Multi-task deep learning for no-reference screen content image quality assessment. In: Proceedings of the 27th International Conference on Multimedia Modeling. 2021, 213– 226

[29] Jerripothula K R , Cai J , Yuan J . Image co-segmentation via saliency co-fusion. IEEE Transactions on Multimedia, 2016, 18( 9): 1896– 1909

[30] Quan R , Han J , Member S , Zhang D , Nie F , Qian X , Li X . Unsupervised salient object detection via inferring from imperfect saliency models. IEEE Transactions on Multimedia, 2017, 20( 5): 1101– 1112

[31] Zhao Y , Liu S , Hu Z , Bai Y , Shen C , Shi X . Separate degree based otsu and signed similarity driven level set for segmenting and counting anthrax spores. Computers and Electronics in Agriculture, 2020, 169 : 1– 15

[32] Qian X , Wang H , Zhao Y , Hou X , Hong R , Wang M , Tang Y Y . Image location inference by multisaliency enhancement. IEEE Transactions on Multimedia, 2017, 19( 4): 813– 821

[33] Fang Y , Lin W , Member S , Lee B S , Lau C T , Chen Z , Lin C W . Bottom-up saliency detection model based on human visual sensitivity and amplitude spectrum. IEEE Transactions on Multimedia, 2012, 14( 1): 187– 198

[34] Imamoglu N , Lin W , Fang Y . A saliency detection model using low-level features based on wavelet transform. IEEE Transactions on Multimedia, 2012, 14( 1): 187– 198

[35] Yang C, Zhang L, Lu H, Ruan X, Yang M H. Saliency detection via graph-based manifold ranking. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2013, 3166−3173

[36] Achanta R, Hemami S, Estrada F, Susstrunk S. Frequency-tuned salient region detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2009, 1597−1604

[37] Cabral B, Leedom L C. Imaging vector fields using line integral convolution. In: Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques. 1993, 263– 270

[38] Winnemller H , Gooch B , Olsen S C . Real-time video abstraction. ACM Transactions on Graphics, 2006, 25( 3): 1221– 1226

[39] Jin Y, Li P, Wang W, Zhang S, Lin D, Yin C. Ganbased pencil drawing learning system for art education on large-scale image datasets with learning analytics. Interactive Learning Environments, 2019

[40] Chen Z, Jin Y, Sheng B, Li P, Sun H. Parallel pencil drawing stylization via structure-aware optimization. In: Proceedings of the 31st International Conference on Computer Animation and Social Agents. 2018, 32– 37

[41] Jin Y , Li P , Sheng B , Nie Y , Kim J , Wu E . SRNPD: spatial rendering network for pencil drawing stylization. Computer Animation and Virtual Worlds, 2019, 30( 3−4): 1– 13

[42] Song Z , Liu S . Sufficient image appearance transfer combining color and texture. IEEE Transactions on Multimedia, 2019, 19( 4): 702– 711

[43] Liu S , Song Z , Zhang X , Zhu T . Progressive complex illumination image appearance transfer based on CNN. Journal of Visual Communication and Image Representation, 2019, 64 : 1– 11

[44] Zhu T, Liu S. Detail-preserving arbitrary style transfer. In: Proceedings of IEEE International Conference on Multimedia and Expo. 2020, 1– 6

[45] Liu S, Zhu T. Structure-guided arbitrary style transfer for artistic image and video. IEEE Transactions on Multimedia, 2021

[46] Li P , Sun H , Sheng B , Shen J . Image stylization with enhanced structure on GPU. Science China Information Sciences, 2012, 55( 5): 1093– 1105

[47] Wang R, Li P, Sheng B, Sun H, Wu E. Real-time video stylization using spatial-temporal gabor filtering. In: Proceedings of the 15th ACM SIGGRAPH Conference on Virtual-Reality Continuum and Its Applications in Industry. 2016, 299– 307

Acknowledgements

This work was partly supported by the National Natural Science Foundation of China (Grant Nos. 62072328 and 61672375).

RIGHTS & PERMISSIONS

2022 Higher Education Press