A CCD based machine vision system for real-time text detection

Abstract

Text detection and recognition is a hot topic in computer vision, which is considered to be the further development of the traditional optical character recognition (OCR) technology. With the rapid development of machine vision system and the wide application of deep learning algorithms, text recognition has achieved excellent performance. In contrast, detecting text block from complex natural scenes is still a challenging task. At present, many advanced natural scene text detection algorithms have been proposed, but most of them run slow due to the complexity of the detection pipeline and cannot be applied to industrial scenes. In this paper, we proposed a CCD based machine vision system for real-time text detection in invoice images. In this system, we applied optimizations from several aspects including the optical system, the hardware architecture, and the deep learning algorithm to improve the speed performance of the machine vision system. The experimental data confirms that the optimization methods can significantly improve the running speed of the machine vision system and make it meeting the real-time text detection requirements in industrial scenarios.

Keywords

machine vision / text detection / optical character recognition (OCR) / deep learning

1 Introduction

Long-period fiber gratings (LPFGs) have been found wide applications in optical communications and smart sensing [1–5]. Among various techniques for fabrication of LPFGs [2], infrared femtosecond laser writing [5–9] is one of the most attractive methods in terms of its flexibility, being applicable to either photo sensitive or insensitive fibers, and possessing no needs for any pre-designed masks. Because the radius of focused femtosecond laser beam is typically a few micrometers or less, both the dimension and location of modified refractive index region can be thus conveniently altered according to specific requirement. This leads to the attractive advantage of writing LPFGs with near infrared femtosecond laser over with CO₂ laser in the aspect of generating more versatile grating profiles [5].

Although fabricating LPFGs by infrared femtosecond laser has been demonstrated for more than ten years [6], attenuation magnitude of the reported maximum rejection band of LPFGs fabricated with this method for standard telecommunication fiber (SMF-28) without hydrogen loading is usually no more than 10 dB [6,9], and in the meanwhile, a relatively large out-of-band loss is always present [6,9,10], which are common problems for all LPFGs fabrication techniques. Reference [11] shows that applying tension to a conventional lightly doped single-mode fiber during the CO₂-laser radiation can enhance the efficiency of LPFGs fabrication. With precise positioning technique and high repetition laser system, such a background loss can be reduced to 0.26–0.3 dB [12,13]. Nevertheless, most of the published data in this area show the maximum attenuation peak of the LPFGs is all centered around 1500 nm (LP_04–06 core-cladding coupling mode) [13], and the tuning of resonant wavelength are usually achieved by adjusting refractive index and the period of grating [14,15]. In fact, by the control of the core-cladding coupling mode, the resonant peaks can be tuned in a relatively broad spectral range. However, it is a pity that little effort has been made in shifting the maximum attenuation peak to a shorter wavelength, and understanding how the selection of a specific favorite resonant peak can be practically achieved in fabricating a LPFG device.

In this paper, we demonstrated that the core-cladding coupling mode can be effectively selected by the confinement of the processed area, and the writing efficiency in standard single mode fiber (SMF) may be greatly improved by applying a certain amount of tension to the fiber during femtosecond laser writing or post-processing the device with low energy density laser irradiation. The transmission spectra of LPFGs fabricated with different laser irradiation energies and number of grating periods in general can be rather different in terms of the total number of resonant attenuation peaks, their spectral positions, and the non-resonant attenuation background. Our experimental data also indicate that the out-of-band loss of the LPFGs can be controlled by selecting a proper duty cycle of the grating pitch in addition to the total number of the grating periods and the average laser irradiation power.

2 Experiments

Our experimental setup for writing LPFGs is shown in Fig. 1. A femtosecond laser amplifier system that produces 50 fs pulses with 1 kHz repetition rate and a central wavelength of 800 nm is used in the experiments. The femtosecond laser output is first directed by two high reflectance (HR) steering mirrors, and then focused into the fiber core by a 25× microscope objective after passing through a couple of neutral density filters and reflected by a 45° dichroic mirror. The single pulse energy irradiated on the fiber is estimated to be from 0.42 to 0.65 µJ. To achieve both fast and precise adjustment in fiber positioning, two charge-coupled device (CCD) cameras are used. One behind objective lens, as shown in Fig. 1, is used to check the accuracy in beam height alignment, and the other is placed above the laser writing site to monitor the distance between the fiber and focal lens.

A single mode optical fiber (SMF-28e, Corning Inc.) without hydrogen loading, is mounted on a three dimensional motor-controlled translation stage with a step resolution of 0.05 µm. Laser scribing along the fiber core is achieved by translating the fiber at a speed of 0.02 mm/s along the fiber axis normal to the incident laser beam. The length of the gratings fabricated by point by point (PBP) laser writing ranges from 20 to 25 mm for a grating period of 500 µm and a duty cycle of 50% unless otherwise specified.

Fig.1 Schematic diagram of experimental setup for the fabrication of LPFGs. During laser writing, the fiber is translated along its axis and normal to the incoming laser beam focused by a 25× microscopic objective. A super continuum light source (Koheras SuperK) and an optical spectrum analyzer (OSA, Ando AQ6315E) are used for in situ monitoring the transmission spectrum of the LPFGs during laser writing

Full size|PPT slide

3 Results and discussion

To ascertain that only the refractive index of the fiber core is modified, a grating is first fabricated with relatively higher pulse energy in a piece of SMF sample with femtosecond laser pulses. The processed fiber is then examined with an optical microscope of 1000× magnification for both close-in side view and the end-view of the cross section of the fiber. After making sure that the refractive index change is indeed confined within the fiber core and the change of cladding index is not observed, the pulse energy is then reduced down to 0.42 µJ to write the LPFGs.

The normalized transmission spectra for the four LPFGs written with axial tension from 0 to 3 N respectively applied to the fiber samples are presented in Fig. 2(a). One can see that LPFG is almost not formed without tension applied. This is likely due to the fact that the fiber without any tension could even slightly sag during and after mounting. When 1 N axial force is applied, the fiber is straight enough and a rejection band centered at 1296 nm emerges. And the contrast of this rejection band is however less than 10 dB. It is estimated from the couple mode theory that this rejection band is caused by the loss of the core mode coupled to LP₀₂ cladding mode. When the applied axial force is increased to 2 N, the rejection band shifts to 1293 nm, and the depth of rejection band is about 14.5 dB. As the axial force is further raised to 3 N the contrast of the rejection band reaches about 17 dB, having a peak centered at 1288 nm and a 3 dB bandwidth of only 7 nm. It is interesting to see that with increasing tension inside the fiber, the wavelength of the resonance peak has a blue shift. Also note that the height or the depth of the resonance peak increases with the tension in this particular case.

Fig.2 (a) Transmission spectra of LPFGs fabricated with 0 to 3 N axial stresses applied along the fiber respectively; (b) growth of transmission spectra as a function of length for 3 and 2 N pulling forces, respectively. In (b), for both forces of 3 and 2 N, the black curves are recorded when the grating length is 20 mm, and the red curves are measured when the grating length is 25 mm

Full size|PPT slide

According to the strain-optic theory [11,16,17], the presence of a tension reduces the refractive index of the fiber core, as well as that of fiber cladding. And the refractive index change inside the fiber core can be frozen by the process of femtosecond laser irradiation. On the other hand, when the laser irradiates on fiber, the core material absorbs photons, which leads to the increase of refractive index. Therefore, when the fiber under tension is inscribed by proper laser radiation the refractive index modulation will be weakened slightly due to the tension-induced index reduction and the larger the tension the weaker the refractive index modulation would be.

It is known that the resonant wavelength of a LPFG, λ₀, is determined by the phase-matching condition [13]:

(1)$${\lambda }_{\mathrm{c}\mathrm{l}}^{\mathrm{t}\mathrm{h}}=(n_{\mathrm{c}\mathrm{o}}-n_{\mathrm{c}\mathrm{l}}^{n_{\mathrm{t}\mathrm{h}}})\mathrm{\Lambda },$$

where $n_{\mathrm{c}\mathrm{o}}$ and $n_{\mathrm{c}\mathrm{l}}^{n_{\mathrm{t}\mathrm{h}}}$ is the effective indices of the fundamental mode within the fiber core and the $n_{\mathrm{t}\mathrm{h}}-order$ guided modes in cladding, and $\mathrm{\Lambda }$ is the length of grating period required to couple the fundamental mode to the $n_{\mathrm{t}\mathrm{h}}-order$ cladding mode. Equation (1) tells us that for a given grating period, the decrease in the effective core index will lead to a blue shift of the resonance peak wavelength. This is consistent with what we can observe in Fig. 2(a). Based on Eq. (1), we can estimate that the core refractive index modulation is decreased approximately by about 1.6 × 10^-5 when the applied tension force is increased from 1 to 3 N.

To further elaborate this point, Fig. 2(b) shows the growth of two LPFGs as a function of the overall grating length. It is found that when the length of the grating written with 2 N force applied is 20 mm the resonance transmission reaches the minimum and further increase of grating length leads to its saturation; whereas to achieve the minimum transmission for the grating written with 3 N force applied the grating length needs to be more than 25 mm.

According to Ref. [15], when the phase matching condition is met the power of the core mode coupled into the n_th-order cladding mode will vary directly with ${\sin }^{\mathrm{2}}(\kappa L)$, in which L is the grating length, κ is the coupling constant related to the overlap integration between the core mode and the specific resonant cladding mode. (κ is also directly proportional to the normalized laser-induced refractive index change). When $\mathit{\kappa }\mathit{L}={\displaystyle \frac{\mathrm{\pi }}{\mathrm{2}}}$, maximum core power transfer to cladding. Further increase L, the core power transferred to cladding is decreasing. Thus, in the case of 3 N tension force in Fig. 2(b), the fact that either a longer LPFG or larger L is required to reach the resonance transmission saturation indicates that the corresponding κ value in this case must be relatively small. When $L=\mathrm{25}\mathrm{m}\mathrm{m}$, the grating with 2 N tension is over coupled, however, the grating with 3 N tension is not over coupled. That is to say, in our case, the effective laser-induced refractive index change with 3 N tension should be smaller than that with 2 N tension. This is also confirmed by the observed phenomenon shown in Fig. 2(a) of the wavelength shift of the resonance peak for different tension forces.

Besides applying a proper pulling force along the fiber during laser inscribing, it is found that post-processing with lower dose of laser irradiation may also effectively improve the depth of LPFG’s resonant peak. For example, the transmission spectra of a LPFG with and without post-processing by further laser irradiation are showed in Fig. 3. In this case, we use laser pulses of 0.02 W average power, 0.2 mm/s sample moving speed, and the whole grating (including the modification area and non-modification area) is irradiated during the post-processing. It can be seen that the depth of LPFG’s maximum rejection band is substantially improved from 15 to 21 dB with little exacerbation of the background loss. This could be ascribed to some minor but critical increase in the core refractive index by the post-processing, which effectively helps to enhance the coupling between the core mode and cladding mode. Accordingly, the slight red-shift of the resonance peak wavelength from 1279 to 1285 nm as evident in Fig. 3 is consistent with the speculation of the minor augment of the core refractive index as result of post laser irradiation treatment.

Fig.3 Comparison of transmission spectra of LPFGs with (hollow round dots) and without (solid squares) post processing

Full size|PPT slide

Note that apart from the result given in an earlier report by our group [9], the 1300 nm resonance peak of a LPFG with a grating period of 500 µm fabricated in SMF with femtosecond laser pulses was either rather weak [6] or even not present [18]. In this work, however, we report that the transmission spectra of LPFGs with a major resonant peak near 1300 nm are repeatedly produced. Such results contrast the other studies that report LPFGs with a major resonant peak near 1550 nm [18]. The reason for such a difference is likely due to the fact that in our case the modification of the refractive index of fiber core is largely non-uniform. As we use tighter focus only part of the fiber core is processed by the laser beam, hence the coupling strength between the core mode and the different cladding modes are differentially changed.

The beam waist of the focused laser, w₂₀, can have significant impact on the actual processing area of fiber core, and this beam waist may be estimated roughly by

(2)$$w_{\mathrm{20}}={\displaystyle \frac{\lambda f}{\mathrm{\pi }{w}_{\mathrm{10}}}},$$

where $\lambda$ is the wavelength of laser, $w_{\mathrm{10}}$ is the radius of laser beam before being focused, and $f$ is the focal length of the microscope objective used, which is also inversely proportional to the numerical aperture (NA) of the objective. It is reasonable to expect that the processing area of the fiber core will be largely reduced when a 10× microscope objective with a numerical aperture of 0.25 is replaced with a 25× objective with a numerical aperture of 0.4. As it is clearly shown in Fig. 4, the LPFG fabricated with 10× microscopic objective and 0.75 µJ single pulse energy presents very different transmission characteristics from that with 25× objective. In particular, in contrast to the traces given in Fig. 2(a), the largest resonant peak now occurs at 1550 nm, corresponding to the loss of core mode coupled to LP₀₅ cladding mode. The result in Fig. 4 is in fact very similar to those reported in literatures.

Fig.4 Transmission spectra of LPFGs fabricated with a 10× objective and single pulse energy is 0.75 µJ

Full size|PPT slide

Fig.5 Evolution of LPFG transmission spectra for different total number of grating periods, indicated on the right of each plot. The traces are better viewed row by row, first starting from the one on the top left N = 2, then to its right N = 4, and then move to the bottom left N = 6, next to it is the bottom right N = 8, and then go back to the top left plot but now N = 10 and so on, up to the spectrum associated with N = 32 given in the bottom right plot

Full size|PPT slide

Variation of the transmission spectra of LPFG is shown in Fig. 5 for different total number of grating periods (N) and a fixed pulse energy of 0.52 µJ. It can be seen that multiple resonant peaks occur when N≥12. In particular, four resonant attenuation bands are marked in Fig. 5, namely the peak A is at 1315 nm, B at 1370 nm, C at 1440 nm and D at 1550 nm. For peak A, which is the first resonant attenuation peak to occur, when N = 4, it presents a 4 dB attenuation; when N = 10, the attenuation becomes more than 16 dB; further increasing N, this rejection band decreases because of over-coupled [19], that is $\kappa L>{\displaystyle \frac{\mathrm{\pi }}{\mathrm{2}}}$, and when N = 30, $\kappa L={\displaystyle \frac{\mathrm{3}\mathrm{\pi }}{\mathrm{2}}}$, it reaches a maximum value of 18 dB. For other rejection bands such as B, C, and D, when N = 12 they are barely all present, but with N increasing, they all get deepened, and the three bands can successively reach 11 dB for N equal to 24, 28 and 30, respectively. Further increasing N, they become over-coupled. Therefore, a multi-bands or a single-band LPFGs can be conveniently fabricated by selecting the number of grating periods for laser irradiation of the relatively higher single pulse energy. This clearly shows the flexibility of femtosecond laser manufacturing of LPFGs.

However, the higher pulse energy also leads to a larger out-of-band loss of more than 3 dB. Such type of background attenuation may be attributed to the Mie scattering of the processed area [20]. This is further evidenced in Fig. 6, in which light scattering or leakage from the laser processed tracks can be clearly seen. (By carefully examining these tracks one can find that light scattering is also not even because of higher energy laser irradiation, which could imply that some granule structure is formed inside the fiber.) It is very likely that the uniformity of fiber material gets more seriously spoiled for higher pulse energy irradiation than that of lower energy laser irradiation, and the granule structures formed in the core at higher irradiation power become the center of Mie scattering.

Fig.6 Microscopic image of light scattering from LPFGs when light from broadband light source is coupled into and propagating through fiber core

Full size|PPT slide

For the 25× microscope objective (NA= 0.4) that is used to focus the femtosecond laser beam into the central of the fiber, the focal spot diameter is estimated to be 0.8 µm without considering the lensing effect of the fiber itself. During laser inscribing, the axial translation speed of the fiber is set at 25 µm/s. Therefore, the number of repeated irradiation pulses at each spot position is no more than 32. Properly increasing the pulse duration so that decreasing peak intensity or lowering pulse energy with higher repetition rate may help to increase the uniformity of laser scanned area [21,22] and decrease the observed Mie scattering.

LPFGs with different duty cycles are produced, and their transmission spectra are shown in Fig. 7. It can be seen that as the duty cycle decreases from 90% which means laser on for 450 µm and off for 50 µm to 20%, namely, laser on for 100 µm and off for 400 µm within each pitch, the background attenuation is decreased approximately from 8 to 4 dB. Besides, the rejection bands also become sharper and deeper for the device of smaller duty cycle. These demonstrate that optimizing the duty cycle of LPFGs can be an effective way to decrease the out-of-band loss of LPFGs and to optimize the attenuation of LPFGs as well.

Fig.7 Transmission spectra of LPFGs written by 2 µJ laser pulses energy for three different duty cycles (Note that the grating period is 500 µm for all the gratings shown here, and thus 100/400 duty cycle means laser on for 100 µm and laser off for 400 µm)

Full size|PPT slide

4 Conclusions

LPFGs with the major resonant attenuation peak near 1300 nm were produced by infrared femtosecond laser PBP writing in a standard SMF. With a 3 N tension force applied to the fiber, the LPFGs written with lower pulse energy show a typical single rejection band over 16 dB attenuation, a 3 dB bandwidth of 7 nm, and nearly zeros out-of-band losses. It is also found that using femtosecond laser irradiation post-processing can be another effective method for enhancing the depth of the rejection band. Tighter focus of the processing laser beam, which leads to the modification of the refractive index only for part of the fiber core, is responsible for the observed change of the major resonant mode. Weakly focused processing laser beam can cause the uniform distribution of processed area inside the fiber core, which normally leads to the multi-peak LPFGs. Controlling the number of grating periods at increased laser irradiation energy can form either single-peak or multi-peak LPFGs with higher out-of-band attenuation. Such background attenuation associated with Mie scattering may be effectively decreased by optimizing the duty cycle of LPFGs.

References

[1] Contes A, Carpenter B, Case C, Satheesh S, Suresh B, Wang T, Wu J D, Ng A Y. Text detection and character recognition in scene images with unsupervised feature learning. In: Proceedings of International Conference on Document Analysis and Recognition. Beijing: IEEE, 2011, 440–445

[2] Ye Q, Doermann D. Text detection and recognition in imagery: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(7): 1480–1500

[3] Zhang X, Gao X, Tian C. Text detection in natural scene images based on color prior guided MSER. Neurocomputing, 2018, 307: 61–71

[4] Smith R. An overview of the tesseract OCR engine. In: Proceedings of International Conference on Document Analysis and Recognition. Parana: IEEE, 2007, 629–633

[5] Epshtein B, Ofek E, Wexler Y. Detecting text in natural scenes with stroke width transform. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Francisco: IEEE, 2010, 2963–2970

[6] Jaderberg M, Simonyan K, Vedaldi A, Zisserman A. Reading text in the wild with convolutional neural networks. International Journal of Computer Vision, 2016, 116(1): 1–20

[7] Ren S, He K, Girshick R, Sun J. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137–1149

[8] Liu W, Anguelov D, Erhan D, Szegedy C, Reed S E, Fu C, Berg A C. SSD: single shot MultiBox detector. In: Proceedings of European Conference on Computer Vision. Berlin: Springer, 2016, 21–37

[9] Redmon J, Divvala S K, Girshick R B, Farhadi A. You only look once: unified, real-time object detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016, 779–788

[10] Tian Z, Huang W, He T, He P, Qiao Y. Detecting text in natural image with connectionist text proposal network. In: Proceedings of European Conference on Computer Vision. Berlin: Springer, 2016, 56–72

[11] Ma J, Shao W, Ye H, Wang L, Wang H, Zheng Y, Xue X. Arbitrary-oriented scene text detection via rotation proposals. IEEE Transactions on Multimedia, 2018, 20(11): 3111–3122

[12] Liu Y, Jin L. Deep matching prior network: toward tighter multi-oriented text detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017, 3454–3461

[13] Shi B, Bai X, Belongie S. Detecting oriented text in natural images by linking segments. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 3482–3490

[14] Liao M, Shi B, Bai X, Wang X, Liu W. TextBoxes: A fast text detector with a single deep neural network. 2016, arXiv:1611.06779

[15] Liao M, Shi B, Bai X. TextBoxes++: a single-shot oriented scene text detector. IEEE Transactions on Image Processing, 2018, 27(8): 3676–3690

[16] Dai Y, Huang Z, Gao Y, Xu Y, Chen K, Guo J, Qiu W. Fused text segmentation networks for multi-oriented scene text detection. 2018, arXiv:1709.03272

[17] Hu H, Zhang C, Luo Y, Wang Y, Han J, Ding E. WordSup: exploiting word annotations for character based text detection. In: Proceedings of IEEE International Conference on Computer Vision. Venice: IEEE, 2017, 4950–4959

[18] Deng J, Dong W, Socher R, Li L J, Li K, Li F F. ImageNet: a large-scale hierarchical image database. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Miami: IEEE, 2009, 248–255

[19] Karatzas D, Shafait F, Uchida S, Iwamura M, Bigorda L G I, Mestre S R, Mas J, Mota D F, Almazàn J A, Heras L P D L. ICDAR 2013 robust reading competition. In: Proceedings of International Conference on Document Analysis and Recognition. Washington, DC: IEEE, 2013, 1484–1493

[20] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. 2014, arXiv:1409.1556

[21] Howard A G, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H. MobileNets: efficient convolutional neural networks for mobile vision applications. 2017, arXiv:1704.04861

[22] Zhang X, Zhou X, Lin M, Sun J. ShuffleNet: an extremely efficient convolutional neural network for mobile devices. 2017, arXiv:1707.01083v2

[23] Iandola F N, Han S, Moskewicz M W, Ashraf K, Dally W J, Keutzer K. SqueezeNet: AlexNet-level accuracy with 50 ´ fewer parameters and<0.5 MB model size. 2016, arXiv:1602.07360v4

RIGHTS & PERMISSIONS

2019 Higher Education Press and Springer-Verlag GmbH Germany, part of Springer Nature