Robot visual guide with Fourier-Mellin based visual tracking

Abstract

Robot vision guide is an important research area in industrial automation, and image-based target pose estimation is one of the most challenging problems. We focus on target pose estimation and present a solution based on the binocular stereo vision in this paper. To improve the robustness and speed of pose estimation, we propose a novel visual tracking algorithm based on Fourier-Mellin transform to extract the target region. We evaluate the proposed tracking algorithm on online tracking benchmark-50 (OTB-50) and the results show that it outperforms other lightweight trackers, especially when the target is rotated or scaled. The final experiment proves that the improved pose estimation approach can achieve a position accuracy of 1.84 mm and a speed of 7 FPS (frames per second). Besides, this approach is robust to the variances of illumination and can work well in the range of 250-700 lux.

Keywords

robot visual guide / target pose estimation / stereo vision / visual tracking / Fourier-Mellin transform (FMT)

1 Introduction

A beam of light, besides energy and momentum, can also carry angular momentum. In particular, the angular momentum of light is associated with its polarization, and more specifically with its circularly polarized components. An optical beam traveling in the direction of the+z axis that is circularly polarized carries an angular momentum of Sz = $\pm \hslash$ per photon ( $\hslash$ being the Planck constant divided by 2p), which is positive/negative if the circular polarization is left-handed/right-handed. This angular momentum content has significant mechanical effects: it can be transferred to a material particle (e.g., by absorption) and to set it in rotation [ 1, 2].

Less widely known is the fact that there is another way a light beam can carry angular momentum, in addition to polarization, which is associated with the transverse spatial structure of the wavefront. More precisely, this angular momentum appears when the wavefront acquires a “helical” structure, and its field spatial dependence contains a helical phase factor having the form exp(ilq), where q is the azimuthal angle of the position vector r around the beam axis z, and l is any integer. It can be shown that in this case the optical beam carries an angular momentum that is given by l $\hslash$ per photon, in addition to the polarization one. By analogy with the case of elementary material particles such as electrons, this second form of angular momentum is called orbital angular momentum (OAM), while the first form associated with polarization is referred to as spin angular momentum (SAM).

OAM of light and its applications was established as an actual research field only in 1992 [ 3]. In the years from around 1992 until today, many important results have been achieved. For example, we would mention the introduction of several methods for the generation of light beams carrying OAM [ 4– 6], the successful transfer of OAM to matter and the related manipulation of optically trapped particles exploiting the mechanical properties of OAM [ 7– 9], the proof-of-principle examples of classical and quantum communication based on OAM-encoding of the information [ 10– 16], the demonstration of methods for the generation and detection of OAM-carrying single photons and correlated photon pairs and their exploitation for quantum information [ 17– 25], the conversion of angular momentum from SAM to OAM [ 26– 29], the generation and manipulation of vector vortices with inhomogeneous states of polarization [ 30– 33], the demonstration of exploring OAM to realize multi-channel optical communication [ 34– 37], etc.

In general, OAM can be regarded as an additional degree of freedom of a light beam or even of a single photon, to be added to the standard ones ordinarily exploited in current photonic technologies. In many respects OAM resembles polarization, with which it shares many features. However, while polarization is characterized by two orthogonal basis states, OAM is defined in an unbounded space (a Hilbert space in the case of photons). Therefore, it is in principle possible to encode a much larger amount of information in the OAM space than in the polarization space, e.g., achieving a greater channel capacity. This property makes OAM highly attractive for future optical communication systems. Specifically, OAM states could be used as a different dimension to create an additional set of data carriers in a space division multiplexing (SDM) system. Moreover, OAM multiplexing does not rely on the wavelength or polarization, indicating that OAM could be used in addition to wavelength division multiplexing (WDM) and polarization division multiplexing (PDM) techniques to boost system capacity.

This paper will discuss the potential applications of OAM of light in optical communication systems, and highlight recent advances in CMOS-compatible silicon photonic integrated device technology for OAM generation, manipulation, and (de)multiplexing.

2 Potential applications of OAM of light beam in future optical communication systems

OAM division multiplexing (OAM-DM) uses the orthogonality between OAM channels as a way of multiplying the capacity of a single physical optical channel, in addition to other multiplexing schemes such as WDM and PDM. Although the usable optical bandwidth of optical communications systems can be limited by the number of WDM channels available (due to the limited gain bandwidth of Erbium-doped optical fiber amplifers (EDFAs)), OAM can be used to increase the number of channels carried by each wavelength significantly. In 2012, a demonstration showed OAM-DM capacity of well over Tb/s over one free space optical link “over meter length scale” [ 35]. Use of the integrated OAM (de-)multiplexer for free space OAM communications channels has also been demonstrated [ 34, 37]. However, free space OAM communication is limited by the need to align the transmitter and receiver in line of sight—not practical for most applications. More recently, a specially designed vortex fiber was demonstrated, and using this vortex fiber, two OAM modes with l = 1 and - 1, and two polarization multiplexed fundamental modes simultaneously propagated through a 1.1 km vortex fiber. This is the first demonstration of optical fiber based OAM-DM systems, which makes the OAM communication more promising.

The main device challenge is OAM (DE)MUX that maps the OAM dimension onto another dimension (e.g., spatial dimension) – the reported devices are large and have poor channel extinction ratio [ 34]. OAM-encoding is another way of doing communication using OAM, and it requires the development of OAM modulators that can generate and detect ensembles of OAM modes with response times in the picosecond domain, which is far from the capabilities of technology.

3 Micro-scale silicon integrated OAM devices

Although OAM offers fascinating opportunities for exploring new ideas in optical communication systems, the generation, detection and manipulation of OAM states of light has been restricted to complex and expensive components. For example, current techniques for generating optical vortices involve passing free space light beams through free space optical elements, which are either rigid (without fast switching or modulation capability) or inefficient (with typical efficiency<40%) and very expensive; electro-optically driven fast manipulation of OAM is virtually non-existent; Moreover, these OAM manipulations have relied on large scale (bulk) optical elements bolted to large optical tables, which are cumbersome to use and with no clear route to scaling or integration, thereby making them inherently inconvenient and confining them to the research laboratories, which severely limits the prospect of its wide use in future photonic systems. Photonic integration has been a major propellant for widespread applications of photonic technologies due to advantages in reliability, miniaturization, and scalability compared to bulk optics. Compact, robust and efficient planar waveguide-based OAM emitters and receivers are critical elements as they can be integrated in large numbers, interconnected via waveguides with each other and with lasers and detectors to form photonic integrated circuits (PICs).

We recently reported micron-sized silicon photonic waveguide OAM devices emitting vector optical vortices carrying well-defined, quantized and tunable OAM, and integrated OAM emitter arrays which emit multiple optical vortices simultaneously [ 38]. Here we review this work and explore its potential in future integrated OAM components and systems.

3.1 Basic concept and structure

Circular optical resonators, such as micro-rings or microdisks, support whispering gallery modes (WGMs) carrying high OAM [ 39]. WGMs are actually angular momentum eigenstates and have discrete azimuthal propagation constants v_WGM = b_pR = r resulting from the self-consistent phase requirement for resonance under the periodic boundary condition. The integer p denotes the azimuthal mode number and physically is the number of optical periods around the resonator; R is the effective radius of the WGM, and p is light’s propagation constants at R. The azimuthal propagation constant, frequently used in rotationally symmetric resonant devices [ 40], describes the optical phase shift per unit azimuthal angle. It is also a measure of angular momentum as the amount of angular momentum carried by every WGM photon is v_WGM = r. Additionally, the SAM of WGM is zero, because WGM has a purely linear polarized state in the cylindrical coordinate. Therefore the angular momentum carried by WGM is purely OAM. To extract the confined WGM into free space emission, we embed angular grating (AG) structures into the WGM resonator (Fig. 1(a)) with a periodic modulation of refractive index in the azimuthal direction.

Fig.1 (a) Integrated OAM emitter device; (b) measured radiation spectrum for a device, near field intensity distributions of the radiated beams, and measured and simulated interference patterns with left-hand circularly polarized (LHCP) and right-hand circularly polarized (RHCP) reference beams

Full size|PPT slide

The principle of operation of the device is to couple the input TE waveguide mode to the rotating WGM of the micro-ring resonator. A second order AG within the resonator then couples the rotating WGM to a vertically emitted propagating OAM mode. By matching the wavelength of the light with the micro-ring resonance and detuning from the Bragg grating resonance, this device is capable of emitting a propagating field of any desired OAM state. The OAM value carried by the output beam is simply decided by the difference between the WGM order, p, and the number of grating periods, q, according to the simple equation of l = p−q. This provides a very simple and robust method of generating OAM emission in which the OAM value is very well defined.

Crucially, we found that the generated OAM beam is a vector beam with cylindrically symmetric polarization distribution. Moreover, the beam can be decomposed into left-handed part with OAM value of l + 1 and right-handed part with l−1, see Fig. 1(b).

3.2 Experimental results

We designed and fabricated the device with radius of R = 3.9 mm on a silicon-on-insulator chip, with their resonance associated with l = 0 to be near the center of our tunable laser’s wavelength range (1470–1580 nm). The emission spectrum and the interference pattern are shown in Fig. 1(b).

Fig.2 Integrated OAM emitter arrays

Full size|PPT slide

To demonstrate the potential of photonic integration, we also fabricated OAM emitter arrays consisting of four identical emitters (R = 7.5 mm, q = 72) coupled to the same access waveguide (Fig. 2(a)). Simultaneous emission of identical vortices has been verified as shown in Figs. 2(b) and 2(c).

4 Tunable silicon integrated OAM devices

In a similar manner to the evolution of wavelength division-multiplexed systems, the future advancement of OAM-based telecommunication systems will require OAM routing flexibility and reconfigurability with components that can perform the fast switching of OAM data channels [ 41, 42].

Fig.3 Tunable integrated OAM devices

Full size|PPT slide

Based on our previous work, we reported a fast tunable integrated OAM device with electrically addressable thermo-optical phase shifters that is capable of actively on–off keying OAM modes at record rates of 10 ms and OAM switching rates of 20 ms [ 43]. A resistive heater device was designed to create a thermal change of refractive index in the waveguide core, and hence tune the WGM mode and emitted OAM mode. Figure 3 shows a micrograph and a scanning electron microscopy image of the tunable vortex beam emitter. The metal resistive line was defined concentrically around the ring resonator, with a slightly larger radius than the silicon ring. This radial offset, while still allowing significant thermo-optic tuning, ensured that the emitted beam did not overlap the absorbing metallic structure.

Fig.4 Dynamic characterization of the tunable integrated OAM devices. Measured optical signal for (a) on–off keying and (b) switching between l = -1 and l = + 1

Full size|PPT slide

Fig.5 (a) Device for manipulating the OAM superposition states, consisting of an OAM emitter, a 3 dB coupler and a phase shifter; (b) far field images of various generated OAM superposition states

Full size|PPT slide

To demonstrate dynamic control over the OAM values of the emitted beam, 10 kHz square-wave driving signals were applied to the device. First, a 10 mW peak power square wave was applied to the device to shift the ring out of resonance, and so, effectively, turn off the vortex beam emission. The measured trace in Fig. 4(a) shows on–off keying of the emission signal corresponding to the driving signal, with a measured rise-time of 10 ms and a fall-time of 1.4 ms. In addition to on–off keying of the vortex emission, the real-time switching between the OAM modes can also be achieved. Figure 4(b) shows the time trace of switching between l = -1 and l = +1. In this case, the switching time was measured as 20 ms.

We further demonstrated OAM integrated functional circuits composed of an OAM emitter and a 3 dB coupler, which can be used to generate and manipulate OAM superposition states [ 44]. The input signals simultaneously excite the clockwise and anticlockwise WGMs in the micro-ring resonator, which generate two OAM states with opposite chirality. The relative phase between two OAM states can be actively modulated on-chip by applying a voltage on a phase shifter integrated with the waveguide (Fig. 5).

5 Mode purity of the OAM beam generated from silicon integrated emitters

The mode purity of the OAM beams is a very important parameter for practical use. In high index contrast silicon photonic devices, unexpected interactions due to WGM modes and angular gratings or the backscattering of the silicon waveguide may give rise to OAM mixing and deteriorated purity. Therefore we need to characterize the purity of the OAM beams generated from the integrated devices.

An experimental setup based on a spatial light modulator (SLM), as shown in Fig. 6, was used to study the mode purity of the emitted OAM beam from the device. After going through the polarization filter, the RHCP or LHCP component of the beam was converted to a linearly polarized beam with OAM value of l−1 or l + 1. The OAM order of the linearly polarized beam was then analyzed by changing the order of the holographic pattern on the SLM, and the mode purity can be studied by measuring the on-axis intensity of each beam determined from the images on an IR CCD array. As an example, Fig. 7 shows the measurement results for l = -10. The two dominant peaks coincide with the desired OAM order, l = -10, and the OAM order from backscattering mode, l = 10. The mode purity of the OAM beam with l = -10 is measured at 94%. Note that the mode purity of the beam with l = 0 is 100%, due to the indistinguishablity between the forward and backward-coupled beams at l = 0.

Fig.6 Schematic diagram of the OAM mode purity measurement

Full size|PPT slide

Fig.7 Mode purity measurement results for l = -10. H is the order of the pattern on the SLM (number of phase dislocations on the Y-branch hologram)

Full size|PPT slide

6 Future work

Although the fundamental principles have now been established, some issues in performance and functionalities still need to be addressed before the integrated OAM devices can find practical applications in the systems. For example, high emission efficiency is important for any practical application, and this can be obtained by engineering the coupling between the waveguide and the micro-ring. We believe that efficiencies of well above 50% can be achieved. OAM purity also needs to be improved by reducing the backscattering of light in the ring and optimizing the design of the angular grating. High refractive index modulation will have to be realized in order to achieve OAM modulation with wide output OAM range.

Fig.8 Sketch of integrated OAM (de-)multiplexer OAM. (a) Ω-shaped device; (b) concentric ring devices

Full size|PPT slide

More importantly, integrated OAM (de-)multiplexer, capable of combining or separating OAM states, will have to be developed, because multiplexing and de-multiplexing is of vital importance for the application of OAM-based optical communications. There are two ways of achieving this function (Fig. 8). One is W-shaped device, and the other is vertically coupled concentric micro-ring OAM devices. The W-shaped device is easy to be fabricated, but the performance is compromised in reduced mode purity as the emitted near field is not a complete ring. The concentric micro-ring OAM device, with optically accessed by vertically coupled waveguides that lie underneath the resonator structures, is more attractive, but the fabrication process of this structure involves wafer bonding technique.

It is generally believed that these problems can be solved using photonic integration technologies based on optimized designs of specific devices, suitable photonic materials and nano-fabrication techniques.

7 Conclusions

We have demonstrated a highly novel, scalable photonic integration approach, which shows great promises to address many problems in the applications of OAM light. The central innovation is the newly discovered principle of coupling WGM of micro-resonator to a free space propagating OAM mode by using angular grating embedded within the micro-resonator. The micro-resonator emitter enables OAM to be efficiently generated and detected on PICs. The preliminary theoretical and experimental results have confirmed the operation principle of the integrated OAM device. Based on the innovative principle and device, we further demonstrate a fast tunable integrated OAM device with electrically addressable thermo-optical phase shifters that is capable of actively on–off keying OAM modes at record rates of 10 ms, and OAM integrated functional circuits composed of an OAM emitter and a 3 dB coupler, which can be used to generate and manipulate OAM superposition states. These devices demonstrate a very simple approach for on-chip dynamical manipulation of OAM states and show the potential of this technology for the development of sophisticated OAM functions on scalable and compact integrated circuits, which can find several applications in future telecommunication systems.

References

[1] Agrawal A, Sun Y, Barnwell J, Raskar R. Vision-guided robot system for picking objects by casting shadows. International Journal of Robotics Research, 2010, 29(2-3): 155–173

[2] Taryudi,Wang M S. 3D object pose estimation using stereo vision for object manipulation system. In: Proceedings of International Conference on Applied System Innovation. Sapporo: IEEE, 2017, 1532–1535

[3] Dinham M, Fang G, Zou J J. Experiments on automatic seam detection for a MIG welding robot. In: Proceedings of International Conference on Artificial Intelligence and Computational Intelligence. Berlin: Springer, 2011, 390–397

[4] Ryberg A, Ericsson M, Christiansson A K, Eriksson K, Nilsson J, Larsson M. Stereo vision for path correction in off-line programmed robot welding. In: Proceedings of IEEE International Conference on Industrial Technology. Vina del Mar: IEEE, 2010, 1700–1705

[5] Dinham M, Fang G. Autonomous weld seam identification and localisation using eye-in-hand stereo vision for robotic arc welding. Robotics and Computer-integrated Manufacturing, 2013, 29(5): 288–301

[6] Yun W H, Lee J, Lee J H, Kim J. Object recognition and pose estimation for modular manipulation system: overview and initial results. In: Proceedings of International Conference on Ubiquitous Robots and Ambient Intelligence. Jeju: IEEE, 2017, 198–201

[7] Dong L, Yu X, Li L, Hoe J K E. HOG based multi-stage object detection and pose recognition for service robot. In: Proceedings of 11th International Conference on Control Automation Robotics & Vision, Singapore: IEEE, 2010, 2495–2500

[8] Nam H, Han B. Learning multi-domain convolutional neural networks for visual tracking. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016, 4293–4302

[9] Held D, Thrun S, Savarese S. Learning to track at 100 FPS with deep regression networks. In: Proceedings of European Conference on Computer Vision. Berlin: Springer, 2016, 749–765

[10] Henriques J F, Rui C, Martins P, Batista J. High-speed tracking with kernelized correlation filters. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(3): 583–596

[11] Stricker D. Tracking with reference images:a real-time and markerless tracking solution for out-door augmented reality applications. In: Proceedings of Conference on Virtual Reality, Archeology, and Cultural Heritage. Glyfada: DBLP, 2001, 77–82

[12] Wu Y, Lim J, Yang M H. Object tracking benchmark. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1834–1848

[13] Yang B, Yan J, Lei Z, Li S. Aggregate channel features for multi-view face detection. In: Proceedings of IEEE International Joint Conference on Biometrics. Clearwater: IEEE, 2014, 1–8

[14] Hirschmuller H. Accurate and efficient stereo processing by semi-global matching and mutual information. In: Proceedings of IEEE Computer Society Conference on Computer Vision & Pattern Recognition. San Diego: IEEE, 2005, 807–814

[15] Zhang Z. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(11): 1330–1334

[16] Li J, Zhao H, Jiang T, Zhou X. Development of a 3D high-precise positioning system based on a planar target and two CCD cameras. In: Proceedings of International Conference on Intelligent Robotics and Applications. Berlin: Springer, 2008, 475–484

[17] Magnusson M, Andreasson H, Nüchter A, Lilienthal A J. Appearance-based loop detection from 3D laser data using the normal distributions transform. Journal of Field Robotics, 2010, 26(11–12):892–914

[18] Li Y, Liu G. Learning a scale-and-rotation correlation filter for robust visual tracking. In: Proceedings of IEEE International Conference on Image Processing. Phoenix: IEEE, 2016, 454–458

[19] Zhang K, Zhang L, Yang M H. Fast Compressive Tracking. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2014, 36(10):2002-2015.

[20] Lu H, Jia X, Yang M H. Visual tracking via adaptive structural local sparse appearance model. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Providence: IEEE, 2012, 1822–1829

[21] Dinh T B, Vo N, Medioni G. Context tracker: exploring supporters and distracters in unconstrained environments. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Colorado Springs: IEEE, 2011, 1177–1184

RIGHTS & PERMISSIONS

2019 Higher Education Press and Springer-Verlag GmbH Germany, part of Springer Nature