Design of computer vision assisted machine learning based controller for the Stewart platform to track spatial objects

Abstract

The present work aims to develop an object tracking controller for the Stewart platform using a computer vision-assisted machine learning-based approach. This research is divided into two modules. The first module focuses on the design of a motion controller for the Physik Instrumente (PI)-based Stewart platform. In contrast, the second module deals with the development of a machine-learning-based spatial object tracking algorithm by collecting information from the Zed 2 stereo vision system. Presently, simple feed-forward neural networks (NN) are used to predict the orientation of the top table of the platform. While training, the x, y, and z coordinates of the three-dimensional (3D) object, extracted from images, are used as the input to the NN. In contrast, the orientation information of the platform (that is, rotation about the x, y, and z-axes) is considered as the output from the network. The orientation information obtained from the network is fed to the inverse kinematics-based motion controller (module 1) to move the platform while tracking the object. After training, the optimised NN is used to track the continuously moving 3D object. The experimental results show that the developed NN-based controller has successfully tracked the moving spatial object with reasonably good accuracy.

Graphical abstract

Keywords

Stewart platform / feed-forward neural networks / motion controller / inverse kinematics / stereo vision

1 Introduction

The tracking of spatial objects plays a major role in various applications, such as the sun tracking system in spacecraft and in identifying satellite debris and celestial bodies. So far, gimbals are used in various tracking applications, which provide only two degrees of freedom while performing the tracking. The Stewart platform is a 6-degree of freedom (DOF) spatial parallel robot manipulator [1–4], which can move in all three (longitudinal, and both transverse) directions. Apart from that, it can also perform roll, pitch and yaw motions. The robot can achieve a desired location {x-axis, y-axis, z-axis, roll, pitch, yaw} by the centroid of its upper plate, while the lower plate is fixed after creating a variation in the lengths of all six prismatic links simultaneously at an instance. All the prismatic links are connected between a fixed plate and the other upper moving plate. In 1965, Stewart [2] published a paper related to this parallel manipulator for flight simulation application. Now, it is commonly used in various important applications and is known as the Stewart platform (and also by different names).

The purpose of this work is to develop certain control strategies for the hexapod robot to track a continuously moving spatial object by utilizing a stereo vision system. Many researchers have worked on issues related to the kinematics [3–5], Jacobian, singularities [6], dynamics [7–11], force analysis [12,13], and path planning [14] of the Stewart platform [15,16]. However, none of the researchers tried to track a spatially moving object with the help of the Stewart parallel manipulator. Basically, the Stewart platform (hexapod robot) is a precise positioning device extensively used in the machining of micro-electro mechanical system (MEMS) components. It is also used for various other applications, such as flight simulation, satellite dish positioning, medical fields, and the simulation of earthquakes. It is important to note that the scientific interest in comets and asteroids lies largely with the idea that there is the debris in the Solar System that may hit the Earth. Institutions like National Aeronautics and Space Administration (NASA) [17] are making efforts to establish a risk management system [18] by tracking and following such asteroids and comets. The real-time tracking and following of asteroids and comets can be achieved by mounting a telescope on a platform, so that the joints of the platform can be controlled so that the telescope can track the object. Further, the solar panels [19,20] that are used to produce power in spacecrafts and in space stations always need to face the sun for maximum power efficiency. At present, gimbals [21,22] are used to rotate them so that they face the Sun all the time. The task of the Sun tracking and following for maximum output has also been carried out using a 2-DOF parallel manipulator [23], apart from a gimbal. But 2-DOF and 3-DOF parallel robots have limited DOF and stability compared with those of the Stewart platform. In addition to this, the tracking and following of the Sun for maximum power efficiency can also be accomplished with the help of the Stewart platform by mounting a Pyranometer on the table of the parallel manipulator with an improved payload. Liu et al. [24] used a Kernelized Correlation Filter-based tracker algorithm to detect and track an arbitrary object in real time using a gimbal mounted on a drone. Yilmaz et al. [25] presented a state of art object tracking system to classify objects, identify different trends; The group discussed various methods on the basis of objects and their motions. Mandava et al. developed [26] and reviewed [27] an adaptive proportional-integral-derivative (PID) controller for two-legged robots. Samaniego et al. [28] explored a deep neural network based solver for partial differential equations (PDEs). Whereas, Anitescu et al. [29] also solved PDEs using artificial neural networks and an adaptive collocation strategy. The key point needs to be noted here that authors are putting more emphasis on using a parallel robot, i.e., the Stewart platform, over serial robots (such as gimbals) because such a system has more stiffness, stability and huge payload capacity due to their six strong prismatic links. Though the Stewart platform has less work-volume ((three-dimensional) 3D space swept by a robot) than the serial (type of a manipulator) one, it may still be better for tracking-and-following applications because of its 6-DOF. The gimbals [30,31] have 2 to 3-DOF for roll, pitch and yaw which is sufficient to track a 3D object but has very less stiffness and cannot sustain a heavy payload such as risk management system for celestial bodies, telescope, etc. The Stewart platform has extra DOFs and may provide a better reach (capability to track the spatial object), along with good stability and stiffness with a huge payload.

In the present research, an attempt is made to develop a computer vision assisted neural networks (NN)-based machine learning algorithm and to track spatial objects using a SPS-type Stewart platform. The novelty of the present study lies in training the NN model in order to get the exact required orientation information of the Stewart platform’s end-effector with respect to the 3D information of the tracking object from the stereo vision system. Further, the trained NN is used to provide orientation information related to the control of the platform. The orientation information obtained by the NN is fed to the motion controller to drive the platform in such a way that it will track the 3D object. The two modules, namely the NN and the motion controller, are integrated in a seamless manner to achieve fast tracking of the spatial object. The authors have divided the work into different sections: problem formulation, Section 2; Module 1, motion controller of the Stewart platform, Section 3; Module 2, machine learning-based tracking, Section 4; results and discussions, Section 5; conclusions, Section 6.

2 Problem formulation

Object tracking refers to the process of locating and following an object over time in a sequence of images or frames. The object may be stationary or moving, and the goal is to identify its location and track its movement accurately. Object following by a robot, on the other hand, refers to the process of controlling the robot’s movement so that it can follow (move along with) and track a specific object, usually a moving object. The robot needs to track the object’s movement and adjust its own movement accordingly to keep the object in its field of view and within a certain range. Object tracking is focused on the accurate location and movement of the object, whereas object following is focused on the robot’s movement to follow the object. Following a spatial 3D object’s motion trajectory by using the Stewart platform implies that making a movement by the upper plate of the Stewart platform in motion while it faces continuously to the said 3D object with a mutual alignment during the nearby motion by the object in 3D-space. Moreover, during this motion, the centroid of the object and the centroid of the upper plate should be aligned. In addition to this, multiple situations are possible in this real motion.

Fig.1 shows the schematic diagram of the SPS-type Stewart platform. It is one form of the Stewart platform, having SPS joint combinations. The types of the Stewart platform are classified based on its joint configuration and its type of actuation. The SPS-type of the Stewart platform is one of the types of a Stewart platform in which both ends of six independent prismatic actuators are connected with the upper and lower plate by the two passive spherical joints. The whole connection is forming a SPS type of joint configuration. In Fig.1, joints F_i and M_i represent passive spherical types of joints, while L_i represents active prismatic types of joints that can be actively controlled by the system. The prismatic joint has a sliding pair between two rigid links in which these two rigid links are sliding with each other actively (powered). Due to this sliding the effective length of the prismatic actuators is increasing. The ground base plate is fixed, while the upper plate is movable. A fixed reference frame zero, denoted by {0}, is fixed at the bottom plate’s centroid, while a moving reference frame one, denoted by {1}, is attached to the upper plate’s centroid of the platform.

2.1 Forward problem

The forward problem of object tracking involves determining the point on a hemisphere where the upper platform is facing and when the configuration (task-space position and orientation) of the Stewart Platform is given. So, the input of the forward problem consists of the generalized coordinates {P_x,P_y,P_{\mathit{\text{z}}},\alpha ,\beta ,\gamma} of the Stewart platform, and the output is the coordinate of the targeted point on the surface of the hemisphere toward which the upper plate of the Stewart platform is facing. An illustration of the forward problem is given in Fig.2(a).

Let’s assume H represents the coordinates of the point in the hemisphere surface toward which the Stewart platform is facing. P represents the coordinates of the coordinate-frame attached to the upper plate with respect to the ground frame. O is the center of the hemisphere as well as the origin of the base coordinate frame. Since we are interested in the points O, P, and H, a simplified schematic is present in Fig.2(b).

The forward problem, determining coordinates of H when generalized coordinates of end-effector, can be stated as,

(1)q={[P_x,P_y,P_{\mathit{\text{z}}},\alpha ,\beta ,\gamma ]}^{\mathrm{T}},

is known. Since point {H=\left[H_x,H_y,H_{\mathit{\text{z}}}\right]}^{\mathrm{T}} lies on a hemisphere, we can write as

(2){H_x}^2+{H_y}^2+{H_{\mathit{\text{z}}}}^2=R^2,

where R is the radius of the hemisphere.

(3)H={\left[H_x,H_y,H_{\mathit{\text{z}}}\right]}^{\mathrm{T}}.

The line PH is along the normal of the upper plate of the Stewart Platform. But the z direction of the fixed coordinate frame U (a frame assigned at the center of the upper plate) is also along the normal of the upper plate. So, the direction cosines of the upper platform (l, m, n) can be easily obtained using the transformation matrix (T_{\mathrm{B}\mathrm{a}\mathrm{s}\mathrm{e}}^{\mathrm{T}\mathrm{o}\mathrm{p}}) as follows:

(4)\left[\begin{array}{c}\begin{array}{c}l\\ m\\ n\end{array}\\ 1\end{array}\right]=T_{\mathrm{B}\mathrm{a}\mathrm{s}\mathrm{e}}^{\mathrm{T}\mathrm{o}\mathrm{p}}\left(p_x,p_y,p_{\mathit{\text{z}}},\alpha ,\beta ,\gamma \right)\times \left[\begin{array}{c}\begin{array}{c}0\\ 0\\ 0\end{array}\\ 1\end{array}\right].

This means the direction cosine of the line PH can be known once the orientation of the upper platform is known. For line PH, we now have starting point coordinates and direction cosine. This is sufficient to derive the equation of the line. Since point H lies in this line, we can write,

(5)\frac{H_x-P_x}{l}=\frac{H_y-P_y}{m}=\frac{H_{\mathit{\text{z}}}-P_{\mathit{\text{z}}}}{n}=k,

(6)\frac{H_x-P_x}{\mathrm{c}\mathrm{o}\mathrm{s}\alpha }=\frac{H_y-P_y}{\mathrm{c}\mathrm{o}\mathrm{s}\beta }=\frac{H_{\mathit{\text{z}}}-P_{\mathit{\text{z}}}}{\mathrm{c}\mathrm{o}\mathrm{s}\gamma }=k.

From Eq. (6) we can get the {\left[H_x,H_y,H_{\mathit{\text{z}}}\right]}^{\mathrm{T}}as given below Eq. (7).

(7)H=\left[\begin{array}{c}{H}_x\\ H_y\\ H_{\mathit{\text{z}}}\end{array}\right]=k\left[\begin{array}{c}\mathrm{c}\mathrm{o}\mathrm{s}\alpha \\ \mathrm{c}\mathrm{o}\mathrm{s}\beta \\ \mathrm{c}\mathrm{o}\mathrm{s}\gamma \end{array}\right]+\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{P}_x\\ P_y\\ P_{\mathit{\text{z}}}\end{array}\right].

From Eqs. (2) and (7) we can reach on a quadratic equation, i.e., Eq (8).

(8)a{k}^2+bk+c=0,

where a, b and c are given as below in Eq. (9).

a=\left({\mathrm{c}\mathrm{o}\mathrm{s}\alpha }^2+{\mathrm{c}\mathrm{o}\mathrm{s}\beta }^2+{\mathrm{c}\mathrm{o}\mathrm{s}\gamma }^2\right),

(9)b=2\left(P_x\mathrm{c}\mathrm{o}\mathrm{s}\alpha +P_y\mathrm{c}\mathrm{o}\mathrm{s}\beta +P_{\mathit{\text{z}}}\mathrm{c}\mathrm{o}\mathrm{s}\gamma \right),

c=\left({P_x}^2+{P_y}^2+{P_{\mathit{\text{z}}}}^2-R^2\right).

After solving the quadratic equation Eq. (8) the solution of k is given as below in Eq. (10)

(10)k=\frac{-b\pm \sqrt{b^2-4ac}}{2a}.

The quadratic equation gives two different solutions. One solution relates to the lower half of the hemisphere, and it may be neglected. After getting the k value from Eq. (10), it can be substituted in Eq. (7) in order to find the H. This is the solution to the forward problem, which also depends on the radius of the hemisphere R. It is worthwhile to note that the aforementioned problem results in a single solution and can be performed analytically. The above method can be repeated for other geometries, such as planar, and conical.

2.2 Inverse problem

The inverse problem of object tracking is just the reverse of the forward problem. Here coordinates of H=[H_x,H_y,H_{\mathit{\text{z}}}{]}^{\mathrm{T}} are provided, and the generalized coordinates of the platform \left\{P_x,P_y,P_{\mathit{\text{z}}},\alpha ,\beta ,\gamma \right\} are to be determined. But the problem is inherently very complex to analyze as we have three equations and six unknowns. The problem is similar to the inverse kinematics problem of serial manipulators and is quite complex to solve analytically; there exist multiple solutions. It can be observed that the Stewart platform can be made to face a single point using multiple configurations.

The objective is to determine one solution out of all the solutions which are feasible. The feasibility in this case is expressed as the shortest distance between the object and the platform. Which is reflected in the data set prepared for the training of NN. When tracking a trajectory, the point H is a function of time. In the present research, the authors use machine learning to solve the said problem. The schematic diagram showing various modules developed to track spatial objects using machine learning-based controllers is shown in Fig.3.

The stereo camera mounted on the top of the Stewart platform captures the details of the object. The details include the exact location of the object in three-dimensional space. The information related to location H, along with other features such as the edge and bounding box of the object, is extracted using image processing at first. Afterwards, the location H is fed to the trained NN. The output of the NN, i.e., \left\{\alpha ,\beta ,\gamma \right\}, is fed to the motion controller of the Stewart platform. The motion controller consists of inverse kinematics, Jacobean, inverse dynamics, a trajectory planner, and a system motion controller. In addition to the motion controller, the individual actuator drive control unit is also there to control joint parameters such as position, velocity, acceleration, and torque. To control joint parameters, the feedback from the sensors and encoders present in the actuators is utilized in the controller. The PID control strategy for the said controllers is used. The detailed discussion regarding both modules, i.e., platform motion controller and NN based object tracking algorithm, is given below.

3 Module 1: Motion controller

Module-1 regards the motion controller of the Stewart platform. In which, the desired 6-DOF motion of the Stewart platform’s upper plate is controlled using the PID controller. The inverse dynamics controller is used as a master controller for the motion. The Stewart platform has six prismatic joints to adjust the desired motion of the platform’s end-effector (upper plate). Therefore, there is also a requirement of the joint motion controller for individual joint drives (prismatic actuator) present in the Stewart platform. The Stewart platform has six Brushless Direct Current (BLDC), high-precision prismatic actuators. The PID controller is used to control the individual prismatic joint motion to achieve its desired position, velocity, acceleration, and torque by using joint-space trajectory planning.

The overall motion is achieved using the inverse kinematics of the robot. Here, the inverse kinematics involves nothing but finding out the joint space variables when the desired Cartesian space variables are given to reach. The relation between these task spaces to joint spaces is given as below.

(11)L_i=\sqrt{{\left(X_{{\mathrm{T}}_i}-X_{{\mathrm{B}}_i}\right)}^2+{\left(Y_{{\mathrm{T}}_i}-Y_{{\mathrm{B}}_i}\right)}^2+{Z_{{\mathrm{T}}_i}}^2}.

The prismatic link lengths L_i, i.e., (L₁, L₂, L₃, L₄, L₅, and L₆), can be determined by the above Eq. (11). The said expression gives the explicit manifestation of the lengths of the legs L_i in terms of the Stewarts platform’s vertices coordinates, i.e., \{X_{{\mathrm{T}}_i},Y_{{\mathrm{T}}_i},Z_{{\mathrm{T}}_i}\}, of the upper plate with respect to fixed frame. The fixed frame was considered at the centroid of the fixed plate, with its vertical axis pointing perpendicularly to this plate. One end of all prismatic actuators is connected to this fixed plate by spherical joint. These coordinates \left\{X_{{\mathrm{T}}_i},Y_{{\mathrm{T}}_i},Z_{{\mathrm{T}}_i}\right\}, further can be expressed with the given position and orientation of the center of the upper platform {p_x,p_y,p_{\mathit{\text{z}}},\alpha ,\beta ,\gamma}, in terms of \left\{u_{{\mathrm{T}}_i},v_{{\mathrm{T}}_i},w_{{\mathrm{T}}_i}\right\}, as below.

(12)\left[\begin{array}{c}{X}_{\mathrm{T}i}\\ Y_{\mathrm{T}i}\\ \begin{array}{c}{Z}_{\mathrm{T}i}\\ 1\end{array}\end{array}\right]=T_{\mathrm{B}\mathrm{a}\mathrm{s}\mathrm{e}}^{\mathrm{T}\mathrm{o}\mathrm{p}}\left(p_x,p_y,p_{\mathit{\text{z}}},\alpha ,\beta ,\gamma \right)\times \left[\begin{array}{c}{u}_{\mathrm{T}i}\\ v_{\mathrm{T}i}\\ \begin{array}{c}{w}_{\mathrm{T}i}\\ 1\end{array}\end{array}\right].

The \{u_{{\mathrm{T}}_i},v_{{\mathrm{T}}_i},w_{{\mathrm{T}}_i}\} are the vertices of the upper plate of the Stewart platform with respect to the movable top frame, which is located at the centroid of the upper plate. Here, its vertical axis points perpendicularly to the plate. During motion, this frame moves with respect to the bottom plate, as discussed above. The {p_x,p_y,p_{\mathit{\text{z}}},\alpha ,\beta ,\gamma} in the above equation are the location parameters of the upper plate centroid in the task space with respect to the above discussed fixed frame to the base plate. In this expression, {p_x,p_y,p_{\mathit{\text{z}}}} describes the position of the upper plate, while {\alpha ,\beta ,\gamma} describes its orientation.

The overall transformation equation {(T}_{\mathrm{B}\mathrm{a}\mathrm{s}\mathrm{e}}^{\mathrm{T}\mathrm{o}\mathrm{p}}) for mapping the top plate’s movable frame with respect to the bottom plate’s fixed base frame is given below.

(13)T_{\mathrm{B}\mathrm{a}\mathrm{s}\mathrm{e}}^{\mathrm{T}\mathrm{o}\mathrm{p}}=\left[\begin{array}{ccc}\mathrm{c}\mathrm{o}\mathrm{s}\beta \mathrm{c}\mathrm{o}\mathrm{s}\gamma +\mathrm{s}\mathrm{i}\mathrm{n}\alpha \mathrm{s}\mathrm{i}\mathrm{n}\beta \mathrm{s}\mathrm{i}\mathrm{n}\gamma & -\mathrm{c}\mathrm{o}\mathrm{s}\beta \mathrm{s}\mathrm{i}\mathrm{n}\gamma +\mathrm{s}\mathrm{i}\mathrm{n}\alpha \mathrm{s}\mathrm{i}\mathrm{n}\beta \mathrm{c}\mathrm{o}\mathrm{s}\gamma & \mathrm{c}\mathrm{o}\mathrm{s}\alpha \mathrm{s}\mathrm{i}\mathrm{n}\beta \\ \mathrm{c}\mathrm{o}\mathrm{s}\alpha \mathrm{s}\mathrm{i}\mathrm{n}\gamma & \mathrm{c}\mathrm{o}\mathrm{s}\alpha \mathrm{c}\mathrm{o}\mathrm{s}\gamma & -\mathrm{s}\mathrm{i}\mathrm{n}\alpha \\ -\mathrm{s}\mathrm{i}\mathrm{n}\beta \mathrm{c}\mathrm{o}\mathrm{s}\gamma +\mathrm{s}\mathrm{i}\mathrm{n}\alpha \mathrm{c}\mathrm{o}\mathrm{s}\beta \mathrm{s}\mathrm{i}\mathrm{n}\gamma & \mathrm{s}\mathrm{i}\mathrm{n}\beta \mathrm{s}\mathrm{i}\mathrm{n}\gamma +\mathrm{s}\mathrm{i}\mathrm{n}\alpha \mathrm{c}\mathrm{o}\mathrm{s}\beta \mathrm{c}\mathrm{o}\mathrm{s}\gamma & \mathrm{c}\mathrm{o}\mathrm{s}\alpha \mathrm{c}\mathrm{o}\mathrm{s}\beta \\ 0& 0& 0\end{array}\begin{array}{c}{p}_x\\ p_y\\ \begin{array}{c}{p}_{\mathit{\text{z}}}\\ 1\end{array}\end{array}\right].

Therefore, this combined overall problem is known as the inverse kinematics problem and is represented by the below function:

(14)L_i=f(p_x,p_y,p_{\mathit{\text{z}}},\alpha ,\beta ,\gamma ).

While the reverse relation of the previous inverse problem is a forward kinematic problem and is given by X_{\mathrm{p}-\mathrm{o}}=g\left(L_i\right), where, X_{\mathrm{p}-\mathrm{o}} is a location that has to be determined while link lengths are given at a particular instant in time. The X_{\mathrm{p}-\mathrm{o}} can be represented by {p_x,p_y,p_{\mathit{\text{z}}},\alpha ,\beta ,\gamma} as given by the Eq. (15).

(15)X_{\mathrm{p}-\mathrm{o}}={\{p_x,p_y,p_{\mathit{\text{z}}},\alpha ,\beta ,\gamma \}}^{\mathrm{T}}.

Therefore, the inverse problem can also be given by the following equation:

(16)L_i=f\left(X_{\mathrm{p}-\mathrm{o}}\right).

This inverse and the forward problems are multi-input and multi-output problems due to multiple degrees of freedom. In this work, the robot motion is considered to only utilize the inverse kinematics of the platform. The velocity of the robot can be analyzed during motion after differentiating the length equation with respect to time. The velocity of the link \dot{L_i} can be represented by the below (Eq. (17)), while the velocities of the centroid of the upper plate {\dot{X}}_{\mathrm{p}-\mathrm{o}} are represented as (Eq. (18)).

After differentiating the length equation (inverse kinematics) concerning time, the robot’s motion velocity may be examined when the velocity of the centroid of the top plate {\dot{X}}_{\mathrm{p}-\mathrm{o}} is given as Eq. (18), the link velocity \dot{L_i} can be computed using the equation below (Eq. (21)).

(17)\dot{L}=\dot{L_i}={\{\dot{L_1},\dot{L_2},\dot{L_3},\dot{L_4},\dot{L_5},\dot{L_6}\}}^{\mathrm{T}},

and

(18)\dot{X_{\mathrm{p}-\mathrm{o}}}={\{\dot{p_x},\dot{p_y},\dot{p_{\mathit{\text{z}}}},\dot{\alpha },\dot{\beta },\dot{\gamma }\}}^{\mathrm{T}}.

The following gives the link velocity expressions.

(19)\dot{L}=J_1\dot{X_{\mathrm{T}}},

(20){\dot{X}}_{\mathrm{T}}=J_2\dot{X_{\mathrm{p}-\mathrm{o}}}.

where X_{\mathrm{T}}\equiv [X_{{\mathrm{T}}_1},Y_{{\mathrm{T}}_1},Z_{{\mathrm{T}}_1},X_{{\mathrm{T}}_2},Y_{{\mathrm{T}}_2},Z_{{\mathrm{T}}_2},X_{{\mathrm{T}}_3},Y_{{\mathrm{T}}_3},Z_{{\mathrm{T}}_3},X_{{\mathrm{T}}_4},Y_{{\mathrm{T}}_4},Z_{{\mathrm{T}}_4},X_{{\mathrm{T}}_5},Y_{{\mathrm{T}}_5},Z_{{\mathrm{T}}_5},X_{{\mathrm{T}}_5},Y_{{\mathrm{T}}_5},Z_{{\mathrm{T}}_5}], J_1 is a (6 \times 9) derivative matrix and J_2 is a (9 × 6) derivative matrix. Below is presented the final velocity expression.

(21)\dot{L}=J{\dot{X}}_{\mathrm{p}-\mathrm{o}}.

When the J_1 and J_2 matrices are multiplied, the resulting matrix, J, is a (6 × 6) Jacobian derivative matrix. A parallel manipulator is said to be in a singular configuration when either J_1 or J_2 or both are singular since two Jacobian matrices are present. There are three distinct categories of singularities. The first has to do with the inverse kinematic singularities determinant \left(J_1\right)=0, the second one with the direct kinematic singularities determinant \left(J_2\right)=0, and the third one with both the inverse kinematic and direct kinematic singularities combined determinant \left(J_1\right)=0 and determinant \left(J_2\right)=0.

The dynamics of the entire system are calculated after the kinematics have been derived, which is useful for figuring out how much kinetic energy is needed to move the platform. The overall meaning is to find out the torque required to move the platform to a certain state. The following mathematical relationship is used to calculate the kinetic and potential energy for the platform’s legs as well. The joint variable in this instance, q, is linear prismatic for the Stewart platform.

K=K\left(q,\dot{q}\right)=\frac{1}{2}{\dot{q}}^{\mathrm{T}}M\left(q\right)\dot{q},

where q\in R^n and q\in R^{n\times n}.

(22)P=P\left(q\right),

where P\left(q\right)\in R^n.

The kinetic energy of the moving platform is given as:

(23)K_{\mathrm{u}\mathrm{p}}=K_{\mathrm{u}\mathrm{p}\left(\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}\right)}+K_{\mathrm{u}\mathrm{p}\left(\mathrm{r}\mathrm{o}\mathrm{t}\right)},

where K_{\mathrm{u}\mathrm{p}\left(\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}\right)} is the translational kinetic energy of a moving plate and K_{\mathrm{u}\mathrm{p}\left(\mathrm{r}\mathrm{o}\mathrm{t}\right)} is the rotational kinetic energy of a moving plate. The potential energy of the moving platform is given as:

(24)P_{\mathrm{u}\mathrm{p}}=m_{\mathrm{u}\mathrm{p}}g{P}_{\mathrm{z}}{X}_{\mathrm{p}}.

The mass of the upper moving plate of the Stewart platform is given by m_{\mathrm{u}\mathrm{p}}.

The kinetic and potential energy of the manipulator’s legs must be calculated using a specific methodology, as described in Ref. [7]. Thus, the overall dynamic equations are derived using the Lagrange formulation as given below in Eq. (25) where q and \dot{q} are joint variables in terms of position and velocity and \tau is joint torque or force provided to the actuators.

(25)\frac{\mathrm{d}}{\mathrm{d}t}\frac{\delta K\left(q,\dot{q}\right)}{\delta \dot{q}}-\frac{\delta K\left(q,\dot{q}\right)}{\delta q}+\frac{\delta P\left(q\right)}{\delta q}=\tau .

The motion of the platform with respect to object movement should synchronize with the trajectory tracking. The task space parameters have to feed into the motion controller of the platform. The schematic diagram showing various steps used by the motion controller of the Stewart platform is given in Fig.4. This module takes the input information from module 2 (that is, the neural network module). This information is required to perform the inverse kinematics of the platform along with the inverse dynamics in order to control the motion. Joint space trajectory planning is used to generate the trajectory points. In this trajectory planning strategy, the inverse kinematics is calculated at the start and beginning of the motion. The intermediate points are calculated using a 5th order polynomial equation. The intermediate points are once again fed to the controller in order to get the desired motion. The error in the motion is minimized using the PID controller. The joint motion is controlled using the joint motion PID controller. The feedback on the position of the platform is grabbed from the sensors and encoders present in the robot stage.

4 Module 2: Machine learning based tracking

The object tracking using the Stewart platform is an inverse tracking and following problem, as discussed above, and has multiple solutions. This problem is quite complex to formulate analytically due to the possibility of multi-alignment existing between the upper plate and the 3D object. Apart from the analytical solution, the problem can also be solved using numerical approaches and machine learning approaches. In this context, the authors have proposed a machine-learning-based solution. First, this problem is addressed with the machine learning training on the tracking and object-following setup, and then a validation of the neural net is also carried out. For the neural network training, a training data set is also required. Therefore, this data set is prepared using the random motion of the Stewart platform with the object’s pre-defined desired tracking location. An object is fixed to the top plate of the platform with the help of a long, rigid rod. A yellow-colored object is detected using the edge detection method using Gaussian blur and masking techniques in this work. A stereo camera, which shows the position of the object in the 3D world, is fixed on the top of the platform (Fig.5). The stereo camera has its own inertial measurement unit built in, which helps to grab the exact location of the upper plate in 3D. As the object is fixed above the upper plate during this random motion while collecting the data; at a particular instant, the Stewart platform’s location along with the object’s location has also been determined. At each instant of time, the location of the 3D object’s center point has been calculated with respect to the ground plate’s fixed frame after mapping from the same object location with respect to the top moving plate.

Fig.5(b) shows the 3D object, which is fixed with a strut perpendicular to the upper plate. In this situation, the tracking occurs vertically upwards, with the camera facing upward. Tracking in a horizontal direction may also be considered, where the camera faces horizontally. But this situation is not considered in this work. The mapping of the 3D object with respect to the ground frame and other transformations is shown in Fig.5(a). The location of a 3D object with respect to the fixed frame O can be expressed as below (Eq. (22)).

(26)\left[\begin{array}{c}\begin{array}{c}{H}_x\\ H_y\\ H_{\mathit{\text{z}}}\end{array}\\ 1\end{array}\right]=T_{\mathrm{B}\mathrm{a}\mathrm{s}\mathrm{e}}^{\mathrm{T}\mathrm{o}\mathrm{p}}\left(p_x,p_y,p_{\mathit{\text{z}}},\alpha ,\beta ,\gamma \right)\times \left[\begin{array}{c}\begin{array}{c}{H}_x^{\prime }\\ H_y^{\prime }\\ H_{\mathit{\text{z}}}^{\prime }\end{array}\\ 1\end{array}\right].

The architecture of the machine-learning based tracking algorithm is shown in Fig.6. The images grabbed using the stereo vision system are fed to the object feature extraction module. The depth estimator of the stereo vision system helps in estimating the distance of the object. The overall location of the 3D object with respect to the ground frame is fed to the NN for forward calculation of the NN. The output of the trained NN can be further supplied to the Stewart platform’s motion controller (as given in module-1) in order to move the upper platform (Fig.6). While solving the problem, the 3D location of the object grabbed from the camera is given as input to the neural net for training, and the orientation of the Stewart platform {\alpha ,\beta ,\gamma} is estimated from the network. During data collection, the network is trained with the help only of information related to the orientation (\alpha ,\beta ,\gamma) of the top plate. The other three position parameters (p_x,p_y,p_{\mathit{\text{z}}}) are not considered during training and are set to be zero, as they present more complexity in training the neural network due to the nonlinearity of the problem. Therefore, the overall neural network structure with three input neurons in the input layer and three output neurons in the output layer is considered, along with three hidden layers.

The training data required to train the network is generated from the random motion of the platform with a fixed spatial object. For the given position of the object, the orientation of the platform is calculated utilizing the concept of inverse kinematics. Around ten thousand data sets are used to train the network.

Once the image data from the stereo camera is collected, the whole data set is pre-processed, and the noise present in the data set is eliminated. Various steps used for data pre-processing are mentioned below.

1) Data cleansing. Data cleansing involves identifying and removing any duplicate entries, inconsistent data, or obvious outliers from the data set. These data points can introduce noise and inaccuracies into the model.

2) Outlier detection and handling. The outlier detection techniques are used to find and address extreme values or anomalies in the data. To mitigate their impact, the outliers are either eliminated from the data or applied the proper data transformation technique, depending on the type of data.

3) Smoothing and filtering. These techniques assist in lowering high-frequency noise that can be harmful to the performance of the model.

4) Feature selection. The features are carefully examined within the data set, and feature selection is conducted to retain only the most relevant and informative attributes. Irrelevant or redundant features can introduce noise into the model.

5) Normalization and standardisation. Data scaling methods, such as normalization and standardisation, are applied to ensure that all variables have a consistent scale.

6) Imputation. Missing data can introduce noise, and a data imputation technique has been implemented to fill in the missing values and maintain the data set’s integrity.

Moreover, some additional data have been separated and used for validation of the neural network. From the available data, 70% data are utilized for training the network, and the remaining 30% data are used for the validation of the network.

The network consists of three types of layers: named as the input layer, hidden layers and output layer. The input layer is fed with the 3D object location with respect to the ground frame. The output layer is associated with the location of the platform with respect to the ground frame when the 3D object is fixed in line with the platform and facing the center point of the camera. The input layer has a total of three neurons in it with parameters, from H={\left[H_x,H_y,H_{\mathit{\text{z}}}\right]}^{\mathrm{T}}vector, H_x,H_y,\mathrm{a}\mathrm{n}\mathrm{d}{H}_{\mathit{\text{z}}}. The output layer may consist of a total of three parameters from the location vector X_{\mathrm{p}-\mathrm{o}}={\{\alpha ,\beta ,\gamma \}}^{\mathrm{T}}. While tracking, the three-dimensional position of the top plate of the platform is fixed, and the platform is allowed to orient itself based on the information received from the neural network module. Apart from this, each layer is also associated with the bias value. Further, the neurons present in the input and output layers of the network are provided with a linear transfer function, whereas the neurons present in the hidden layer are provided with a rectified linear unit (RELU) transfer function.

The fitness function f used for the neural network is given by the Eq. (27), in which d is the number of training scenarios, n is the number of outputs and {\alpha }_{k\mathrm{t}} is target output and {\alpha }_{k\mathrm{p}} is predicted output.

(27)f=\frac{1}{d}\sum _{i=1}^d\left[\sqrt{\frac{1}{2}\sum _{k=1}^n{\left({\alpha }_{k\mathrm{t}}-{\alpha }_{k\mathrm{p}}\right)}^2}\right].

The trained and validated NN model is used for tracking the object. The NN model provides the orientation of the Stewart platform at a particular instant in time, corresponding to the 3D location of the object. After that, the Stewart platform’s motion controller is used to move the platform to the desired location. The motion is performed through 5th order joint space trajectory planning after consideration without via points. The start point of the trajectory is taken as the current position of the Stewart platform, while the goal point of the trajectory is taken as the target point, which comes out of the Artificial Neural Network (ANN) model. The whole trajectory is divided into small time intervals. The inverse kinematics has been calculated for the start and end points. The middle joint points have been generated from the joint space trajectory generation algorithm.

4.1 Stereo vision

The Zed stereo camera, as shown in Fig.7, is used to grab the 3-dimensional world map related to the object. The ZED is a camera with dual lenses. It captures high-definition 3D video with a wide field of view and outputs two synchronised left and right video streams in side-by-side format on USB 3.0. The ZED stereo camera has a frame rate of 100 FPS and a larger field of view (110° (H) × 70° (V)). It is also capable of capturing depth information from 0.3 to 20 m with a resolution of 1280 × 720 at 60 FPS.

5 Results and discussion

Fig.8 shows the setup used to collect the data while tracking the object using NN. The stereo camera is mounted on the Stewart platform’s upper plate along with the fixed 3D object. In the present case, considered in this work, the stereo camera faces vertically upwards and is located above the Stewart platform’s upper plate. The training of the neural network is also carried out along with the validation. The stereo camera has the capability to capture the distance to the object. The varying distance data of the 3D object has also been collected using a stereo camera. The object’s motion perpendicular to the camera frame is achieved after adjusting the sliding strut which connected above the Stewart platform.

A Dell OptiPlex with a 64-bit operating system with x64-based processor, having the 11th Gen Intel (R) Core (TM) i9-11900 @ 2.50 GHz by 2.50 GHz with 32 GB RAM is used to collect the data. An external GPU and NVIDIA GeForce GT 730 graphics card is used for processing the 3D depth map and point cloud data of the sensed environment. Further, PYTHON 3.6 and CUDA 10.2 software are used to collect and process the data.

Once the training of the network is done, the trained neural network is utilized for tracking the real-time object in a 3D environment. To track an object with the help of the Stewart parallel manipulator, an experimental setup, as shown in Fig.9, is prepared. The Stewart platform is placed at the bottom of a prepared frame. The stereo camera is fixed on the top of the upper plate of the platform. The camera is allowed to face vertically upward in order to sense the 3D object. At the top of the frame, a 2-DOF spatial manipulator configuration is fixed, and a ball is fixed at the wrist end of the manipulator to allow for a three dimensional motion for the ball (that is, the object). This arrangement is able to provide random motion to the object in three-dimensional space above the Stewart platform. The motion of the object in three dimensional space is fully independent of the motion of the Stewart platform.

A systematic study is conducted to optimize the architecture of the feed-forward back-propagation type neural network with the camera facing vertically up. The network is designed to have three hidden layers. In the network, the first, second and third hidden layers are seen to have 24, 18, and 11 hidden neurons, respectively. The back propagation (BP) algorithm is used to train the architecture of the network. The learning rate for the network is chosen to be equal to 0.45.

The convergence plot shows the training of a neural network using the BP algorithm, as shown in Fig.10.

Fig.10 shows the semi-log plot representing the variation between Root Mean Squared Error (RMSE) verses epochs. In the above figure, the RMSE value is seen to reach zero. Once the training for the network is over, the trained neural network is used to provide real-time information related to the orientation of the platform after taking the information from the stereo camera.

5.1 Real time object tracking

Before conducting the actual experiments, certain experiments are performed to study the characteristics and performance of the vision and tracking systems with the help of pixel error plots in the x- and y-directions. The pixel errors corresponding to the position of the object and target in the image frame along x- and y-directions are shown in Fig.11–Fig.13, respectively. These errors emphasize the performance of real-time object tracking. For perfect tracking, the pixel error needs to be zero in the x- and y-directions. Fig.11 shows the pixel error in x- and y-directions, which seems to settle down with time. In the figure, both curves are showing an oscillating trend at the beginning and then settled after a certain duration. This may be due to the levitating and shaking of the platform in an uncontrolled manner at the start of tracking.

Sometimes the controller is able to settle down the instability very quickly, but in most cases, it is found to take a long time. This instability may lead to tracking loss of the object and also lead to mechanical damage to the Stewart platform. After settling the zero-pixel error, the platform is seen to follow the trajectory accurately.

The Stewart platform used in the study has a specified range of motion for the six parameters {P_x,P_y,P_{\mathit{\text{z}}},\alpha ,\beta ,\gamma}, which are limited to {±50, ±50, ±25, ±15, ±15, ±30 mm}. Fig.12 shows the information related to tracking when the object is seen to be out of the range of the platform’s motion limits. Three spikes on the graph show that the object has been out of the range of the platform’s work space three times and has shown an absolute pixel error between 60 and 80 pixels while tracking the object. During tracking, the pixel error corresponding to the x-axis has also shown some deviation from the zero line. This might have happened to reduce the pixel error corresponding to the y-axis by the robot motion controller and causing to disturbance in x-axis movement.

Moreover, the ambient light of the environment of the vision sensor also plays an important role while detecting and tracking the object. In the present experiment, normal lighting conditions are considered for the tracking. In the present study, edge detection and masking techniques are used to detect the object. Fig.13 shows the tracking pixel error corresponding to the insufficient lighting condition. From the figure, one can observe that the pixel error is not settling to zero, showing that the variation is instable. The platform seems to vibrate while reaching the target. It is also observed that during the process of tracking, the tracker loses the object for a certain time span, as shown by the spikes in the curve in the figure.

This problem can be eliminated by providing brighter lighting in the environment. It is also important to note that the velocity of the moving object also has a certain relationship with the maximum speed of the platform (Fig.14).

It is important to note that the range of the angular velocity of the Stewart platform is seen to be in the range of 0 to 200 m·rad/s. The maximum tracking capability of the platform also depends on its maximum angular velocity. If the object velocity is higher and the specified velocity of the platform is not sufficient to match for the object tracking, tracking of the object is not possible. To chase the object, the specified velocity of the platform needs to be higher than the velocity of the moving object. Fig.14 shows the phenomenon of tracking when the angular velocity of the Stewart platform’s upper plate is not sufficient enough to match the velocity of the object. Both the pixel errors of the x- and y-axis are seen to be oscillating and exhibiting instability; the platform vibrates in an uncontrolled manner. In such situations, the platform never settles on the desired zero-pixel error. Therefore, later on, while conducting experiments, the velocity of the moving object is limited to be within the limit of capability of the platform to turn.

Once the vision and tracking systems are validated, the tracking experiments are conducted. The images corresponding to the real time tracking of the object in the camera coordinates, with the help of the trained network, are shown in Fig.15. From the figures, it can be observed that the Stewart parallel manipulator is seen to follow the object, which is moving in 3D space. Fig.15 also shows the enlarged view during real-time tracking where camera faces vertically upward on the platform; in this case the error is reduced after detection of the object until it reaches its target position.

Only three frames are shown for illustration purposes. The intermediate frames at equal intervals of time are also shown in Fig.16.

While conducting the experiments, the information related to the position, velocity, and acceleration of the prismatic joints of the platform was also collected with respect to time and is shown in Fig.17, Fig.18, and 19, respectively.

The peaks in velocity and acceleration represent sudden changes in the positions of the object. When there is a change in the position of the object, there is a sudden spike in the velocity and acceleration plots.

The prismatic joint velocities and accelerations change during changes of position of the object. Once the object is located and tracked, the velocities and accelerations once again are close to zero. This phenomenon shows that the developed motion planner has tracked the object very closely in 3D space.

6 Conclusions

In conclusion, this work aimed to develop a controller for the Stewart platform using a computer vision-assisted machine learning approach. Two modules, namely a motion controller for the Physik Instrumente-based Stewart platform and a machine learning-based spatial object tracking algorithm, were developed to tackle this problem. A feed-forward NN with a backpropagation algorithm is designed for predicting the 3D location of the object. The orientation information obtained from the network was then fed to the inverse kinematics-based motion controller to move the platform while tracking the object. The experimental results demonstrated that the developed NN-based controller successfully tracked the continuously moving 3D object with reasonably good accuracy. By combining computer vision, machine learning, and motion control techniques, this research contributes to the advancement of autonomous tracking systems for Stewart platforms. Further studies could focus on enhancing the robustness and adaptability of the controller, considering various environmental conditions and object dynamics. Additionally, investigating the applicability of the developed approach to other similar systems or expanding its capabilities to handle complex scenarios would be valuable future research in this field.

References

[1] MittalR KNagrathI J. Robotics and Control. New York, NY: Tata McGraw-Hill, 2003

[2] Stewart D. A platform with six degrees of freedom. Proceedings of the Institution of Mechanical Engineers, 1965, 180(1): 371–386

[3] Liu K, Fitzgerald J M, Lewis F L, Member S. Kinematic analysis of a Stewart platform manipulator. IEEE Transactions on Industrial Electronics, 1993, 40(2): 282–293

[4] Zhou W, Chen W, Liu H. A new forward kinematic algorithm for a general Stewart platform. Mechanism and Machine Theory, 2015, 87: 177–190

[5] Chauhan D K S, Vundavilli P R. Forward kinematics of the Stewart parallel manipulator using machine learning. International Journal of Computational Methods, 2022, 2142009(8): 1–22

[6] Liu K, Lewis F, Lebret G, Taylor D. The singularities and dynamics of a Stewart platform manipulator. Journal of Intelligent & Robotic Systems, 1993, 8(3): 287–308

[7] Lebret G, Liu K, Lewis F L. Dynamic analysis and control of a Stewart platform manipulator. Journal of Robotic Systems, 1993, 10(5): 629–655

[8] Mostashiri N, Akbarzadeh A, Dhupia J. A comprehensive inverse dynamics problem of a Stewart platform by means of Lagrangian formulation. In: Proceedings of Dynamic Systems and Control Conference. New York, NY: American Society of Mechanical Engineers, 2017, 58271: V001T30A003

[9] Meng Q, Zhang T, He J F, Song J Y, Han J W. Dynamic modeling of a 6-degree-of-freedom Stewart platform driven by a permanent magnet synchronous motor. Frontiers of Information Technology & Electronic Engineering, 2010, 11: 751–761

[10] Tien T X, Cirkl D. Solution of inverse dynamics of Stewart−gough platform using substructure technique. Journal of Theoretical and Applied Mechanics, 2021, 59: 329–342

[11] Dasgupta B, Mruthyunjaya T S. A Newton−Euler formulation for the inverse dynamics of the Stewart platform manipulator. Mechanism and Machine Theory, 1998, 33(8): 1135–1152

[12] Liu M J, Li C X, Li C N. Dynamics analysis of the Gough−Stewart platform manipulator. IEEE Transactions on Robotics and Automation, 2000, 16(1): 94–98

[13] Jishnu A K, Chauhan D K S, Vundavilli P R. Design of neural network-based adaptive inverse dynamics controller for motion control of stewart platform. International Journal of Computational Methods, 2022, 19(8): 2142010

[14] Gosselin C M, Hadj-Messaoud A. Automatic planning of smooth trajectories for pick-and-place operations. In: Proceedings of International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. New York, NY: American Society of Mechanical Engineers, 1991, 97768: 531–536

[15] Ömürlü V E, Yildiz I. A Stewart platform as a FBW flight control unit. Journal of Electrical Engineering, 2011, 62(4): 213–219

[16] Koszewnik A, Troc K, Słowik M. PID controllers design applied to positioning of ball on the Stewart platform. Acta Mechanicaet Automatica, 2014, 8(4): 214–218

[17] JosephI. Space debris: Understanding the risks to NASA spacecraft. 2023. Available at the website of NASA

[18] Wikipedia. Space debris. 2023. Available at the website of Wikipedia

[19] Ashith Shyam R B, Acharya M, Ghosal A. A heliostat based on a three degree-of-freedom parallel manipulator. Solar Energy, 2017, 157: 672–686

[20] Ashith Shyam R B, Ghosal A. Path planning of a 3-UPU wrist manipulator for sun tracking in central receiver tower systems. Mechanism and Machine Theory, 2018, 119: 130–141

[21] Tharamuttam J K, Ng A K. Design and development of an automatic solar tracker. Energy Procedia, 2017, 143: 629–634

[22] Baouche F Z, Abderezzak B, Ladmi A, Arbaoui K, Suciu G, Mihaltan T C, Raboaca M S, Hudișteanu S V, Țurcanu F E. Design and simulation of a solar tracking system for PV. Applied Sciences, 2022, 12(19): 9682

[23] Cammarata A. Optimized design of a large-workspace 2-DOF parallel robot for solar tracking systems. Mechanism and Machine Theory, 2015, 83: 175–186

[24] Liu X, Yang Y, Ma C, Li J, Zhang S. Real-time visual tracking of moving targets using a low-cost Unmanned Aerial Vehicle with a 3-axis stabilized gimbal system. Applied Sciences, 2020, 10(15): 5064

[25] Yilmaz A, Javed O, Shah M. Object tracking: A survey. ACM Computing Surveys, 2006, 38(4): 13

[26] Mandava R K, Vundavilli P R. Near optimal PID controllers for the biped robot while walking on uneven terrains. International Journal of Automation and Computing, 2018, 15(6): 689–706

[27] Khan M S, Mandava R K. A review on gait generation of the biped robot on various terrains. Robotica, 2023, 41(6): 1888–1930

[28] Samaniego E, Anitescu C, Goswami S, Nguyen-Thanh V M, Guo H, Hamdia K, Zhuang X, Rabczuk T. An energy approach to the solution of partial differential equations in computational mechanics via machine learning: Concepts, implementation and applications. Computer Methods in Applied Mechanics and Engineering, 2020, 362: 112790

[29] Anitescu C, Atroshchenko E, Alajlan N, Rabczuk T. Artificial neural network methods for the solution of second order boundary value problems. Computers, Materials & Continua, 2019, 59(1): 345–359

[30] Regner D J, Salazar J D, Buschinelli P V, Machado M, Oliveira D, Santos J M, Marinho C A, Pinto T C. Object tracking control using a gimbal mechanism. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2021, 43: 189–196

[31] Cong Danh N. The stability of a two-axis gimbal system for the camera. The Scientific World Journal, 2021, 2021: 2–9

RIGHTS & PERMISSIONS

Higher Education Press