Introduction
The measurement of the 3D profile is very important in both 3D modeling and 3D display. It attracted lots of attention in the last decades [1,2]. Generally, to measure the whole 3D profile of an object, one needs to take multi-measurement which is partially overlapped, and then fuses the data in different perspective to get the whole point cloud data of the object under the same coordinate. The fusion of the data measured is a quite difficult task in the reconstruction of 3D profile. There are some of the common fusion methods. One is using accurately controlled devices. This method is based on the rotation and translation matrix offered by equipment which needs rather accurate equipment and is complicated to operate [3].
Calculating transformation matrix based on mark points pasted on overlap area is a widely used method for image fusion too [6,7]. Though this method needs to paste points manually, it is easy to operate and has certain accuracy [8]. Therefore, on the base of mark points, this paper presents a new method for 3D multi-perspective image fusion.
The data source for image fusion in this paper is 3D spatial information of a head model by using structured light projection. Structured light projection is a method to get depth information of object from demodulated stripe on the surface of the object by using projecting grating images from a digital projector [9]. For this method, phase information (depth information) is acquired by phase unwrapping algorithm, and plane position information is acquired by system calibration, thus we can acquire the original 3D spatial information of certain point on the surface of the object [10]. In the experiment, it needs to take an original texture image without grating stripe for every perspective. Image fusion is mostly based on these texture images and the pasted mark points in their overlapped area.
Mark points design
If we want to find the corresponding relationship between images of different perspectives, we need to create a series of points with unique characteristic which can easily be distinguished from the background image.
Mark points fall into two categories: coded and non-coded. Coded mark points have a wide use in the process of image fusion, because it can provide the matching condition for multi-view connection by using the coding points along with the position information. For this reason, we will use coded mark points for image fusion in our experiment.
It’s common that circular point is selected as mark point for the fusion process of a 3D measuring system, because the detection method of circular points can greatly reduce the error probability of point detection [11]. Therefore, most coded mark points are composed by a circular point in the center and some coded points around the central point [12], see Fig. 1. The size of the central point is generally bigger than that of the coded points. Central point provides the position information, while the surrounding points provide the coding information by the presence or absence of the point in certain position. It’s decided by the arrangement of the coded points that how many different kinds of mark points it can present.
When choosing the type of mark points for 3D head image fusion, we have some considerations as following:
1) Limited by the effective area, if the size of coded points is too big, it will reduce the number of mark points; however, if the size is too small, it will be hard to detect after projection and distortion;
2) Because of the projective distortion, the absolute position of the coded points in the captured image will change, which reduce the accuracy of matching.
3) The more circular points on the model, the more time it will take to calculate for compute, while immediate processing is needed.
4) The color of head mostly consists of red, black and skin color. The relationship of RGB channels of these colors will stay still in general no matter how the luminance changes.
5) Based on the considerations above, color-coded mark points are designed for this research. Parameters of the mark points are in Table 1.
Tab.1 Parameters of color-coded mark points |
| point | values of RGB | example | point | values of RGB | example |
|---|---|---|---|---|---|
| type 1 | 0 255 0 | <InlineMediaObject OutputMedium="Online"><ImageObject FileRef="images\hcm0000470534.jpg" ScaleToFitWidth="" ScaleToFit="1"/></InlineMediaObject><InlineMediaObject OutputMedium="All"><ImageObject FileRef="images\hcm0000470533.tif" Format="TIFF" OrigFileRef="foe-12028-gz-tb1fig1.tif" ScaleToFitHeight="27.0pt" ScaleToFitWidth="" ScaleToFit="1"/></InlineMediaObject> | type 3 | 0 255 255 | <InlineMediaObject OutputMedium="Online"><ImageObject FileRef="images\hcm0000470532.jpg" ScaleToFitWidth="" ScaleToFit="1"/></InlineMediaObject><InlineMediaObject OutputMedium="All"><ImageObject FileRef="images\hcm0000470531.tif" Format="TIFF" OrigFileRef="foe-12028-gz-tb1fig3.tif" ScaleToFitHeight="24.255pt" ScaleToFitWidth="" ScaleToFit="1"/></InlineMediaObject> |
| type 2 | 0 0 255 | <InlineMediaObject OutputMedium="Online"><ImageObject FileRef="images\hcm0000470570.jpg" ScaleToFitWidth="" ScaleToFit="1"/></InlineMediaObject><InlineMediaObject OutputMedium="All"><ImageObject FileRef="images\hcm0000470569.tif" Format="TIFF" OrigFileRef="foe-12028-gz-tb1fig1.tif" ScaleToFitHeight="24.255pt" ScaleToFitWidth="" ScaleToFit="1"/></InlineMediaObject> | type 4 | 255 0 255 | <InlineMediaObject OutputMedium="Online"><ImageObject FileRef="images\hcm0000470568.jpg" ScaleToFitWidth="" ScaleToFit="1"/></InlineMediaObject><InlineMediaObject OutputMedium="All"><ImageObject FileRef="images\hcm0000470567.tif" Format="TIFF" OrigFileRef="foe-12028-gz-tb1fig4.tif" ScaleToFitHeight="26.5125pt" ScaleToFitWidth="" ScaleToFit="1"/></InlineMediaObject> |
| Actual size: 5.0 mm | |||||
In addition, there are some principles when pasting the points on the model as follows: first, there should be certain distance between two mark points and should be non-collinear. Second, mark points should be pasted on relatively smooth areas like forehead or cheek and points at a similar position had better in different colors. Moreover, the number of points in overlap regions of adjacent perspectives should be no fewer than three.
Mark point detection and image fusion
Image preprocessing
Before image fusion, we need to do preprocessing of the images that is detecting the mark points from edge images, which belongs in object recognition domain.
In luminance image, drastic change of luminosity function happens on the borders of an object. Border of an object in the image is a property of single pixel, which can be calculated by image function in the neighborhood of the pixel. It is a vector with both amplitude and direction. The amplitude to border is that of gradient, and the direction of border is the gradient direction rotated by -90°. Gradient direction is the maximizing growth direction of the function.
After experiments and comparison, Canny edge detection algorithm is treated as the optimal method for image fusion based on color-coded mark points in our research [13]. The calculations of the algorithm are as follows:
1) Get the image to make convolution integral with Gaussian function with a scale factor S.
2) Evaluate the normal direction of the border through every pixel of the image.
3) Find the edge by using non-maximum suppression.
4) Calculate the edge intensity.
5) Do lag threshold processing to the edge image to eliminate false response.
Quasi-ellipse mark points detection
After getting edge image, the next step is to distinguish the mark points from background. Mark point is single-color-circular-point with white grounding. It will be distorted to quasi-ellipse based on the properties of perspective projection transformation, which will have certain interference to detection. There are a lot of algorithms for ellipse detection. Examples are like Hough transform, method based on inertia, method with geometrical conditions and some improved algorithm based on the methods above [14-18]. However, those methods have some common disadvantages, like iteration of invalid data point, time-consuming, large storage and calculations, and what’s more, no anti-interference quality for quasi-arc.
Therefore, we design a new detection method for quasi-ellipse mark points on 3D model. It is more targeted as well as efficient.
According to the position of suspected points and the included angle between the tangent plane and image plane, the projected size and deformation of mark points can be forecasted. Thus, we can set threshold to test and verify whether a suspected point is mark point or not. Figure 2 shows the flow chart for distinguishing mark points.
When checking whether a certain area is a mark point, there are some criteria as follows:
1) Use both the edge image and texture image to estimate whether a pixel is of an edge according to the gray value of the pixel, since in the edge image value 0 means black which is the color of edges and 255 means white of the background.
2) Check whether the distance of boundaries is in the threshold of the length of mark points.
3) Check whether the edge is closed.
4) Use arc criterion to see to distinguish ellipse and rectangle.
In addition, by using the color of mark points, the same points can be matched between two perspective images directly and rapidly. This requires that when pasting mark points, in an area of certain size there are no mark point in same color. Thus, according to geometric constraint and color constraint, we make it possible to detect and match mark points on 3D model rapidly and automatically.
According to the multi-criteria like edge, internal color, size and closed figure, it can detect mark points and find its center accurately, and it can avoid the interference of non-mark-point image.
Point cloud data fusion
Because of transform error, the transformed data cannot totally be consistent with the data in original image in most time. To eliminate the gap in overlap area in transformed image, we need to use equalization method in the following process.
Experimental result and evaluation
By using turn table to take photos from different perspectives, we can get the original images for panoramic reconstruction. Figure 3 shows two original texture images from adjacent perspectives. And Fig. 4 shows the edge images of the two texture images by using Canny edge detection algorithm.
The size of the images is 500×550 pixels each and the detected corresponding mark points are described in Table 2.
Tab.2 Detected corresponding mark points in each perspective |
| perspective 1 | perspective 2 |
|---|---|
| (565,263,233) | (454,262,151) |
| (566,286,238) | (454,285,163) |
| (617,363,204) | (502,362,192) |
| (643,356,169) | (539,355,208) |
| (575,477,191) | (473,473,140) |
| (550,481,207) | (419,383,42) |
After obtaining matched point pairs, we choose this SVD method to calculate transformation matrix, including rotation matrix BoldItalic and translation matrix BoldItalic, because of the self-monitoring property of SVD. The final reconstruction head model based on the two images above is shown in Fig. 5.
To evaluate the transforming quality of the point cloud data of perspective 2, the transforming errors are listed in Table 3.
Tab.3 Comparison of mark point coordinates before and after transformation |
| perspective 1 | perspective 2 before transformation | perspective 2 after transformation | transforming error/% |
|---|---|---|---|
| (565,263,233) | (454,262,151) | (558.7992,267.7609,234.4135) | 1.1167 |
| (566,286,238) | (454,285,163) | (571.3145,289.9258,236.5492) | 1.1067 |
| (617,363,204) | (502,362,192) | (612.2411,371.3796,209.5037) | 2.1167 |
| (643,356,169) | (539,355,208) | (646.4414,348.9348,176.1394) | 2.1167 |
| (575,477,191) | (473,473,140) | (576.1719,490.5505,196.7524) | 1.8500 |
| (550,481,207) | (419,383,42) | (542.5946,484.5465,208.3486) | 0.7867 |
Conclusions
In this paper, we have presented a way to splice images from different perspectives based on the point cloud data obtained by structured light projection. According to the characteristics of 3D head model, we proposed a new method by pasting color-coded mark points. The experiment shows that this method has some advantages as follows: it occupies less space than other position coded mark points so that it has more flexibility; and it avoids the reduction of accuracy caused by deformation of coded points; meanwhile it improves computational efficiency. Moreover, colors of the mark point make it possible for corresponding points to match rapidly and automatically. For the covered area by mark points, it can be complemented by interpolation method after image fusion.