Occlusion removal technique for improved recognition of partially occluded 3D objects in computational integral imaging

In this paper, we propose an occlusion removal technique for improved recognition of 3D objects that are partially occluded in computational integral imaging (CII). In the reconstruction process of a 3D object which is partially occluded by other objects, occlusion degrades the resolution of reconstructed 3D images and thus this affects negatively the recognition of a 3D object in CII. To overcome this problem, we introduce a method to eliminate occluding objects in elemental image array (EIA) and the proposed method is applied to 3D object recognition by use of CII. To our best knowledge, this is the first time to remove occlusion in CII. In our method, we apply the elemental image to sub-image (ES) transform to EIA obtained by a pickup process and those sub-images are employed for occlusion removal. After the transformation, we correlate those sub-images with a reference sub-image to locate occluding objects and then we eliminate the objects. The inverse ES transform provides a modified EIA. Actually, the modified EIA is considered to be an EIA without the object that occludes the object to be reconstructed. This can provide a substantial gain in terms of the image quality of 3D objects and in terms of recognition performance. To verify the usefulness of the proposed technique, some experimental results are carried out and the results are presented.     

Three-dimensional recognition of occluded objects by using computational integral imaging

We have proposed a method to recognize partially occluded three-dimensional (3D) objects by using 3D volumetric reconstruction integral imaging (II). An II system captures multiple perspectives of occluded objects by using a microlens array. The reconstruction of the occluded 3D scene and target recognition are done digitally to reduce the effects of the occlusion. To verify system performance, we have implemented an optimum filter for object recognition. Both two-dimensional (2D) images and 3D II volumetric reconstructed images are considered. The correlation results of occluded 3D images for volumetric reconstruction show substantial improvements compared with those for conventional 2D imaging of occluded images.     

Depth-controlled reconstruction of 3D integral image using synthesized intermediate sub-images

In this paper, we propose a method that controls the depth of the three-dimensional (3D) object existing over the depth-of-focus in integral imaging. The depth control method is performed only in a computer by synthesizing the intermediate sub-images between original sub-images obtained by transforming the captured elemental images. In the reconstruction process, we can obtain reconstructed 3D images with the better image quality within depth-of-focus than that reconstructed over the depth-of-focus. To demonstrate the feasibility of our method, optical and computational experiments are carried out and its results are presented.     

Three-dimensional imaging and visualization of partially occluded objects using axially distributed stereo image sensing

In this Letter, we propose a multiperspective three-dimensional (3D) imaging system using axially distributed stereo image sensing. In this proposed method, the stereo camera is translated along its optical axis and multiple axial elemental image pairs for a 3D scene are collected. The captured elemental images are reconstructed in 3D using a computational reconstruction algorithm based on ray back-projection. The proposed method is applied to partially occluded object visualization. Optical experiments are performed to verify the approach.     

Performance-enhanced recognition of a far and partially occluded 3-D object by use of direct pixel-mapping in computational curving-effective integral imaging

In this paper, we propose a novel approach to enhance the recognition performance of a far and partially occluded three-dimensional (3-D) target in computational curving-effective integral imaging by using the direct pixel-mapping (DPM) method. With this scheme, the elemental image array (EIA) originally picked up from a far and partially occluded 3-D target can be converted into a new EIA just like the one virtually picked up from a target located close to the lenslet array. Due to this characteristic of DPM, resolution and quality of the reconstructed target image can be highly enhanced, which results in a significant improvement of recognition performance of a far 3-D object. In addition, the computational time required for reconstruction of a far 3-D target could be also reduced because the distance between the lenslet array and image plane is virtually shortened in the new EIA transformed by DPM. Experimental results reveal that image quality of the reconstructed target image and object recognition performance of the proposed system have been improved by 1.75 dB and 4.56% on the average in PSNR (peak-to-peak signal-to-noise ratio) and NCC (normalized correlation coefficient), respectively, compared to the conventional system.     

Regeneration of elemental images in integral imaging for occluded objects using a plenoptic camera

In this Letter, we propose an elemental image regeneration method of three-dimensional(3D) integral imaging for occluded objects using a plenoptic camera. In conventional occlusion removal techniques, the information of the occlusion layers may be lost. Thus, elemental images have cracked parts, so the visual quality of the reconstructed 3D image is degraded. However, these cracked parts can be interpolated from adjacent elemental images. Therefore, in this Letter, we try to improve the visual quality of reconstructed 3D images by interpolating and regenerating virtual elemental images with adjacent elemental images after removing the occlusion layers. To prove our proposed method, we carry out optical experiments and calculate performance metrics such as the mean square error(MSE) and the peak signal-to-noise ratio(PSNR).     

基于立体像素匹配的图像重构技术研究

为了解决目前全景成像技术中分辨率低的问题,提出了一种新的基于3D场景立体像素光线映射的全景图像计算机重构技术.在全景成像技术中,3D场景的每个立体像素点经全景成像系统的编码系统分别映射在一定区域的多个体元素图像的不同像素点上.在计算机重构全景图像时,根据逆光学路径原理,提出了从立体像素映射到的体元素图像区域中提取对应立体像素的多个2D像素点来重构全景图像,使重构的全景图像最大分辨率可达到传统成像方法图像分辨率的N倍（N为映射区域面积）.提出的立体像素的匹配技术大大提高了重构的计算机全景图像分辨率.     

Computational integral imaging reconstruction scheme of far 3D objects by additional use of an imaging lens

In this paper, we propose a novel performance-enhanced computational integral imaging reconstruction (CIIR) scheme by additional use of an imaging lens. In the proposed scheme, elemental images can be obtained by using a simultaneous pickup scheme of far three-dimensional (3D) objects from the lenslet array in both real and virtual image fields. And additional imaging lens produces an image shift effect of 3D objects located far away from the lenslet array and improve the visual quality of reconstructed images in CIIR by overcoming limitation of pickup range in integral imaging. To show the usefulness of the proposed system, some experiments are carried out for real 3D objects and its results are presented.     

Three-dimensional visualization of objects in scattering medium using integral imaging and spectral analysis

This paper presents a three-dimensional visualization method of 3D objects in a scattering medium. The proposed method employs integral imaging and spectral analysis to improve the visual quality of 3D images. The images observed from 3D objects in the scattering medium such as turbid water suffer from image degradation due to scattering. The main reason is that the observed image signal is very weak compared with the scattering signal. Common image enhancement techniques including histogram equalization and contrast enhancement works improperly to overcome the problem. Thus, integral imaging that enables to integrate the weak signals from multiple images was discussed to improve image quality. In this paper, we apply spectral analysis to an integral imaging system such as the computational integral imaging reconstruction. Also, we introduce a signal model with a visibility parameter to analyze the scattering signal. The proposed method based on spectral analysis efficiently estimates the original signal and it is applied to elemental images. The visibility-enhanced elemental images are then used to reconstruct 3D images using a computational integral imaging reconstruction algorithm. To evaluate the proposed method, we perform the optical experiments for 3D objects in turbid water. The experimental results indicate that the proposed method outperforms the existing methods.     

A novel three-dimensional digital watermarking scheme basing on integral imaging

A novel integral imaging-based three-dimensional (3D) digital watermarking scheme is presented. In the proposed method, an elemental image array (EIA) obtained by recording the rays coming from a 3D object through a pinhole array in the integral imaging system is employed as a new 3D watermark. The EIA is composed of a number of small elemental images having their own perspectives of a 3D object, and from this recorded EIA various depth-dependent 3D object images can be reconstructed by using the computational integral imaging reconstruction (CIIR) technique. This 3D property of the EIA watermark can make a robust reconstruction of the watermark image available even though there are some data losses in the embedded watermark by attacks. To show the robustness of the proposed scheme against attacks, some experiments are carried out and the results are discussed.     

3D image reconstruction with a controllable overlapping number of elemental images in computational integral imaging

In this Letter, we propose a three-dimensional(3D) image reconstruction method with a controllable overlapping number of elemental images in computational integral imaging. The proposed method can control the overlapping number of pixels coming from the elemental images by using the subpixel distance based on ray optics between a 3D object and an image sensor. The use of a controllable overlapping number enables us to provide an improved 3D image visualization by controlling the inter-pixel interference within the reconstructed pixels.To find the optimal overlapping number, we simulate the pickup and reconstruction processes and utilize the numerical reconstruction results using a peak signal-to-noise ratio(PSNR) metric. To demonstrate the feasibility of our work in optical experiments, we carry out the preliminary experiments and present the results.     

Efficient compression of motion-compensated sub-images with Karhunen-Loeve transform in three-dimensional integral imaging

An approach to highly enhance the compression efficiency of the integral images by applying the Karhunen-Loeve transform (KLT) algorithm to the motion-compensated sub-images is proposed. The sub-images transformed from the elemental images picked-up from the three-dimensional (3D) object might represent the different perspectives of the object. Thus, the similarity among the sub-images gets better than that among the elemental images, so that an improvement of compression efficiency of the sub-images could be obtained. However, motion vectors occurred among the sub-images might result in an additional increase of image data to be compressed. Accordingly, in this paper, motion vectors have been estimated and compensated in all sub-image in advance. Then the KLT algorithm was applied to these motion-compensated sub-images for compression. It is shown from some experimental results that compression efficiency of the proposed method has been improved up to 24.44%, 40.62%, respectively, on the average compared to that of the conventional KLT compression method and that of the JPEG.     

Enhanced compression of integral images by combined use of residual images and MPEG-4 algorithm in three-dimensional integral imaging

In this paper, we proposed a novel approach to enhance the compression rate of integral images by combined use of the residual images generated from the sub-images and the MPEG-4 algorithm. In the proposed method, elemental images picked up from a three-dimensional object are transformed into sub-images, and these sub-images are sequentially rearranged with a spiral scanning topology and the first sub-image is assigned as the reference image. Then, by sequentially computing the differences between the reference image and other consecutive sub-images, a sequence of residual images is generated. Here, the residual images together with the reference image are modeled as the consecutive video frames just like a conventional moving picture. Finally, these residual images are compressed with the MPEG-4 algorithm. Experimental results show that compression efficiency of the proposed method has been improved up to 61.56% as compared to those of the JPEG-based compression scheme and up to 151.54% as compared to those of the conventional method averagely.     

基于视差信息的三维图像的计算机重构*

为了解决全景成像技术中观察者位于观察区域之外看到的图像会存在失真的问题,提出了一种基于视差信息的计算机重构3D视图技术.利用3D场景中的物体点经过不同微透镜在元素图像中记录的视差信息,根据光学路径分析,对重构视图中的失真部分用其它元素图像中存在的同一物体点的匹配像素进行替代,从而得到无失真的3D视图.该技术能够在更宽的观察区域内产生3D图像.     

基于视差信息的三维图像的计算机重构

为了解决全景成像技术中观察者位于观察区域之外看到的图像会存在失真的问题,提出了一种基于视差信息的计算机重构3D视图技术.利用3D场景中的物体点经过不同微透镜在元素图像中记录的视差信息,根据光学路径分析,对重构视图中的失真部分用其它元素图像中存在的同一物体点的匹配像素进行替代,从而得到无失真的3D视图.该技术能够在更宽的观察区域内产生3D图像.     

Modified smart pixel mapping method for displaying orthoscopic 3D images in integral imaging

In this paper, we propose a modified smart pixel mapping (MSPM) method for displaying orthoscopic three-dimensional (3D) images with a function of depth control in integral imaging system. In the proposed MSPM, the depth-converted elemental image array (EIA) is obtained through the pixel mapping process and the image interpolation technique. The proposed method gives us the depth conversion at distances different from the position of 3D object and provides various types of EIAs using only an original EIA for orthoscopic images. To show the usefulness of the proposed method, we carry out the preliminary experiments and present the experimental results.     

Optical display of true 3D objects in depth-priority integral imaging using an active sensor

In this paper, we propose a system combining the pickup process using an active sensor and the display process using depth-priority integral imaging (DPII) system to display true three-dimensional (3D) objects within large depth through real and virtual image fields. The active sensor provides depth map and color images of 3D objects. Using captured depth map and original color images, elemental images are computationally synthesized and displayed optically in DPII system. Proposed system provides scaling of 3D scenes for true 3D object. To show the usefulness of proposed system, we carry out the experiment for true 3D objects of three character patterns and present the experimental results.     

3D image reconstruction with controllable spatial filtering based on correlation of multiple periodic functions in computational integral imaging

We propose a novel method of slice image reconstruction with controllable spatial filtering by using the correlation of periodic delta-function arrays(PDFAs) with elemental images in computational integral imaging. The multiple PDFAs, whose spatial periods correspond to object's depths with the elemental image array(EIA), can generate a set of spatially filtered EIAs for multiple object depths compared with the conventional method for the depth of a single object. We analyze a controllable spatial filtering effect by the proposed method.To show the feasibility of the proposed method, we carry out preliminary experiments for multiple objects and present the results.     

Three-dimensional polarimetric integral imaging

A three-dimensional (3D) polarimetric image sensing and display technique based on integral imaging is proposed. Three-dimensional polarization distribution of reflected light from a 3D object can be measured as elemental image arrays by a rotating linear polarizer. After the measurement of the polarization of the 3D object, the 3D polarimetric object can be reconstructed optically by displaying the polarization-selected elemental images in spatial light modulators with two quarter-wave plates. Experimental demonstration of 3D polarimetric imaging of a 3D object attached to two orthogonal linear polarizers is presented. To the best of our knowledge, this is the first report on 3D polarimetric sensing imaging and 3D optical reconstruction by integral imaging.     

Three-dimensional integral imaging with improved visualization using subpixel optical ray sensing

In this Letter, we propose an improved three-dimensional (3D) image reconstruction method for integral imaging. We use subpixel sensing of the optical rays of the 3D scene projected onto the image sensor. When reconstructing the 3D image, we use a calculated minimum subpixel distance for each sensor pixel instead of the average pixel value of integrated pixels from elemental images. The minimum subpixel distance is defined by measuring the distance between the center of the sensor pixel and the physical position of the imaging lens point spread function onto the sensor, which is projected from each reconstruction point for all elemental images. To show the usefulness of the proposed method, preliminary 3D imaging experiments are presented. Experimental results reveal that the proposed method may improve 3D imaging visualization because of the superior sensing and reconstruction of optical ray direction and intensity information for 3D objects.