Spatio-temporal segmentation of image sequences for object-oriented low bit-rate image coding

This paper describes a new method of segmentation of time-varying image sequences whose goal is object-oriented image coding. The segmentation represents a partition of each frame of the sequence into a set of regions which are homogeneous with regard to motion criterion. The region borders correspond to spatial contours of objects in the frame. Each spatio-temporal region is characterized by its temporal component, which is a model-dependent vector of motion parameters, and a structural component representing the polygonal approximation of the spatial contour of the region.

The construction of spatio-temporal segmentation includes two phases: the initialization step and temporal tracking. The initialization step is based on the spatial segmentation of the first frame of the sequence. Then homogeneous spatial regions are merged through motion estimation in accordance with a motion-based criterion. The temporal tracking consists of the projection of the segmentation along the time axis, and its adjustment. Special attention is paid to the processing of occlusions.

A predictive coding scheme is proposed which is based on the temporal coherence of the segmentation. This scheme is promising for a low bit-rate image compression.

The results for teleconference and TV sequences show the high visual quality of reconstructed only by prediction images. Moreover, the bit-rates for motion coding are very low: from 0.002 to 0.007 bit/pixel for teleconference sequence and from 0.004 to 0.021 bit/pixel for complex TV sequence. A scheme for encoding of the structural information is proposed which requires 0.083 – 0.17 bit per pixel depending on the content of the sequence.     

Hierarchical morphological segmentation for image sequence coding

This paper deals with a hierarchical morphological segmentation algorithm for image sequence coding. Mathematical morphology is very attractive for this purpose because it efficiently deals with geometrical features such as size, shape, contrast, or connectivity that can be considered as segmentation-oriented features. The algorithm follows a top-down procedure. It first takes into account the global information and produces a coarse segmentation, that is, with a small number of regions. Then, the segmentation quality is improved by introducing regions corresponding to more local information. The algorithm, considering sequences as being functions on a 3-D space, directly segments 3-D regions. A 3-D approach is used to get a segmentation that is stable in time and to directly solve the region correspondence problem. Each segmentation stage relies on four basic steps: simplification, marker extraction, decision, and quality estimation. The simplification removes information from the sequence to make it easier to segment. Morphological filters based on partial reconstruction are proven to be very efficient for this purpose, especially in the case of sequences. The marker extraction identifies the presence of homogeneous 3-D regions. It is based on constrained flat region labeling and morphological contrast extraction. The goal of the decision is to precisely locate the contours of regions detected by the marker extraction. This decision is performed by a modified watershed algorithm. Finally, the quality estimation concentrates on the coding residue, all the information about the 3-D regions that have not been properly segmented and therefore coded. The procedure allows the introduction of the texture and contour coding schemes within the segmentation algorithm. The coding residue is transmitted to the next segmentation stage to improve the segmentation and coding quality. Finally, segmentation and coding examples are presented to show the validity and interest of the coding approach.     

A fast quadtree motion segmentation for image sequence coding

A new fast quadtree-based variable size block matching scheme is devised for image sequence coding. The novelty of the method lies in the quadtree-based block matching with significantly less computations. In the proposed method, the computational requirements for matching are largely transferred from the evaluation of the mean absolute difference to the computations of the sum norms of the blocks which can be easily obtained. The fast quadtree decomposition scheme partitions an image frame into uniform motion blocks (larger blocks) and non-uniform motion blocks (small blocks). Standard VQ-based interframe coding is applied to uniform motion blocks while non-uniform blocks are intraframe coded. Implementation of this method shows a significantly lower computational requirement and a good-quality reconstruction of images at low bit-rates.     

Image segmentation for compression of images and image sequences

Region-based image coding schemes, the so-called second generation techniques, have gained much favour in recent years. For still picture coding, they can increase the compression ratio obtained by transform coding by an order of magnitude, while maintaining adequate image representation. The success of these techniques relies on the ability to describe regions in an image succinctly by their shape and size. The algorithms presented describe methods for segmenting images. Unlike most other region based algorithms, these algorithms incorporate knowledge of the border coding process in deciding how to partition the image. The extension from single image compression to sequential image compression is also considered. A new, efficient segmentation scheme is proposed that exploits temporal redundancies between successive images, and reduces some problems associated with error accumulation in error images     

Three-dimensional parameter estimation from stereo image sequences for model-based image coding

It is shown that the analysis of moving image sequences for 3D modelling can be performed in a relatively straightforward manner if the scene is captured in stereo. Output from a stereo disparity estimation process using calibrated cameras gives absolute 3D surface coordinates from a single stereo pair. When combined with monocular motion cues, the true 3D motion parameters of moving objects can be accurately calculated. Further analysis enables segmentation of body elements according to motion while the 3D surface feature structure, although available from the start, can be integrated and checked for anomalies over the sequence. These results are expected to alleviate the known problems of ambiguity suffered by monocular-source model-based coders.     

Motion-based region growing segmentation of image sequences

This paper proposes a motion-based region growing segmentation scheme for the object-based video coding, which segments an image into homogeneous regions characterized by a coherent motion. It adopts a block matching algorithm to estimate motion vectors and uses morphological tools such as open-close by reconstruction and the region-growing version of the watershed algorithm for spatial segmentation to improve the temporal segmentation. In order to determine the reliable motion vectors, this paper also proposes a change detection algorithm and a multi-candidate pro- screening motion estimation method. Preliminary simulation results demonstrate that the proposed scheme is feasible. The main advantage of the scheme is its low computational load.     

Adaptive entropy-coded subband coding of image sequences

A new approach to image sequence coding based on variable-rate entropy-constrained subband coding (ECSBC) is described. The corresponding practical implementation of the ECSBC scheme for fixed-rate channels is developed by extending recent adaptive entropy-coded (AEC) quantization techniques. Although the entropy-constrained design of subband coding systems provides improve coding efficiency compared to level-constrained design approaches, the resulting coding system generates variable-rate outputs which must be buffered before fixed-rate transmissions. In this case, the finite buffer, however large, that interfaces the encoder and the channel will eventually overflow or underflow, resulting in a catastrophic loss of encoder-decoder synchronism with an associated large amount of distortion. A buffer-adaptive arithmetic-coded implementation of the ECSBC scheme, called adaptive entropy-coded subband coding (ECSBC/AEC), is described to completely eliminate the associated encoder buffer overflow/underflow problems, even with a very small encoder buffer     

基于图像序列的模糊聚类嘴唇分割算法

嘴唇分割是唇读系统中很重要的一步。由于人们说话是一个连续的过程,所以最终需要处理的是连续的帧。但是,现有的大多数嘴唇分割算法都是针对于孤立的图片而不是连续的嘴唇序列,因此在嘴唇分割的过程中忽略了连续两帧之间的联系。文中提出了基于模糊聚类的图像序列嘴唇分割算法。该算法是在传统模糊聚类框架中融入了运动信息和参考颜色信息。文中首先利用已有的算法来处理图像序列,发现实验结果并不稳定。通过分析发现:当像素点在连续两帧间运动较小或静止时,人们希望它们的分类和上一帧保持连续性;对于同一个人的嘴唇序列,其嘴唇聚类中心不会发生较大的偏移。基于以上两点,提出了基于运动信息的模糊聚类图像序列嘴唇分割算法。     

3D structure extraction coding of image sequences

This paper presents a 3D structure extraction coding scheme that first computes the 3D structural properties such as 3D shape, motion, and location of objects and then codes image sequences by utilizing such 3D information. The goal is to achieve efficient and flexible coding while still avoiding the visual distortions through the use of 3D scene characteristics inherent in image sequences. To accomplish this, we present two multiframe algorithms for the robust estimation of such 3D structural properties, one from motion and one from stereo. The approach taken in these algorithms is to successively estimate 3D information from a longer sequence for a significant reduction in error. Three variations of 3D structure extraction coding are then presented — 3D motion interpolative coding, 3D motion compensation coding, and “viewpoint” compensation stereo image coding — to suggest that the approach can be viable for high-quality visual communications.     

基于分段均衡的红外热图像伪彩色编码

根据红外热图像的特点,并克服传统伪彩色处理方法的缺陷,提出一种基于分段均衡的红外热图像伪彩色编码方法。先通过对红外热图像采用改进的K-means算法进行聚类,对聚类结果的各类像素点坐标进行统计,根据统计结果在原始红外热图中进行分段均衡,再通过彩虹编码法对分段均衡后的图像进行伪彩色编码。实验结果表明:通过该方法编码后的红外热图像色彩丰富,细节增强,具有更好的层次感,便于人眼观察分析以及应用。     

Subband coding of image sequences at low bit rates

In this paper a low bit rate subband coding scheme for image sequences is described. Typically, the scheme is based on temporal DPCM in combination with an intraframe subband coder. In contrast to previous work, however, the subbands are divided into blocks onto which conditional replenishment is applied, while a bit allocation algorithm divides the bits among the blocks assigned for replenishment. A solution is given for the ‘dirty window’ effect by setting blocks to zero that were assigned to be replenished but received no bits. The effect of motion compensation and the extension to color images are discussed as well. Finally, several image sequence coding results are given for a bit rate of 300 kbit/s.     

Object-based coding of stereoscopic and 3D image sequences

The author begins by discussing what object based coding is and goes on to consider the structure of object based stereoscopic coders. Different techniques for object-based stereoscopic image sequence coding are reviewed. These techniques basically differ in the way they define models and estimate model parameters. We review the various models used for representing motion and structure. Then we review segmentation techniques, and discuss coding of object parameters and image synthesis     

Global-motion estimation in image sequences of 3-D scenes for coding applications

A technique for global-motion estimation and compensation in image sequences of 3-D scenes is described in this paper. Each frame is segmented into regions whose motion can be described by a single set of parameters and a set of motion parameters is estimated for each segment. This is done using an iterative block-based image segmentation combined with the estimation of the parameters describing the global motion of each segment. The segmentation is done using a Gibbs-Markov model-based iterative technique for finding a local optimum solution to a maximum a posteriori probability (MAP) segmentation problem. The initial condition for this process is obtained by applying a Hough transform to the motion vectors of each block in the frame obtained by block matching. In each iteration, given a segmentation, the motion parameters are estimated using the least-squares (LS) technique. To obtain the final segmentation and the more appropriate higher-order motion model for each segment, a final stage of splitting/merging of segments is needed. This step is performed on the basis of maximum-likelihood decisions combined with the determination of the higher-order model parameters by LS. The incorporation of the proposed global-motion estimation technique in an image-sequence coder was found to bring about a substantial reduction in bit-rate without degrading the perceived quality or the PSNR.     

一种新的结合稀疏编码的红外图像聚类分割算法

聚类作为一种重要的图像分割方法得到了大量研究,提出了一种新的结合稀疏编码的红外图像聚类分割算法,扩展了传统的基于K-means聚类的图像分割方法。结合稀疏编码的聚类算法能有效融合图像的局部信息,而且易于利用像素之间的内在相关性,但是对于分割会出现过分割和像素难以归类的问题。为此,在字典的学习过程中,将原子的聚类算法引入其中,有助于缩减字典中原子所属类别的数目防止出现过分割;同时将稀疏编码系数同原子对聚类中心的隶属程度相结合来判断像素所属的类别。这种处理方式能更好地实现利用像素的内在相关性进行聚类分割,并在其中自然引入了局部空间信息,达到更好分离目标区域和背景区域的目的。实验结果表明,结合稀疏编码的K-means聚类分割算法能更好的实现复杂背景下红外图像重要区域的准确分割提取。     

The coding of image sequences by wavelets, wavelet packets andother subband coding schemes

For the terrestrial broadcasting of high-definition television (HDTV) in North America the discrete cosine transform (DCT), Huffman (lossless) coding and motion detection, estimation and compensation have been used to compress raw HDTV rates of 1.0-1.2 Gbps. Compressed and compacted video bit rates range from 12.59 Mbps to 13.60 Mbps for 16-QAM transmission and from 17.47 to 18.88 for 32-QAM transmission. MPEG source coding schemes, are used in a proposed HDTV system and in the Grand Alliance proposal. They share features in common with other source coding schemes which have been employed. It seems likely that compression and compaction to an HDTV video bit rate below 10 Mbps should be attainable. The methods discussed in this paper provide hope for realizing this goal     

Wavelet image coding using variable blocksize vector quantization with optimal quadtree segmentation

In this paper, we propose an image coding scheme by using the variable blocksize vector quantization (VBVQ) to compress wavelet coefficients of an image. The scheme is capable of finding an optimal quadtree segmentation of wavelet coefficients of an image for VBVQ subject to a given bit budget, such that the total distortion of quantized wavelet coefficients is minimal. From our simulation results, we can see that our proposed coding scheme has higher performance in PSNR than other wavelet/VQ or subband/VQ coding schemes.     

Semi-automatic foreground/background segmentation of motion picture images and image sequences

Segmentation of images into foreground (an actor) and background is required for many motion picture special effects. To produce these shots, the unwanted background must be removed so that none of it appears in the final composite shot. The standard approach requires the background to be a blue screen. Systems that are capable of segmenting actors from more natural backgrounds have been proposed, but many of these are not readily adaptable to the resolution involved in motion picture imaging. An algorithm is presented that requires minimal human interaction to segment motion picture resolution images. Results from this algorithm are quantitatively compared with alternative approaches. Adaptations to the algorithm, which enable segmentation even when the foreground is lit from behind, are described. Segmentation of image sequences normally requires manual creation of a separate hint image for each frame of a sequence. An algorithm is presented that generates such hint images automatically, so that only a single input is required for an entire sequence. Results are presented that show that the algorithm successfully generates hint images where an alternative approach fails.     

Block-based motion field segmentation for video coding

In the past few years, motion compensation has been widely used in the coding of image sequences. Most of motion estimation and compensation schemes belong to block-based framework. The framework simplifies the complexity of motion estimation, but gives over constraints to the motion field, which results in worse accuracy on the boundary of moving objects. This paper presents a novel technique for raising motion field accuracy. It uses several pre-defined pattern types to segment the motion fields of the previous frame of a sequence. The segmentation is based on the MAP framework that uses iterative method to obtain the solution. In addition, we develop a predictive scheme to predict the location of motion field discontinuities in the current frame, which further reduces the side information for the representation of segmentation.     

Polygonal approximation for image segmentation

A new learning model for real-time, grey-level image segmentation is presented. The model gives excellent results for images with different shapes.     

Bottom-up motion compensated prediction in wavelet domain for spatially scalable video coding

A new in-band motion compensation algorithm for wavelet-based video coding is proposed: the bottom-up prediction algorithm (BUP). This algorithm overcomes the periodic shift-invariance of the discrete wavelet transform (DWT) and is formalised into prediction rules using filtering operations. The combination of all prediction rules of the BUP algorithm defines a new transform: the bottom-up overcomplete DWT or BUP ODWT, which is shift-invariant. The envisaged application for the BUP algorithm is spatially scalable wavelet video coding.