Joint geometric and photometric visual tracking based on Lie group

07/11/2017
Publication GSI2017
OAI : oai:www.see.asso.fr:17410:22434
contenu protégé  Document accessible sous conditions - vous devez vous connecter ou vous enregistrer pour accéder à ou acquérir ce document.
- Accès libre pour les ayants-droit
 

Résumé

This paper presents a novel efficient and robust direct visual tracking method under illumination variations. In our approach, non-Euclidean Lie group characteristics of both geometric and photometric transformations are exploited. These transformations form Lie groups and are parameterized by their corre-sponding Lie algebras. By applying the efficient second-order minimization trick, we derive an efficient second-order optimization technique for jointly solving the geometric and photometric parameters. Our approach has a high convergence rate and low iterations. Moreover, our approach is almost not affected by linear illu-mination variations. The superiority of our proposed method over the existing direct methods, in terms of efficiency and robustness is demonstrated through experiments on synthetic and real data.

Joint geometric and photometric visual tracking based on Lie group

Collection

application/pdf Joint geometric and photometric visual tracking based on Lie group Chenxi Li, Zelin Shi, Yunpeng Liu, Tianci Liu
Détails de l'article
contenu protégé  Document accessible sous conditions - vous devez vous connecter ou vous enregistrer pour accéder à ou acquérir ce document.
- Accès libre pour les ayants-droit

Joint geometric and photometric visual tracking based on Lie group
application/pdf Joint geometric and photometric visual tracking based on Lie group

Média

Voir la vidéo

Métriques

0
0
771.5 Ko
 application/pdf
bitcache://eed5715e23322c69f003e192440b6827b59ae289

Licence

Creative Commons Aucune (Tous droits réservés)

Sponsors

Sponsors Platine

alanturinginstitutelogo.png
logothales.jpg

Sponsors Bronze

logo_enac-bleuok.jpg
imag150x185_couleur_rvb.jpg

Sponsors scientifique

logo_smf_cmjn.gif

Sponsors

smai.png
logo_gdr-mia.png
gdr_geosto_logo.png
gdr-isis.png
logo-minesparistech.jpg
logo_x.jpeg
springer-logo.png
logo-psl.png

Organisateurs

logo_see.gif
<resource  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                xmlns="http://datacite.org/schema/kernel-4"
                xsi:schemaLocation="http://datacite.org/schema/kernel-4 http://schema.datacite.org/meta/kernel-4/metadata.xsd">
        <identifier identifierType="DOI">10.23723/17410/22434</identifier><creators><creator><creatorName>Chenxi Li</creatorName></creator><creator><creatorName>Zelin Shi</creatorName></creator><creator><creatorName>Yunpeng Liu</creatorName></creator><creator><creatorName>Tianci Liu</creatorName></creator></creators><titles>
            <title>Joint geometric and photometric visual tracking based on Lie group</title></titles>
        <publisher>SEE</publisher>
        <publicationYear>2018</publicationYear>
        <resourceType resourceTypeGeneral="Text">Text</resourceType><subjects><subject>Lie algebra</subject><subject>Visual tracking</subject><subject>illumination variations</subject><subject>efficient second-order minimization</subject><subject>Lie group</subject></subjects><dates>
	    <date dateType="Created">Sat 3 Mar 2018</date>
	    <date dateType="Updated">Sat 3 Mar 2018</date>
            <date dateType="Submitted">Fri 20 Jul 2018</date>
	</dates>
        <alternateIdentifiers>
	    <alternateIdentifier alternateIdentifierType="bitstream">eed5715e23322c69f003e192440b6827b59ae289</alternateIdentifier>
	</alternateIdentifiers>
        <formats>
	    <format>application/pdf</format>
	</formats>
	<version>37115</version>
        <descriptions>
            <description descriptionType="Abstract">This paper presents a novel efficient and robust direct visual tracking method under illumination variations. In our approach, non-Euclidean Lie group characteristics of both geometric and photometric transformations are exploited. These transformations form Lie groups and are parameterized by their corre-sponding Lie algebras. By applying the efficient second-order minimization trick, we derive an efficient second-order optimization technique for jointly solving the geometric and photometric parameters. Our approach has a high convergence rate and low iterations. Moreover, our approach is almost not affected by linear illu-mination variations. The superiority of our proposed method over the existing direct methods, in terms of efficiency and robustness is demonstrated through experiments on synthetic and real data.
</description>
        </descriptions>
    </resource>
.

adfa, p. 1, 2011. © Springer-Verlag Berlin Heidelberg 2011 Joint geometric and photometric visual tracking based on Lie group Chenxi Li 1, 2, 3,4,* , Zelin Shi 1, 3, 4 , Yunpeng Liu 1, 3, 4 , Tianci Liu 1, 2, 3,4 1 Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, Liaoning 110016, China 2 University of Chinese Academy of Sciences, Beijing 100049, China 3 Key Laboratory of Opto-electronic Information Processing, Chinese Academy of Sciences, Shenyang, Liaoning 110016, China 4 The Key Lab of Image Understanding and Computer Vision, Liaoning Province 110016, China Abstract. This paper presents a novel efficient and robust direct visual tracking method under illumination variations. In our approach, non-Euclidean Lie group characteristics of both geometric and photometric transformations are exploited. These transformations form Lie groups and are parameterized by their corre- sponding Lie algebras. By applying the efficient second-order minimization trick, we derive an efficient second-order optimization technique for jointly solving the geometric and photometric parameters. Our approach has a high convergence rate and low iterations. Moreover, our approach is almost not affected by linear illu- mination variations. The superiority of our proposed method over the existing direct methods, in terms of efficiency and robustness is demonstrated through experiments on synthetic and real data. Keywords: Visual tracking, illumination variations, Lie algebra, efficient sec- ond-order minimization, Lie group 1 Introduction Direct visual tracking can be formulated as finding the incremental transformations be- tween a reference image and successive frames of a video sequence. As utilizing all information of pixels of interest to estimate the transformation, it can give sub-pixel accuracy, which is necessary for certain applications, e.g., augmented reality, vision- based robot control [1], medical image analysis. Direct visual tracking problem can be made as complex as possible by considering illumination variations, occlusions and multiple-modality. In this paper, we focus on direct visual tracking which is formulated as iterative registration problem under global illumination variations. Traditional direct visual tracking methods often assume intensity constancy under Lucas-Kanade framework, where the sum of squared differences (SSD) is used as sim- ilarity metric. The inverse compositional (IC) method [2] and the efficient second-order * E-mail: lichenxi@sia.cn minimization (ESM) method [1] are two of the most efficient method. The drawback of these methods is their sensitivity to illumination variations. Two different strategies have been employed to improve the robustness to illumi- nation variations. First, robust similarity metrics are used, such as the normalized cor- relation coefficient (NCC) , the mutual information (MI) [3], the enhanced correlation coefficient (ECC) [4], and the sum of conditional variance (SCV) [5]. Recently, robust multi-dimensional features are used [6]. These methods have superior robustness to illumination variations, even multi-modality. However, these advantages come either at a high computational cost, or at low convergent radius. The second approach relies on modeling the illumination variations [7-12]. The affine photometric model is often used to compensate for illumination variations, either in a global way [7, 12] or in a local way [8-11]. In these approaches, all of them but DIC algorithm [12] used additive rule to update the photometric parameters in the optimization process. In this paper, we propose a very efficient and robust direct image registration ap- proach that jointly performs geometric and photometric registration by extending the efficient second-order minimization method. We also use the affine transformation to model illumination variations. Different from [11] where the photometric parameters were updated using additive rule, we employ the compositional rule to update both the geometric and photometric parameters, similar to [12]. Based on the joint Lie algebra parameterization of geometric and photometric transformation we derive a second-or- der optimization technique for image registration. Our approach preserves the ad- vantages of the original ESM with low iteration number, high convergence frequency. The rest of the paper is organized as follows. In section 2, we give the necessary theoretical background of our work. The details of our algorithm are given in section 3. Experimental results are presented in section 4. A conclusion is provided in section 5. 2 Theoretical Background 2.1 Lie algebra Parameterization of Geometric transformations We consider homography as the geometric transformations as it is the most general cases for planar objects. The coordinates of a pixel * q in the interest region *  of ref- erence image *  are related to its corresponding q in the current image  by a pro- jective homography G from which a warp ( ) * ; w q G can be induced. We employ the same parameterization way of homographies as in [1, 11]. The set of homographies is identified with the 3-D special linear group defined as ( ) 3  . The Lie group ( ) 3  and its Lie algebra ( ) 3 sl are related via exponential map, then a homog- raphy ( ) ( ) ∈ 3 G x  can be parameterized as follows: ( )=exp( ( )) ( ( )) . ! i i i ∞ =1 1 = ∑ G x Α x Α x (1) where ( ) Α x is the Lie algebra element of ( ) G x , [ , , ]T x x 8 1 8 = ∈ x   the geometric parameters vector [1, 11]. 2.2 Lie Algebra Parameterization of Photometric Transformation For gray-level images, we model the global illumination variations as affine photomet- ric transformation, which is also referred as the gain and bias model. Based on this model, the reference image *  and the current image  are related as *       . where  is the gain and  the bias. We rewrite this relation as matrix form *                            0 1 1 1   . Then the set of photometric transformations can be identified with 1-D affine Lie group ( ) ( , ) , 1 0 1 P                                  . In our problem, 0   . The Lie algebra associated with ( ) 1  is 2 ( )= t t t t 1 2 1 1 0 0 ,                        ga . Let { , } 2 1 B B be a basis of Lie algebra ( ) 1 ga . Each element ( ) ∈ 1 B ga can be written as a combination of i B , ( ) i i t 2 =1 = ∑ i B t B , with [ , ]T t t 2 1 2 = ∈ t  the photometric parameters vector. In this paper, we choose , 1 0   =   0 0   1 B . 2 0 1   =   0 0   B For each photometric transformation ( , ) P   ( ) 1   , we can parameterize it using its Lie algebra via ex- ponential map like geometric transformations as follows: ( ( ), ( )) exp( ( )) ! i i t t i 1 2 1 1 0 0 P t t B t                        (2) We remake that this parameterization is smooth and one-to-one onto, with a smooth inverse, for all , , 0       . In the following, if we emphasis on the photometric parameters t , then the Lie algebra parameterized photometric transformation ( ( ), ( )) P t t   simply denoted as ( ) P t . For convenience, we define the group action of the photometric transformation ( , ) P   on the intensity  as ( , ) P          . It satisfies ( , ) ( , ) ( , ) ( ( , ) ) ( ( , ) ( , )) 2 2 1 1 2 2 1 1 2 2 1 1 P P P P P P                       (3) according to the group properties of Lie group ( ) 1  . 3 The Proposed Visual Tracking Method Considering illumination variations, the visual tracking problem can be formulated as a search for the optimal geometric and photometric transformations between two the reference image and the current frame. Given an estimated geometric transformation Ĝ and an estimated photometric transformation ˆ ˆ ˆ ( , ) P   which are often given by pre- vious frame, our considered problem can be formulated as ( ) ( ) * * * * * , ˆ ˆ [ ( ) ; ( ) ( )] min i i i 8 2 2 ∈ ∈ ∈ 1 − 2 ∑ x t q P t P w q GG x q        (4) where ( ) G x and ( ) P t are the incremental geometric and photometric transformation. Problem (4) can be explicitly written as ( ) ( ) * * * * * , ˆ ˆ ˆ [( ; ( ) ) ( ) ( ) ( )] min i i i α β α β 8 2 2 ∈ ∈ ∈ 1 + + − 2 ∑ x t q w q GG x t t q      (5) Note that the geometric and photometric transformation in problem (4) can be de- fined over the joint Lie group ( ) (1) 3    whose corresponding Lie algebra is ( ) ( ) 3 ⊕ 1 sl ga . If we denote =[ , ] T T T θ x t to be the joint parameters, then ( ) ( ) 10 ∈ 3 ⊕ 1 = θ  sl ga . Let the error function in problem (5) be denoted as ( ) ( ) * * * * ˆ ˆ ˆ ( ) ( ; ( ) ) ( ) ( ) ( ) i i i D α β α β = + + − q θ w q GG x t t q   , It can be shown that the sec- ond-order approximation of * ( ) i Dq θ around = θ 0 is given by * * * * ( ) ( ) ( ( ) ( )) ( ) i i i i D D 3 1 = + + + 2 q q q q θ 0 J 0 J θ θ O θ (6) where * * * ( ) [ ( ), ( )] i i i D D = ∇ ∇ x t q q q J 0 0 0 and * * * ( ) [ ( ), ( )] i i i D D = ∇ ∇ x t q q q J θ θ θ are the Jaco- bians evaluated in 0 and θ respectively. The expressions for * ( ) i D ∇x q 0 and * ( ) i D ∇t q 0 can be directly computed as * ˆ ( ) i D α ∇ = x w G q 0 J J J  , * ( ) i D ∇ = t q 0 ( ) * ˆ ˆ ˆ [ ( ; ) , ] i α β + 1 w q G  . The detailed computation of the derivatives J , w J and G J can be found in [1]. * ( ) i D ∇x q θ and * ( ) i D ∇t q θ depend on the unknown parameters θ , therefore are not easy to compute. However, suppose that * * * =[ , ] T T T θ x t are the solu- tion of problem (5), based on the Lie algebra parameterization of both geometric trans- formations and photometric transformations, we can get that * * ( ) i D ∇x q θ * = w G J J J  [1] and * * * * ( ) [ ( ), ] i i D ∇ = 1 t q θ q  . Let gp esm J be the following (1 10) × matrix: ( ) * * * * * * * * ˆ ˆ ˆ ˆ ( ) ( ( ) ( )) [( ) , ( ; ) ( ), ] i i gp esm i i i α α β 1 1 = + = + + + 2 2 2 w G q q J q J 0 J θ J J J J w q G q     (7) Then the problem (5) can be approximated as a linear least squares problem: * * * * [ ( ) ( ) ] min i i gp esm i D 10 2 ∈ ∈ 1 + 2 ∑ q θ q 0 J q θ   (8) The solution of problem (8) is given by * * * * * * * * [ , ] ( ( ) ( )) ( ( ) ( )) i i i T T T T T gp gp gp esm i esm i esm i D −1 0 0 0 ∈ ∈ = = − ⋅ ∑ ∑ q q q θ x t J q J q J q 0   (9) The geometric transformation Ĝ and photometric transformation P̂ are simultane- ously updated as follows: ˆ ˆ ˆ ˆ ˆ ˆ ( ) exp( ( )) & ( ) exp( ( )) 0 0 0 0 ← = ← = G GG x G Α x P P t P B t P (10) The process is iterated until ε 0 < θ , where 2 =1 10 ε − × in our experiments. 4 Experimental Results In this section, we compare our algorithm with four different algorithms which are also designed for template tracking under illumination variations. They are DIC [12]; the algorithm proposed in [8], which we terms as ESM-PA because the photometric pa- rameters are updated using additive rule; (ECC) [4]; SCV [5]. Our implementation uses a PC equipped with Intel® Core™ i5-3470 CPU at 3.20GHz and 4G RAM. Fig. 1. The selected images used for synthetic experiments 4.1 Convergence comparison Fig. 1 shows the set of images used as the reference images whose gray-lever versions were used. Two templates with size of 100 100 × pixels were cropped from each of the reference image. As in [2, 4], a homography was simulated by adding Gaussian noise with standard deviation γ (γ captures the magnitude of geometric deformations) to the coordinates of the four corners of the template. The current image was generated by the simulated homography in conjunction with a gain  and a bias  . The initial val- ues for geometric and photometric transformations were set to identity maps. The max number of iterations was set to 50. We reported the performance comparison in terms of number of iterations and con- vergence frequency in Fig. 2. The results were averaged over 500 trials according to 500 random geometric transformations. As shown in Fig. 2a, our proposed algorithm always has the lowest number of iter- ations under different magnitude of the geometric transformation. Here the gain and bias are fixed to 1.3 and 15 respectively. One trial is considered to be converged if the difference between the estimated coordinates of the corners of the template and the ground truth is below 2 pixels. The convergence frequency is the percentage of the convergent trials over the whole 500 trials. As shown in Fig. 2b, our proposed algorithm always has the highest convergence frequency. From Fig. 2c, we can see that our proposed algorithm always has the lowest number of iterations under different gain values. Here the geometric deformation magnitude was fixed to 5 pixels. Note that our proposed algorithm is almost free of the influence of linear illumination variations, similar to ECC and DIC. Obviously, ESM-PA which use additive update rule of photometric parameters is heavily affected by illumination variations. Fig. 2d presents the convergence frequencies versus the gain. Our proposed algorithm has almost the same best results with ESM-PA and ECC. DIC performs slightly worse. SCV is affected heavily by variations of the gain. (a) (b) (c) (d) Fig. 2. Performance comparison on image registration task. 4.2 Tests on Template Tracking We selected three videos from (www.cs.cmu.edu/˜halismai/bitplanes). These videos contain sudden illumination variations and low light, therefore are challenging for di- rect visual tracking. While the original videos are recorded at 120 Hz, we extracted images from the videos at 40 Hz resulting in three image sequences where large inter- frame displacements are induced. In this experiment, we fixed the max number of iter- ations for each algorithm to 30 and each frame was resized to180 320 × . Fig. 3 plots the number of iterations for each frame during tracking. Fig. 4 presents some examples of tracking results. The legends in Fig. 4 correspond to those in Fig. 3. Sever illumination variation occurs at frame #172 and #362 in the first image sequence, at frame #116 in the second image sequence and at frame #414 in the third image se- quence. Note that the number of iterations of ESM-PA increases dramatically when sever illumination variation occurs as shown in Fig. 3. In the first and third image se- quences, SCV is affected heavily by the sever illumination variations as well. In fact, SCV failed at frame #433 in the first sequence, as shown in Fig. 4a. DIC failed in all the three sequences as shown in Fig. 4. ECC failed in the second image sequence at frame #281. Our proposed algorithm can successfully track the template in all of these image sequences. Table 1 shows the average number of iterations and runtime per frame. We can see that in all of the three image sequences our proposed algorithm needs the lowest iterations and runtime. Table 1. Template tracking average number of iterations per frame. In parenthesis we show the average runtime (seconds). N/A stands for tracking failure. Sequence (tem- plate size) Our proposed DIC ESM-PA SCV ECC Sequence 1 (128 163 × ) 5.834 (0.0446) N/A 14.50 (0.1139) N/A 7.567 (0.1183) Sequence 2 (133 140 × ) 3.833 (0.0305) N/A 11.45 (0.0873) 3.903 (0.0351) N/A Sequence 3 (145 175 × ) 4.900 (0.0481) N/A 9.675 (0.0949) 5.740 (0.0646) 8.102 (0.1523) (a) (b) (c) Fig. 3. The number of iterations for each algorithm when tracking in the three image sequences. (a) Sequence 1 (b) Sequence 2 (c) Sequence 3. (a) Sequence 1 with medium texture. (b) Sequence 2 with ambiguous texture. (c) Sequence 3 with high texture. Fig. 4. Tracking results in the three image sequences with gray-level intensities. (a) Sequence 1 with medium texture. (b) Sequence 2 with ambiguous texture. (c) Sequence 3 with high texture. 5 Conclusions In this paper, we have proposed an efficient and robust direct visual tracking algorithm based on the efficient second-order minimization method. In our approach, Lie group structure of both the photometric and geometric transformations are exploited. As a second-order optimization technique, our algorithm preserves the permits of the origi- nal ESM which has high convergence frequency and low number of iterations. The efficiency and robustness of our proposed algorithm is verified by comparing with several well-known algorithms through synthetic data and real data. Compared to ESM-PA, our algorithm is more efficient under illumination variations. References 1. Benhimane, S. and E. Malis, Homography-based 2D visual tracking and servoing. International Journal of Robotics Research, 2007. 26(7): p. 661-676. 2. Baker, S. and I. Matthews, Lucas-Kanade 20 years on: A unifying framework. International Journal of Computer Vision, 2004. 56(3): p. 221-255. 3. Dame, A. and E. Marchand, Second-Order Optimization of Mutual Information for Real- Time Image Registration. Ieee Transactions on Image Processing, 2012. 21(9): p. 4190-4203. 4. Evangelidis, G.D. and E.Z. Psarakis, Parametric image alignment using enhanced correlation coefficient maximization. Ieee Transactions on Pattern Analysis and Machine Intelligence, 2008. 30(10): p. 1858-1865. 5. Richa, R., et al., Visual Tracking Using the Sum of Conditional Variance. 2011 Ieee/Rsj International Conference on Intelligent Robots and Systems, 2011: p. 2953-2958. 6. Alismail, H., B. Browning, and S. Lucey, Robust Tracking in Low Light and Sudden Illumination Changes. Proceedings of 2016 Fourth International Conference on 3d Vision (3dv), 2016: p. 389-398. 7. Luong, H.Q., et al., Joint photometric and geometric image registration in the total least square sense. Pattern Recognition Letters, 2011. 32(15): p. 2061-2067. 8. Silveira, G. and E. Malis, Unified Direct Visual Tracking of Rigid and Deformable Surfaces Under Generic Illumination Changes in Grayscale and Color Images. International Journal of Computer Vision, 2010. 89(1): p. 84-105. 9. Gouiffes, M., et al., A study on local photometric models and their application to robust tracking. Computer Vision and Image Understanding, 2012. 116(8): p. 896-907. 10. Fouad, M.M., R.M. Dansereau, and A.D. Whitehead, Image Registration Under Illumination Variations Using Region-Based Confidence Weighted M-Estimators. Ieee Transactions on Image Processing, 2012. 21(3): p. 1046-1060. 11. Silveira, G. and E. Malis, Real-time visual tracking under arbitrary illumination changes. 2007 Ieee Conference on Computer Vision and Pattern Recognition, Vols 1-8, 2007: p. 1-6. 12. Bartoli, A., Groupwise Geometric and Photometric Direct Image Registration. Ieee Transactions on Pattern Analysis and Machine Intelligence, 2008. 30(12): p. 2098-2108.