Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities

28/08/2013
OAI : oai:www.see.asso.fr:2552:4855
DOI :

Résumé

Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities

Métriques

677
198
232.4 Ko
 application/pdf
bitcache://f4aa9b5f23e1d1cdf49d7a4e443f128d4755e9a5

Licence

Creative Commons Aucune (Tous droits réservés)

Sponsors

Sponsors scientifique

logo_smf_cmjn.gif

Sponsors financier

logo_gdr-mia.png
logo_inria.png
image010.png
logothales.jpg

Sponsors logistique

logo-minesparistech.jpg
logo-universite-paris-sud.jpg
logo_supelec.png
Séminaire Léon Brillouin Logo
logo_cnrs_2.jpg
logo_ircam.png
logo_imb.png
<resource  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                xmlns="http://datacite.org/schema/kernel-4"
                xsi:schemaLocation="http://datacite.org/schema/kernel-4 http://schema.datacite.org/meta/kernel-4/metadata.xsd">
        <identifier identifierType="DOI">10.23723/2552/4855</identifier><creators><creator><creatorName>Jean-François Bercher</creatorName></creator></creators><titles>
            <title>Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities</title></titles>
        <publisher>SEE</publisher>
        <publicationYear>2013</publicationYear>
        <resourceType resourceTypeGeneral="Text">Text</resourceType><dates>
	    <date dateType="Created">Mon 16 Sep 2013</date>
	    <date dateType="Updated">Mon 25 Jul 2016</date>
            <date dateType="Submitted">Fri 17 Aug 2018</date>
	</dates>
        <alternateIdentifiers>
	    <alternateIdentifier alternateIdentifierType="bitstream">f4aa9b5f23e1d1cdf49d7a4e443f128d4755e9a5</alternateIdentifier>
	</alternateIdentifiers>
        <formats>
	    <format>application/pdf</format>
	</formats>
	<version>9632</version>
        <descriptions>
            <description descriptionType="Abstract"></description>
        </descriptions>
    </resource>
.

The Begin The ChiBeta Divergence GeneralizedCR FII The End Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities Jean-François Bercher Laboratoire d’Informatique - Institut Gaspard Monge, UMR 8049 Université Paris-Est, Esiee-Paris, France GSI – session Divergence Geometry & Ancillarity Paris, august 29, 2013 0 / Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Averaging Estimation context: estimate h(θ) by a function T(X) of the data In estimation, the error is T(X)−h(θ), and Idea: compute a moment of order α of the error 1 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Averaging Estimation context: estimate h(θ) by a function T(X) of the data In estimation, the error is T(X)−h(θ), and Idea: compute a moment of order α of the error with respect to a distribution g(x;θ) instead of f(x;θ): 1 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Averaging Estimation context: estimate h(θ) by a function T(X) of the data In estimation, the error is T(X)−h(θ), and Idea: compute a moment of order α of the error with respect to a distribution g(x;θ) instead of f(x;θ): the bias can be evaluated as´ X (T(x)−h(θ)) f(x;θ)dx = Ef [T(X)−h(θ)] = η(θ)−h(θ), while a general moment of the error can be computed with respect to another probability distribution g(x;θ), as in Eg |T(X)−h(θ)|β = ´ X |T(x)−h(θ)|α g(x;θ)dx. 1 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Averaging Estimation context: estimate h(θ) by a function T(X) of the data In estimation, the error is T(X)−h(θ), and Idea: compute a moment of order α of the error with respect to a distribution g(x;θ) instead of f(x;θ): the bias can be evaluated as´ X (T(x)−h(θ)) f(x;θ)dx = Ef [T(X)−h(θ)] = η(θ)−h(θ), while a general moment of the error can be computed with respect to another probability distribution g(x;θ), as in Eg |T(X)−h(θ)|β = ´ X |T(x)−h(θ)|α g(x;θ)dx. The two distributions f(x;θ) and g(x,θ) can be chosen very arbitrary, e.g as a pair of escorts f(x;θ) = g(x;θ)q ´ g(x;θ)qdx and g(x;θ) = f(x;θ)¯q ´ f(x;θ)¯qdx 1 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Agenda In this presentation... A variant of a χβ -divergence and its properties A generalized Fisher Information Extensions of Cramér-Rao inequality For general norms and powers and weighting Matrix forms Particular case of location parameter, with escort distributions Extension of the Fisher information Inequality 2 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End The χβ divergence and Fisher information χβ -divergence, β > 1, between f1 and f2 χβ (f1,f2) = Ef2 1− f1 f2 β the Fisher information of fθ is nothing but I2[fθ ;θ] = lim |t|→0 χ2 (fθ+t ,fθ )/t2 Vajda’s generalized Fisher information Iβ [fθ ;θ] = lim |t|→0 χβ (fθ+t ,fθ )/|t|β = Efθ   ˙fθ fθ β   Vajda, I.– χα -divergence and generalized Fisher information. In: Transactions of the Sixth Prague Conference on Information Theory, Statistical Decision Functions and Random Processes. pp. 873-886 (1973) 3 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End A modified χβ -divergence and generalized Fisher information A χβ -divergence modified in order to average wrt a third distribution g(x;θ) : χ β g (f1,f2) = Eg f2 −f1 g β 4 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End A modified χβ -divergence and generalized Fisher information A χβ -divergence modified in order to average wrt a third distribution g(x;θ) : χ β g (f1,f2) = Eg f2 −f1 g β Generalized Fisher information, taken wrt gθ Iβ [fθ |gθ ;θ] = lim |t|→0 χ β g (fθ+t ,fθ )/|t|β = Egθ   ˙fθ gθ β   4 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End A modified χβ -divergence and generalized Fisher information A χβ -divergence modified in order to average wrt a third distribution g(x;θ) : χ β g (f1,f2) = Eg f2 −f1 g β Generalized Fisher information, taken wrt gθ Iβ [fθ |gθ ;θ] = lim |t|→0 χ β g (fθ+t ,fθ )/|t|β = Egθ   ˙fθ gθ β   In the multivariate case, Iβ [fθ |gθ ;θ] = Egθ ∇fθ gθ β β 4 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End A modified χβ -divergence and generalized Fisher information A χβ -divergence modified in order to average wrt a third distribution g(x;θ) : χ β g (f1,f2) = Eg f2 −f1 g β Generalized Fisher information, taken wrt gθ Iβ [fθ |gθ ;θ] = lim |t|→0 χ β g (fθ+t ,fθ )/|t|β = Egθ   ˙fθ gθ β   In the multivariate case, Iβ [fθ |gθ ;θ] = Egθ ∇fθ gθ β β This generalized Fisher information is involved in a generalized Cramér-Rao inequality for parameter estimation. 4 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Properties 1 - The modified χ β g -divergence has information monotonicity. Coarse-graining the data leads to a loss of information: χ β g (f1,f2) ≥ χ β ˜g (˜f1, ˜f2). Data processing inequality: if Y = φ(X), then χ β g (f1,f2) ≥ χ β gφ (f φ 1 ,f φ 2 ), Iβ [fθ |gθ ;θ] ≥ Iβ [f φ θ |g φ θ ;θ]. 2 - Matrix Fisher data processing inequality:I2,g[θ] ≥ I2,gφ [θ] Taylor expansion fθ+t = fθ +∑i ti∂ifθ + 1 2 ∑i ∑j titj∂2 ij fθ +... χ2 g (fθ ,fθ+t ) = Eg fθ+t −fθ gθ 2 = Eg ∑i ∑j titj ∂i fθ ∂j fθ g2 = tT I2,g[θ]t, where I2,g[θ] = Eg ψgψT g = Eg ∇fθ gθ ∇fT θ gθ Since χ2 g (fθ ,fθ+t ) ≥ χ2 gφ (f φ θ ,f φ θ+t ), then I2,g[θ] ≥ I2,gφ [θ] 5 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End 3 - If T(X) is a statistic, then with α−1 +β−1 = 1, α ≥ 1, we have Ef2 [T]−Ef1 [T] ≤ Eg |T|α 1 α χ β g (f1,f2) 1 β . It suffices here to consider Eg T f2−f1 g = Ef2 [T]−Ef1 [T] and then apply the Hölder inequality 6 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End 3 - If T(X) is a statistic, then with α−1 +β−1 = 1, α ≥ 1, we have Ef2 [T]−Ef1 [T] ≤ Eg |T|α 1 α χ β g (f1,f2) 1 β . It suffices here to consider Eg T f2−f1 g = Ef2 [T]−Ef1 [T] and then apply the Hölder inequality Generalized Cramér-Rao inequality. Set f2 = fθ+t , f1 = fθ , and denote η = Ef [T(X)]. Then divide both sides by t, substitute T(X) by T(X)−h(θ), and take the limit t → 0.Then Eg |T(X)−h(θ)|α 1 α Iβ [fθ |gθ ;θ] 1 β ≥ d dθ η with Iβ [fθ |gθ ;θ] = Egθ ˙fθ gθ β . 6 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End 3 - If T(X) is a statistic, then with α−1 +β−1 = 1, α ≥ 1, we have Ef2 [T]−Ef1 [T] ≤ Eg |T|α 1 α χ β g (f1,f2) 1 β . It suffices here to consider Eg T f2−f1 g = Ef2 [T]−Ef1 [T] and then apply the Hölder inequality Generalized Cramér-Rao inequality. Set f2 = fθ+t , f1 = fθ , and denote η = Ef [T(X)]. Then divide both sides by t, substitute T(X) by T(X)−h(θ), and take the limit t → 0.Then Eg |T(X)−h(θ)|α 1 α Iβ [fθ |gθ ;θ] 1 β ≥ d dθ η with Iβ [fθ |gθ ;θ] = Egθ ˙fθ gθ β . Multivariate case In the multivariate case, similar results exist involving a vector score function and general norms (jfb – J. Phys. A: Math. Theor., march 2013) 6 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Generalized Cramér-Rao - matrix version When h(θ) is scalar valued, we have the following variation on the Cramér-Rao bound theme, which involves a Fisher information matrix, but unfortunately in a non-explicit form. Theorem Let T(x) be an estimator of a scalar-valued function h(θ) and set η(θ) = Ef [T(X)]. Then, under some regularity conditions Eg |T(X)−h(θ)|α 1 α ≥ sup A>0 ˙η(θ)T A ˙η(θ) Eg ˙η(θ)T Aψg(X;θ) β 1 β . with a certain equality condition, and where ψg(x;θ) a score function given with respect to g(x;θ) : ψg(x;θ) := ∇θ f(x;θ) g(x;θ) . The inverse of the matrix A which maximizes the right hand side is the Fisher information matrix of order β. 7 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Quadratic case β = 2, maximum attained for A−1 = I2,g[θ] = Eg[ψgψt g], and Eg |T(X)−h(θ)|2 ≥ ∇θ η(θ)t I2,g[θ]−1 ∇θ η(θ) Inequality for the covariance of the estimation error. Proof following (Liese, 2008) Eg (T(X)−h(θ))(T(X)−h(θ))t ≥ ˙ηt I2,g[θ]−1 ˙η with ˙η = ∇θ Ef [T(X)t ]. [Related inequality in (Naudts, 2004)] Scalar case (or the case of a single component of θ) Eg |T(X)−h(θ)|α 1 α ≥ | ˙η|/Eg ψg(X;θ) β 1 β , This inequality recovers at once the generalized Cramér-Rao inequality we presented in the univariate case. Suggests possible applications in robust estimation jfb - On generalized Cramér-Rao inequalities, generalized Fisher informations and characterizations of generalized q-Gaussian distributions, J. Phys. A: Mathematical and Theoretical, 2012 8 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Case of a translation family In the particular case of a translation parameter, the generalized Cramér-Rao inequalities leads to new functional inequalities and to new characterizations of q-Gaussians. Let θ be a location parameter, and define by f(x;θ) the family of density f(x;θ) = f(x −θ). In this case, we have that ∇θ f(x;θ) = −∇x f(x −θ) Without loss of generality, assume that f(x) has zero mean. The estimator ˆθ(x) = x is unbiased (wrt f). Assume the θ = 0 9 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Case of a translation family In the particular case of a translation parameter, the generalized Cramér-Rao inequalities leads to new functional inequalities and to new characterizations of q-Gaussians. Let θ be a location parameter, and define by f(x;θ) the family of density f(x;θ) = f(x −θ). In this case, we have that ∇θ f(x;θ) = −∇x f(x −θ) Without loss of generality, assume that f(x) has zero mean. The estimator ˆθ(x) = x is unbiased (wrt f). Assume the θ = 0 This leads to a very general functional inequality (skipped) In the special case of an escort pair, 9 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Generalized q-Cramér-Rao inequality Corollary Under some regularity conditions, and for a pair f,g of escort distributions, the following holds Eg X α 1 α Iβ,q [g] 1 β ≥ n, with Iβ,q [g] = (q/Mq [g])β Eg g(x)β(q−1) ∇x g(x) g(x) β ∗ , with equality if and only if g(x) is a generalized q-Gaussian, i.e. g(x) ∝ 1−γ x α 1 q−1 + , with γ > 0. Also: Cov(X) ≥ I2,q(X)−1 with equality iff X is a q-Gaussian This generalizes the well-known fact that the Gaussian with a given variance minimizes Fisher information Complements the entropic characterization of generalized q-Gaussians 10 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Fisher Information Inequality Standard proofs of the FII (eg Stam 1959 or Rioul 2011) I2,1[f1 ∗f2]−1 ≥ I2,1[f1]−1 +I2,1[f2]−1 use two basic ingredients 1 monotonicity and 2 additivity. 11 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Fisher Information Inequality Standard proofs of the FII (eg Stam 1959 or Rioul 2011) I2,1[f1 ∗f2]−1 ≥ I2,1[f1]−1 +I2,1[f2]−1 use two basic ingredients 1 monotonicity and 2 additivity. However the additivity of Fisher information does not hold in the extended context. In the case of a pair of escorts, we still have (by Minkovski inequality) Iβ,q[fX fY ] 1 β ≤ Mβ(q−1)+1[fY ] 1 β Mq[fY ] Iβ,q[fX ] 1 β + Mβ(q−1)+1[fX ] 1 β Mq[fX ] Iβ,q[fY ] 1 β , and an equality in the quadratic case I2,q[fX fY ] = M2q−1[fY ] Mq[fY ]2 I2,q[fX ]+ M2q−1[fX ] Mq[fX ]2 I2,q[fY ] 11 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Following (Zamir, 1998) and using only the monotonicity property for the Fisher Information Matrix, we have also a matrix form of the FII: At I2,g(AX)A ≤ I2,g(X) I2,g(AX) ≤ AI2,g(X)−1At −1 12 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Following (Zamir, 1998) and using only the monotonicity property for the Fisher Information Matrix, we have also a matrix form of the FII: At I2,g(AX)A ≤ I2,g(X) I2,g(AX) ≤ AI2,g(X)−1At −1 FII and case of equality - If (f,g) is a pair of escorts, With X = [U1 ...UM]t , Ui independents, A = [1...1], we have I2,q(∑ i Ui )−1 ≥ ∑ i I2,q(Ui )−1 Equality if the Ui are independent, ∑i Ui is a q-Gaussian with ∑i I2,q(Ui )−1 = ∑i Var(Ui ). 12 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Following (Zamir, 1998) and using only the monotonicity property for the Fisher Information Matrix, we have also a matrix form of the FII: At I2,g(AX)A ≤ I2,g(X) I2,g(AX) ≤ AI2,g(X)−1At −1 FII and case of equality - If (f,g) is a pair of escorts, With X = [U1 ...UM]t , Ui independents, A = [1...1], we have I2,q(∑ i Ui )−1 ≥ ∑ i I2,q(Ui )−1 Equality if the Ui are independent, ∑i Ui is a q-Gaussian with ∑i I2,q(Ui )−1 = ∑i Var(Ui ). → Follows from Cov(X) ≥ I2,q(X)−1 with equality if X is a q-Gaussian and AI2,q(X)−1At = ACov(X)At : then Cov(AX) ≥ I2,q(AX)−1 ≥ AI2,q(X)−1At = ACov(X)At . 12 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End The q-Entropy-qFisher-qGauss setting The classical Shannon-Fisher-Gauss setting can be extended to q-entropies, higher moments, and a generalized Fisher info (*) jfb - Some properties of generalized Fisher information in the context of nonextensive thermostatistics, Physica A, august 2013 13 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End The q-Entropy-qFisher-qGauss setting The classical Shannon-Fisher-Gauss setting can be extended to q-entropies, higher moments, and a generalized Fisher info 1 Maximum entropy: (Lutwak et al 2005) mα [f] 1 2 Nq[f] ≥ mα [G] 1 2 Nq[G] 2 Extended de Bruijn’s identity (nonlinear diffusion equation) (*) d dt Sq[f] = m q β−1 Mq[f]β Iβ,q[f] 3 Generalized Stam’s inequality (*) Iβ,q [f] 1 β Nq[f] 1 2 ≥ Iβ,q [G] 1 β Nq[G] 1 2 (*) jfb - Some properties of generalized Fisher information in the context of nonextensive thermostatistics, Physica A, august 2013 13 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End The q-setting (continued) 4 Generalized Cramér-Rao inequality Eg X α 1 α Iβ,q [g] 1 β ≥ n 5 Fisher Information Inequality I2,q(X +Y)−1 ≥ I2,q(X)−1 +I2,q(Y)−1 14 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Remarks, further and future work Different results in paper and presentation (uncertainty relations, matrix form, FII) 15 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Remarks, further and future work Different results in paper and presentation (uncertainty relations, matrix form, FII) We got new Cramér-Rao inequalities in the estimation setting 15 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Remarks, further and future work Different results in paper and presentation (uncertainty relations, matrix form, FII) We got new Cramér-Rao inequalities in the estimation setting Which also suggests new robust estimation methods 15 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Remarks, further and future work Different results in paper and presentation (uncertainty relations, matrix form, FII) We got new Cramér-Rao inequalities in the estimation setting Which also suggests new robust estimation methods It is possible to get even more general CR inequalities for Young cost functions 15 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities The Begin The ChiBeta Divergence GeneralizedCR FII The End Remarks, further and future work Different results in paper and presentation (uncertainty relations, matrix form, FII) We got new Cramér-Rao inequalities in the estimation setting Which also suggests new robust estimation methods It is possible to get even more general CR inequalities for Young cost functions Entropy Power Inequality Nq(X +Y) ≥ cq Nq(X)+Nq(Y) (q ≥ 1) has been proved very recently and the subject is very hot (Bobkov, S. G.; Chistyakov – submitted in may 2013, also Wang, L; Madiman, M. – july 2013) In the usual case de Bruijn+FII →EPI. No such proof here, but there are ongoing efforts 15 / 15 Some results on a χ-divergence, an extended Fisher information and generalized Cramér-Rao inequalities