Computing Boundaries in Local Mixture Models

28/10/2015
Publication GSI2015
OAI : oai:www.see.asso.fr:11784:14263

Résumé

Local mixture models give an inferentially tractable but still flexible alternative to general mixture models. Their parameter space naturally includes boundaries; near these the behaviour of the likelihood is not standard. This paper shows how convex and differential geometries help in characterising these boundaries. In particular the geometry of polytopes, ruled and developable surfaces is exploited to develop efficient inferential algorithms.

Computing Boundaries in Local Mixture Models

Collection

application/pdf Computing Boundaries in Local Mixture Models Vahed Maroufy, Paul Marriott

Média

Voir la vidéo

Métriques

157
4
3.43 Mo
 application/pdf
bitcache://117cb2ec12cd75dcaeb5e153e10b5777b6fa9bec

Licence

Creative Commons Attribution-ShareAlike 4.0 International

Sponsors

Organisateurs

logo_see.gif
logocampusparissaclay.png

Sponsors

entropy1-01.png
springer-logo.png
lncs_logo.png
Séminaire Léon Brillouin Logo
logothales.jpg
smai.png
logo_cnrs_2.jpg
gdr-isis.png
logo_gdr-mia.png
logo_x.jpeg
logo-lix.png
logorioniledefrance.jpg
isc-pif_logo.png
logo_telecom_paristech.png
csdcunitwinlogo.jpg
<resource  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                xmlns="http://datacite.org/schema/kernel-4"
                xsi:schemaLocation="http://datacite.org/schema/kernel-4 http://schema.datacite.org/meta/kernel-4/metadata.xsd">
        <identifier identifierType="DOI">10.23723/11784/14263</identifier><creators><creator><creatorName>Paul Marriott</creatorName></creator><creator><creatorName>Vahed Maroufy</creatorName></creator></creators><titles>
            <title>Computing Boundaries in Local Mixture Models</title></titles>
        <publisher>SEE</publisher>
        <publicationYear>2015</publicationYear>
        <resourceType resourceTypeGeneral="Text">Text</resourceType><subjects><subject>Computing boundaries</subject><subject>Computational information geometry</subject><subject>Embedded manifolds</subject><subject>Local mixture models</subject><subject>Polytopes</subject><subject>Ruled and developable surfaces</subject></subjects><dates>
	    <date dateType="Created">Sat 7 Nov 2015</date>
	    <date dateType="Updated">Wed 31 Aug 2016</date>
            <date dateType="Submitted">Tue 15 May 2018</date>
	</dates>
        <alternateIdentifiers>
	    <alternateIdentifier alternateIdentifierType="bitstream">117cb2ec12cd75dcaeb5e153e10b5777b6fa9bec</alternateIdentifier>
	</alternateIdentifiers>
        <formats>
	    <format>application/pdf</format>
	</formats>
	<version>24656</version>
        <descriptions>
            <description descriptionType="Abstract">
Local mixture models give an inferentially tractable but still flexible alternative to general mixture models. Their parameter space naturally includes boundaries; near these the behaviour of the likelihood is not standard. This paper shows how convex and differential geometries help in characterising these boundaries. In particular the geometry of polytopes, ruled and developable surfaces is exploited to develop efficient inferential algorithms.

</description>
        </descriptions>
    </resource>
.

Computing Boundaries in Local Mixture Models Computing Boundaries in Local Mixture Models Vahed Maroufy & Paul Marriott Department of Statistics and Actuarial Science University of Waterloo October 28 GSI 2015, Paris Computing Boundaries in Local Mixture Models Outline Outline 1 Influence of boundaries on parameter inference 2 Local mixture models (LMM) 3 Parameter space and boundaries Hard boundaries and Soft boundaries 4 Computing the boundaries for LMMs 5 Summary and future direction Computing Boundaries in Local Mixture Models Boundary influence When boundary exits: MLE does not exist =⇒ find the Extended MLE MLE exists, but does not satisfy the regular properties Examples Binomial distribution, logistic regression, contingency table, log-linear and graphical models Geyer (2009), Rinaldo et al. (2009), Anaya-Izquierdo et al. (2013) Computing boundary is a hard problem, Fukuda (2004) Many mathematical results in the literature polytope approximation, Boroczky and Fodor (2008), Barvinok (2013) smooth surface approximation, Batyrev (1992), Ghomi (2001, 2004) Computing Boundaries in Local Mixture Models Boundary influence When boundary exits: MLE does not exist =⇒ find the Extended MLE MLE exists, but does not satisfy the regular properties Examples Binomial distribution, logistic regression, contingency table, log-linear and graphical models Geyer (2009), Rinaldo et al. (2009), Anaya-Izquierdo et al. (2013) Computing boundary is a hard problem, Fukuda (2004) Many mathematical results in the literature polytope approximation, Boroczky and Fodor (2008), Barvinok (2013) smooth surface approximation, Batyrev (1992), Ghomi (2001, 2004) Computing Boundaries in Local Mixture Models Boundary influence When boundary exits: MLE does not exist =⇒ find the Extended MLE MLE exists, but does not satisfy the regular properties Examples Binomial distribution, logistic regression, contingency table, log-linear and graphical models Geyer (2009), Rinaldo et al. (2009), Anaya-Izquierdo et al. (2013) Computing boundary is a hard problem, Fukuda (2004) Many mathematical results in the literature polytope approximation, Boroczky and Fodor (2008), Barvinok (2013) smooth surface approximation, Batyrev (1992), Ghomi (2001, 2004) Computing Boundaries in Local Mixture Models Boundary influence When boundary exits: MLE does not exist =⇒ find the Extended MLE MLE exists, but does not satisfy the regular properties Examples Binomial distribution, logistic regression, contingency table, log-linear and graphical models Geyer (2009), Rinaldo et al. (2009), Anaya-Izquierdo et al. (2013) Computing boundary is a hard problem, Fukuda (2004) Many mathematical results in the literature polytope approximation, Boroczky and Fodor (2008), Barvinok (2013) smooth surface approximation, Batyrev (1992), Ghomi (2001, 2004) Computing Boundaries in Local Mixture Models LMMs Local Mixture Models Definition Marriott (2002) g(x; µ, λ) = f (x; µ) + k j=2 λj f (j) (x; µ), λ ∈ Λµ ⊂ Rk−1 Properties Anaya-Izquierdo and Marriott (2007) g is identifiable in all parameters and the parametrization (µ, λ) is orthogonal at λ = 0 The log likelihood function of g is a concave function of λ at a fixed µ0 Λµ is convex Approximate continuous mixture models when mixing is “small” M f (x, µ) dQ(µ) Family of LMMs is richer that Family of mixtures Computing Boundaries in Local Mixture Models LMMs Local Mixture Models Definition Marriott (2002) g(x; µ, λ) = f (x; µ) + k j=2 λj f (j) (x; µ), λ ∈ Λµ ⊂ Rk−1 Properties Anaya-Izquierdo and Marriott (2007) g is identifiable in all parameters and the parametrization (µ, λ) is orthogonal at λ = 0 The log likelihood function of g is a concave function of λ at a fixed µ0 Λµ is convex Approximate continuous mixture models when mixing is “small” M f (x, µ) dQ(µ) Family of LMMs is richer that Family of mixtures Computing Boundaries in Local Mixture Models Example and Motivation Example LMM of Normal f (x; µ) = φ(x; µ, σ2 ), (σ2 is known). g(x; µ, λ) = φ(x; µ, σ2 ) 1 + k j=2 λj pj (x) , λ ∈ Λµ pj (x) polynomial of degree j. Why we care about λ and Λµ? They are interpretable    µ (2) g = σ2 + 2λ2 µ (3) g = 6λ3 µ (4) g = µ (4) φ + 12σ2 λ2 + 24λ4 (1) λ represents the mixing distribution Q via its moments in M f (x, µ) dQ(µ) Computing Boundaries in Local Mixture Models Example and Motivation Example LMM of Normal f (x; µ) = φ(x; µ, σ2 ), (σ2 is known). g(x; µ, λ) = φ(x; µ, σ2 ) 1 + k j=2 λj pj (x) , λ ∈ Λµ pj (x) polynomial of degree j. Why we care about λ and Λµ? They are interpretable    µ (2) g = σ2 + 2λ2 µ (3) g = 6λ3 µ (4) g = µ (4) φ + 12σ2 λ2 + 24λ4 (1) λ represents the mixing distribution Q via its moments in M f (x, µ) dQ(µ) Computing Boundaries in Local Mixture Models Example and Motivation The costs for all these good properties and flexibility are Hard boundary =⇒ Positivity (boundary of Λµ) Soft boundary =⇒ Mixture behavior We compute them for two models here: Poisson and Normal We fix k = 4 Computing Boundaries in Local Mixture Models Boundaries Hard boundary Λµ = λ | 1 + k j=2 λj qj (x; µ) ≥ 0, ∀x ∈ S , Λµ is intersection of half-spaces so convex Hard boundary is constructed by a set of (hyper-)planes Soft boundary Definition For a density function f (x; µ) with k finite moments let, Mk (f ) := (Ef (X), Ef (X2 ), · · · , Ef (Xk )). and for compact M define C = convhull{Mr (f )|µ ∈ M} Then, the boundary of C is called the soft boundary. Computing Boundaries in Local Mixture Models Boundaries Hard boundary Λµ = λ | 1 + k j=2 λj qj (x; µ) ≥ 0, ∀x ∈ S , Λµ is intersection of half-spaces so convex Hard boundary is constructed by a set of (hyper-)planes Soft boundary Definition For a density function f (x; µ) with k finite moments let, Mk (f ) := (Ef (X), Ef (X2 ), · · · , Ef (Xk )). and for compact M define C = convhull{Mr (f )|µ ∈ M} Then, the boundary of C is called the soft boundary. Computing Boundaries in Local Mixture Models Computing hard boundary Poisson model Λµ = λ | A2(x) λ2 + A3(x)λ3 + A4(x) λ4 + 1 ≥ 0, ∀x ∈ Z+ , Figure : Left: slice through λ2 = −0.1; Right: slice through λ3 = 0.3. Theorem For a LMM of a Poisson distribution, for each µ, the space Λµ can be arbitrarily well approximated, as measured by volume for example, by a finite polytope. Computing Boundaries in Local Mixture Models Computing hard boundary Poisson model Λµ = λ | A2(x) λ2 + A3(x)λ3 + A4(x) λ4 + 1 ≥ 0, ∀x ∈ Z+ , Figure : Left: slice through λ2 = −0.1; Right: slice through λ3 = 0.3. Theorem For a LMM of a Poisson distribution, for each µ, the space Λµ can be arbitrarily well approximated, as measured by volume for example, by a finite polytope. Computing Boundaries in Local Mixture Models Computing hard boundary Normal model let y = x−µ σ2 Λµ = {λ | (y2 − 1)λ2 + (y3 − 3y)λ3 + (y4 − 6y2 + 3)λ4 + 1 ≥ 0, ∀y ∈ R}. We need a more geometric tools to compute this boundary. Computing Boundaries in Local Mixture Models Ruled and developable surfaces Ruled and developable surfaces Definition Ruled surface: Γ(x, γ) = α(x) + γ · β(x), x ∈ I ⊂ R, γ ∈ Rk Developable surface: β(x), α (x) and β (x) are coplanar for all x ∈ I. Computing Boundaries in Local Mixture Models Ruled and developable surfaces Definition The family of planes, A = {λ ∈ R3 | a(x) · λ + d(x) = 0, x ∈ R}, each determined by an x ∈ R, is called a one-parameter infinite family of planes. Each element of the set {λ ∈ R3 |a(x) · λ + d(x) = 0, a (x) · λ + d (x) = 0, x ∈ R} is called a characteristic line of the surface at x and the union is called the envelope of the family. A characteristic line is the intersection of two consecutive planes The envelope is a developable surface Computing Boundaries in Local Mixture Models Ruled and developable surfaces Boundaries for Normal LMM Hard boundary of for Normal LMM (y2 − 1)λ2 + (y3 − 3y)λ3 + (y4 − 6y2 + 3)λ4 + 1 = 0, ∀y ∈ R . λ2 λ3 λ4 λ4 λ3 λ2 Figure : Left: The hard boundary for the normal LMM (shaded) as a subset of a self intersecting ruled surface (unshaded); Right: slice through λ4 = 0.2. Computing Boundaries in Local Mixture Models Ruled and developable surfaces Boundaries for Normal LMM Soft boundary of for Normal LMM recap : Mk (f ) := (Ef (X), Ef (X2 ), · · · , Ef (Xk )). For visualization purposes let k = 3, (µ ∈ M, fix σ) M3(f ) = (µ, µ2 + σ2 , µ3 + 3µσ2 ), M3(g) = (µ, µ2 + σ2 + 2λ2, µ3 + 3µσ2 + 6µλ2 + 6λ3). Figure : the 3-D curve ϕ(µ); Middle: the bounding ruled surface γa(µ, u); Right: the convex subspace restricted to soft boundary. Computing Boundaries in Local Mixture Models Ruled and developable surfaces Boundaries for Normal LMM Ruled surface parametrization Two boundary surfaces, each constructed by a curve and a set of lines attached to it. γa(µ, u) = ϕ(µ) + u La(µ) γb(µ, u) = ϕ(µ) + u Lb(µ) where for M = [a, b] and ϕ(µ) = M3(f ) La(µ): lines between ϕ(a) and ϕ(µ) Lb(µ): lines between ϕ(µ) and ϕ(b) Computing Boundaries in Local Mixture Models Summary Summary Understanding these boundaries is important if we want to exploit the nice statistical properties of LMM The boundaries described in this paper have both discrete aspects and smooth aspects The two example discussed represent the structure for almost all exponential family models It is a interesting problem to design optimization algorithms on these boundaries for finding boundary maximizers of likelihood Computing Boundaries in Local Mixture Models References Anaya-Izquierdo, K., Critchley, F., and Marriott, P. (2013). when are first order asymptotics adequate? a diagnostic. Stat, 3(1):17–22. Anaya-Izquierdo, K. and Marriott, P. (2007). Local mixture models of exponential families. Bernoulli, 13:623–640. Barvinok, A. (2013). Thrifty approximations of convex bodies by polytopes. International Mathematics Research Notices, rnt078. Batyrev, V. V. (1992). Toric varieties and smooth convex approximations of a polytope. RIMS Kokyuroku, 776:20. Boroczky, K. and Fodor, F. (2008). Approximating 3-dimensional convex bodies by polytopes with a restricted number of edges. Contributions to Algebra and Geometry, 49(1):177–193. Fukuda, K. (2004). From the zonotope construction to the minkowski addition of convex polytopes. Journal of Symbolic Computation, 38(4):1261–1272. Geyer, C. J. (2009). Likelihood inference in exponential familes and direction of recession. Electronic Journal of Statistics, 3:259–289. Ghomi, M. (2001). Strictly convex submanifolds and hypersurfaces of positive curvature. Journal of Differential Geometry, 57(2):239–271. Ghomi, M. (2004). Optimal smoothing for convex polytopes. Bulletin of the London Mathematical Society, 36(4):483–492. Marriott, P. (2002). On the local geometry of mixture models. Biometrika, 89:77–93. Rinaldo, A., Fienberg, S. E., and Zhou, Y. (2009). On the geometry of discrete exponential families with application to exponential random graph models. Electronic Journal of Statistics, 3:446–484. Computing Boundaries in Local Mixture Models END Thank You