Sparse recovery in convex hulls via entropy penalization

Abstract

Let (X, Y) be a random couple in S×T with unknown distribution P and (X₁, Y₁), …, (X_n, Y_n) be i.i.d. copies of (X, Y). Denote P_n the empirical distribution of (X₁, Y₁), …, (X_n, Y_n). Let h₁, …, h_N: S↦[−1, 1] be a dictionary that consists of N functions. For λ∈ℝ^N, denote f_λ:=∑_j=1^Nλ_jh_j. Let ℓ: T×ℝ↦ℝ be a given loss function and suppose it is convex with respect to the second variable. Let (ℓ•f)(x, y):=ℓ(y; f(x)). Finally, let Λ⊂ℝ^N be the simplex of all probability distributions on {1, …, N}. Consider the following penalized empirical risk minimization problem $$\begin{eqnarray*}\hat{\lambda}^{\varepsilon}:={\mathop{\textrm{argmin}}_{\lambda\in \Lambda}}\Biggl[P_{n}(\ell \bullet f_{\lambda})+\varepsilon \sum_{j=1}^{N}\lambda_{j}\log \lambda_{j}\Biggr]\end{eqnarray*} $$ along with its distribution dependent version $$\begin{eqnarray*}\lambda^{\varepsilon}:={\mathop{\textrm{argmin}}_{\lambda\in \Lambda}}\Biggl[P(\ell \bullet f_{\lambda})+\varepsilon \sum_{j=1}^{N}\lambda_{j}\log \lambda_{j}\Biggr],\end{eqnarray*}$$ where ɛ≥0 is a regularization parameter. It is proved that the “approximate sparsity” of λ^ɛ implies the “approximate sparsity” of λ̂^ɛ and the impact of “sparsity” on bounding the excess risk of the empirical solution is explored. Similar results are also discussed in the case of entropy penalized density estimation.