Integral formula of the unitary inversion operator for the minimal representation of $O(p, q)$

Abstract

The indefinite orthogonal group $G = O(p,q)$ has a distinguished infinite dimensional unitary representation $\pi$, called the \textit{minimal representation} for $p+q$ even and greater than $6$. The \textit{Schr\"odinger model} realizes $\pi$ on a very simple Hilbert space, namely, $L^2(C)$ consisting of square integrable functions on a Lagrangean submanifold $C$ of the minimal nilpotent coadjoint orbit, whereas the $G$-action on $L^2(C)$ has not been well-understood. This paper gives an explicit formula of the unitary operator $\pi(w_0)$ on $L^2(C)$ for the `conformal inversion' $w_0$ as an integro-differential operator, whose kernel function is given by a Bessel distribution. Our main theorem generalizes the classic Schr\"odinger model on $L^2(\mathbf R^n)$ of the Weil representation, and leads us to an explicit formula of the action of the whole group $O(p,q)$ on $L^2(C)$. As its corollaries, we also find a representation theoretic proof of the inversion formula and the Plancherel formula for Meijer's $G$-transforms.