SUBGRADIENTS OF DISTANCE FUNCTIONS AT OUT-OF-SET POINTS

Abstract

This paper deals with the classical distance function to closed sets and its extension to the case of set-valued mappings. It has been well recognized that the distance functions play a crucial role in many aspects of variational analysis, optimization, and their applications. One of the most remarkable properties of even the classical distance function is its intrinsic nonsmoothness, which requires the usage of generalized differential constructions for its study and applications. In this paper we present new results in theser directions using mostly the generalized differential constructions introduced earlier by the first author, as well as their recent modifications. We pay the main attention to studying subgradients of the distance functions in out-of-set points, which is essentially more involved in comparison with the in-set case. Most of the results obtained are new in both finite-dimensional and infinite-dimensional settings; some of them of provide essential improvements of known results even for convex sets.