Kernel Metrics and Learning for Borel MDPs: Identifiability and Adaptive Control

We consider a Markov decision process with standard Borel spaces and an unknown transition kernel under the average cost criterion.

We do not impose any parametrization on the set of possible kernels.

To facilitate our analysis, we first develop implication relations between several topologies on kernels defined by pointwise, continuous, or uniform weak convergence; we then review robustness properties on the space of kernels, and finally we establish compactness conditions on the space of kernels.

Building on this regularity analysis, we then present two data-driven identifiability results; the first one being Bayesian and the second one empirical.

Our conditions for the Bayesian setting are significantly more relaxed compared with prior work which considered either finite or parametric models, though we do not obtain a rate of convergence.

Our analysis is asymptotic and builds on measurability in terms of the tail $\sigma$-field of the available information.

Identifiability results are then used to design near-optimal adaptive control policies which alternate between periods of exploration, where the controller acts according to a policy which is conducive to the identification of the true kernel, and periods of exploitation where the controller's information on the kernel is utilized.

We will establish that such policies are near optimal.

In summary, our contribution is with regard to the general standard Borel setup where there is no apriori parametric representation.