Distances for Markov chains from sample streams

This document presents a novel approach for estimating the similarity between Markov chains using only sampled data, without requiring full knowledge of their transition probabilities. The authors leverage the recent finding that bisimulation metrics, a tool for quantifying stochastic process similarity, are equivalent to optimal transport distances. They reformulate the problem as a linear program and propose a stochastic primal-dual optimization algorithm (SOMCOT) to solve it based on sampled state transitions. Theoretical analysis provides sample complexity guarantees for their method, and empirical results demonstrate its effectiveness for tasks like representation learning and model selection in various environments. The work highlights the potential of this approach for addressing real-world scenarios in areas like machine learning where complete system dynamics are often unknown.