A Survey of Learn-to-Compute Paradigms for Rate-Distortion-Type Problems

Rate-distortion (RD) theory and its related formulations play a central role in understanding efficient information representation, but computing these quantities remains challenging in high-dimensional settings.

Classical iterative methods such as the Blahut-Arimoto algorithm become impractical in high-dimensional domains due to the curse of dimensionality and the intractability of mutual-information terms.

Recent advances in neural modeling and differentiable optimization offer a promising alternative through a learn-to-compute paradigm, in which probability distributions and objective functionals are represented by flexible neural parameterizations.

This survey presents an overview of neural approaches for evaluating the RD-type objectives.

We present three representative families of methods: variational inference, neural mutual-information estimation, and dual-form optimization.

By reviewing their theoretical principles, algorithmic techniques, and consistency properties, we elucidate how these methods collectively transform classical RD-type problems into scalable differentiable objectives suitable for deep learning, though challenges remain in large-scale applications.

Together, these perspectives offer promising avenues for scaling information-theoretic computation to complex, high-dimensional machine learning systems.