Symmetry-Aware Transformer Training for Automated Planning

While transformers excel in many settings, their application in the field of automated planning is limited.

Prior work like PlanGPT, a state-of-the-art decoder-only transformer, struggles with extrapolation from easy to hard planning problems.

This in turn stems from problem symmetries: planning tasks can be represented with arbitrary variable names that carry no meaning beyond being identifiers.

This causes a combinatorial explosion of equivalent representations that pure transformers cannot efficiently learn from.

We propose a novel contrastive learning objective to make transformers symmetry-aware and thereby compensate for their lack of inductive bias.

Combining this with architectural improvements, we show that transformers can be efficiently trained for either plan-generation or heuristic-prediction.

Our results across multiple planning domains demonstrate that our symmetry-aware training effectively and efficiently addresses the limitations of PlanGPT.