Efficient Bayes-Adaptive Reinforcement Learning with Temporal Logic Specifications

Researchers posted arXiv:2609.20954v1, presenting an end-to-end model-based reinforcement learning algorithm that synchronizes a Limit-Deterministic Büchi Automaton representation of Linear Temporal Logic specifications with a Bayes-Adaptive Markov Decision Process environment representation. The work introduces a Bayes-Adaptive Monte-Carlo Planning (BAMCP) algorithm for approximate Bayes-optimal strategy synthesis in the synchronized construct, and finite- and infinite-horizon task experiments plus ablation studies show improved property satisfaction and sample efficiency over traditional model-free approaches, including reduced task violations in cautious RL.

arXiv:2609.20954v1 Announce Type: new Abstract: We present a novel end-to-end model-based Reinforcement Learning RL algorithm for efficient policy synthesis under given Linear Temporal Logic LTL specifications e.g., safety or reachability in unknown environments. To do so, a Limit-Deterministic B{\"u}chi Automaton LDBA representation of the LTL task is synchronised with a Bayes-Adaptive Markov Decision Process BAMDP representation of the environment, which allows us to leverage an enhanced exploration-exploitation trade-off that is achieved via Bayesian RL, as opposed to traditional non-Bayesian approaches. We further propose a novel Bayes-Adaptive Monte-Carlo Planning BAMCP algorithm to allow for approximate Bayes-optimal strategy synthesis in the synchronised BAMDP construct. A range of finite- and infinite-horizon task experiments demonstrate the effectiveness of our approach in terms of both property satisfaction and sample efficiency, when compared to traditional model-free approaches. Additional ablation studies also successfully highlight the value of the novel BAMCP algorithm in comparison to classical BAMCP for LTL task satisfaction. Finally, we also showcase a successful application of our approach for \textit{cautious} RL, namely to reduce the number of task violations incurred during policy training.