Wednesday, June 28, 2023, 12pm

This thesis studies algorithms for teaching autonomous agents to complete tasks through trial and error learning. Typically, this problem is posed as a reinforcement learning (RL) problem, wherein agents attempt to maximize a user-provided reward function. The algorithms studied here take a different approach, largely eschewing the reward function and instead learning to achieve desired outcomes directly from data. This approach allows users to employ algorithmic tools from the supervised and unsupervised learning, while also surfacing an interface that allows non-expert users to teach agents new tasks.

The main challenge in the design of these methods is predicting the probability of desired outcomes, especially when the outcomes only occur hundreds of steps into the future, and especially when using off-policy data. To this end, the first part of this thesis develops an algorithm based on recursive classification that estimates the probability of future states via a temporal difference update. This method is directly applicable to environments with continuous states and actions, does not require any hand-crafted distance metrics, and leads to an algorithm for goal-conditioned RL that outperforms prior methods. We then generalize this idea to tasks that can be solved in many ways, allowing more flexible task specification and providing broader generalization capabilities.

While framing control problems in terms of desired outcomes provides an easy mechanism to specify what the task is, it leaves no room for specifying how the task should be solved, raising the question of whether these methods are restricted to simple tasks. To lift this limitation, we consider inferring the structure of solutions to complex tasks. Because the algorithms introduced in the first part are probabilistic in nature, it is easy to incorporate this structure as an unobserved latent variable. These new algorithms infer this task structure; in doing so, they decompose the control problem into a series of easier problems, thereby accelerating learning. We first discuss the goal-conditioned setting, where this inferential perspective leads to a simple and theoretically justified method for integrating goal-conditioned RL into classical planning pipelines. This approach substantially outperforms standard goal-conditioned RL algorithms. We then consider a different way of structuring the task solution: as a composition of a learned dynamics model and policy. The result is an algorithm for model-based RL where the model and policy are jointly optimized using the same objective, which is a lower bound on expected returns.

Thesis Committee:
Ruslan Salakhutdinov (Chair)
Jeff Schneider
Leslie Kaelbling (Massachusetts Institute of Technology)
Sergey Levine (University of California, Berkeley)

Additional Information

In Person and Zoom Participation. See announcement.

Event Type: Thesis Orals
Room Number: In Person and Virtual - ET
Building: Reddy Conference Room, Gates Hillman 4405 and Zoom
Speaker's Name: BENJAMIN EYSENBACH
Speaker Website: ben-eysenbach.github.io
Speaker's Professional Title: Ph.D. Candidate, Machine Learning Department, Carnegie Mellon University
Talk Title: Probabilistic Reinforcement Learning: Using Data to Define Desired Outcomes, and Inferring How to Get There
For More Information: stidle@andrew.cmu.edu
Affiliations: Machine Learning Department (MLD)
Organization(s): SCS