Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment Newsletter #29, published by Rohin Shah on the AI Alignment Forum. Highlights Deep Imitative Models for Flexible Inference, Planning, and Control (Nicholas Rhinehart et al): It's hard to apply deep RL techniques to autonomous driving, because we can't simply collect a large amount of experience with collisions in order to learn. However, imitation learning is also hard, because as soon as your car deviates from the expert trajectories that you are imitating, you are out of distribution, and you could make more mistakes, leading to accumulating errors until you crash. Instead, we can model the expert's behavior, so that we can tell when we are moving out of distribution, and take corrective action. They split up the problem into three different stages. First, they generate a set of waypoints along the path to be followed, which are about 20m away from each other, by using A search on a map. Next, they use model-based planning using an imitative model to generate a plan (sequence of states) that would take the car to the next waypoint. Finally, they use a simple PID controller to choose low-level actions that keep the car on target towards the next state in the plan. The key technical contribution is with the imitative model, which is a probabilistic model P(s_{1:T}, G, φ), where φ is the current observation (eg. LIDAR), s_{1:T} is the planned trajectory, and G is a goal. We can learn P(s_{1:T} | φ) from expert demonstrations. The goal G can be anything for which you can write down a specification P(G | s_{1:T}, φ). For example, if you simply want to reach a waypoint, you can use the normal distribution on the distance between the final state s_T and the waypoint. You can also incorporate a hand-designed cost on each state. They evaluate in simulation on a static world (so no pedestrians, for example). They show decent transfer from one map to a second map, and also that they can avoid artificially introduced potholes at test time (despite not seeing them at training time), simply by adding a cost on states over a pothole (which they can take into account because they are performing model-based planning). Rohin's opinion: I really like this paper, it showcases the benefits of both model-based planning and imitation learning. Since the problem has been decomposed into a predictive model, a goal G, and a planner, we can edit G directly to get new behavior at test time without any retraining (as they demonstrate with the pothole experiment). At the same time, they can get away with not specifying a full reward function, as many features of good driving, like passenger comfort and staying in the correct lane, are learned simply by imitating an expert. That said, they initially state that one of their goals is to learn from offline data, even though offline data typically has no examples of crashes, and "A model ignorant to the possibility of a crash cannot know how to prevent it". I think the idea is that you never get into a situation where you could get in a crash, because you never deviate from expert behavior since that would have low P(s_{1:T} | φ). This is better than model-based planning on offline data, which would consider actions that lead to a crash and have no idea what would happen, outputting garbage. However, it still seems that situations could arise where a crash is imminent, which don't arise much (if at all) in the training data, and the car fails to swerve or brake hard, because it hasn't seen enough data. Interpretability and Post-Rationalization (Vincent Vanhoucke): Neuroscience suggests that most explanations that we humans give for a decision are post-hoc rationalizations, and don't reflect the messy underlying true reasons for the decision. It turns out that decision making, perception, and all the other tasks we're hoping to outsource to...