A drive to Taco Bell

The Promise

Earlier this year, we promised to drive to Taco Bell without a disengagement, on a stock car, with open-source code, on a comma three. Over the last few weeks, we’ve developed a version of openpilot that can do that. Today, we’re releasing a video of one of those drives, the logs, and the code with a full write-up below.

We will be driven to Taco Bell:

• without a disengagement
• on a comma three
• in a stock, production, car
• using open source code
• by the end of the year#level2 #adas #leftturn #rightturn #stoplight #fsd pic.twitter.com/5VpXjchhM5

— comma (@comma_ai) February 19, 2022

Why Taco Bell?

At comma, we want to create a superhuman driving agent. In the simplest terms, a self-driving system needs to control the steering wheel, control the gas and brake pedals, and navigate you to your destination. We find these three parts to be a good way to split up the problem, and over the last few years, we have been working on those three parts in that order. The Taco Bell drive serves as the first demo where all three of these parts come together.

• End-to-end lateral planning: shipped behind a toggle in openpilot 0.8.3, and then became the default in openpilot 0.8.15
• End-to-end longitudinal planning: shipped behind a toggle in openpilot 0.9
• Navigate-on-openpilot: where openpilot drives to a destination you enter; still in development branches as we prepare to ship it

Where we started

At the time of the tweet, openpilot 0.8.13 had just shipped; end-to-end lateral planning was still behind a toggle, and the other two components hadn’t shipped in any form. An openpilot 0.8.13 drive on the Taco Bell route required nearly a dozen user inputs and disengagements for adjusting the set speed, stopping for red lights, going too fast around turns, changing lanes, and taking the appropriate on-ramps and exits.

The Drive

The Diff

The Taco Bell drive was completed on the taco2 branch. It’s based on openpilot master plus a net +54 line diff, including a new driving model.

At comma, we constantly ask ourselves “is what we’re working on going to end up in the final system?” As a result of this focus, we generally avoid writing hacks, and this holds true for the Taco Bell drive too. Aside from the navigation-initiated lane changes, we plan to ship everything in this branch in a similar form with more refinement and validation.

EV6 blinkers and low speed torque

This commit makes a few car-specific changes for our taco car, the Kia EV6:

• increase steering actuation limits at low speeds
• set a flag to enable controlling the blinkers
• use the stalk position to read blinker state instead of the lamp to avoid reading back openpilot-commanded blinkers

openpilot’s actuation limits are set to enforce the safety model. For steering, we want to limit how much lateral acceleration and jerk can be achieved, since those quantities best represent how quickly the car can change direction. For cars with a torque-based steering interface, like the EV6, openpilot enforces a single set of limits on the torque command. Since the amount of lateral acceleration achieved for a given torque depends on the speed, this approach trades off some performance for implementation simplicity, i.e. maximum lateral acceleration can only be achieved in a narrow speed range.

To improve low speed turns, we implemented a speed-based torque limit that allows the car to achieve lateral accelerations and jerks closer to the limit at low speeds. We eventually want to roll out increased torque limits on some cars, but it’ll require a good understanding of how much lateral acceleration every car can achieve for every torque value at every speed.

Driving model with navigate-on-openpilot and driving style

Taking exits, on-ramps, turns, etc. requires navigation knowledge. This commit includes a first version of the “nav model,” which takes in a picture of the map with a navigation path and outputs a vector which goes into the driving model policy. This vector plays a similar role to the “desire” vector that we use in the shipped driving model, but it’s much richer in context.

The nav model is an 8-bit quantized EfficientNet B0 that runs on the comma three’s DSP. The architecture can be visualized using netron.

We also experimented with ways to control how “chill” the model drives. To do so, we classified some drivers in our fleet based on their driving style, defined by their lateral and longitudinal acceleration profiles. This commit contains a new driving model that takes desired driving style as an input. The taco branch takes advantage of this by setting the driving style to the most chill setting, which results in plans that are more achievable with openpilot’s lateral and longitudinal actuation limits.

Auto lane change initiation

openpilot’s driving model has been able to lane change when the user turns the blinker on since openpilot 0.7.1. This commit adds some logic to enable openpilot to initiate its own lane changes with the nav instructions.

Changing lanes into and out of exit-only lanes requires long distance understanding of nav instructions and lane information. Since the nav model currently lacks that input, we used some simple heuristics around the nav instructions to tell the model when to lane change:

• After passing an on-ramp, lane change once to avoid a potential exit-only lane.
• When approaching an exit, lane change in the direction of the oncoming exit until no adjacent lanes are visible.

These two heuristics are undeniably not robust to the variety of exits and highways in the field. In the future, we plan to give the nav model enough context to suggest lane changes coming up to an exit.

For cars with blind-spot monitors, openpilot already blocks lane changes if a car is detected in the driver’s blind spot. Currently, we rely on those sensors to determine if a lane change is safe. In the future, we plan to complement the blind spot monitors with a model trained on comma10k, which recently included images from the comma three’s front-facing camera.

Lateral planner retune and slow down for turns

In this commit, we make some small changes to the MPC planning algorithms. The lateral planner gets a small cost retune so that it moves the steering wheel more at lower speeds. Since cars that have poor lateral tuning can have many jerky steering wheel movements with this change, we plan to ship this change once lateral control is well-tuned on all supported cars.

Additionally, we make the longitudinal planner slow down when approaching a sharp turn. The driving model already plans to slow down for sharp turns, but at low speeds, openpilot does not have sufficient torque to take 90° turns like a human would, so additional slowdown on top of the end-to-end policy is needed to have reliable turning behavior. We plan to ship this when we have a good understanding of the low speed torque limitations of cars.

We’re hiring

Like what you see here? We’re hiring!