How does openpilot work?
openpilot is an open source driving agent. It is capable of controlling the gas, brake, and steering on certain cars, reaching up to 6 minutes with no user action required (besides paying attention!). Let’s talk about how it works.
This post will probably be most useful for those wanting to get into openpilot porting and development. If you are interested in comma.ai in general, check out Our Road to Self Driving Victory
How do you talk to a car?
Most all modern cars use several CAN buses to link the various modules of the car together. One CAN bus is exposed on the OBD2 port, others can be found hidden behind panels.
openpilot can use either a NEO or a panda as its CAN interface. Since the platform is open, it would be easy to add support for the OpenXC or Kvaser or the CANBus Triple.
On the Honda Civic and ILX, the first two cars supported by openpilot, all communication happens over 2 CAN buses, one vehicle CAN, and one radar CAN. Other cars may be different.
What language does the car speak?
CAN is a simple protocol. It’s a bus, where any device can send a message to all others. A message contains an identifier, 11-bits long in standard CAN, 29-bits long in extended, and a message, which can be up to 8 bytes long.
The identifier determines how to parse the message. DBC files are the standard way of specifying this parsing. Here is a snippet from our Honda Civic DBC showing the parsing of a steering control packet.
BO_ 228 STEERING_CONTROL: 5 ADAS
SG_ STEER_TORQUE : 7|16@0- (1,0) [-3840|3840] “” EPS
SG_ STEER_TORQUE_REQUEST : 23|1@0+ (1,0) [0|1] “” EPS
SG_ CHECKSUM : 39|4@0+ (1,0) [0|15] “” EPS
SG_ COUNTER : 33|2@0+ (1,0) [0|3] “” EPS
The first line says it’s a STEERING_CONTROL packet with message identifier is 228, or 0xE4. The next four show how four fields are bitpacked into the 5 byte message.
Our code to parse CAN messages is in can_parser.py, which uses our implementation of DBC in dbc.py.
How do I steer to a position?
If you notice, the packet doesn’t specify a position you want the wheel to turn to, instead it defines how much torque to put on the wheel. At the lowest levels, this is how motors are controlled. In order to command the wheel to go to a position, we need to use some controls to close the loop. The steering angle is available on the CAN bus as well.
BO_ 330 STEERING_SENSORS: 8 EPS
SG_ STEER_ANGLE : 7|16@0- (-0.1,0) [-500|500] “deg” NEOSo now we have a desired angle, current angle, and torque command. The perfect application for a PID loop. Since torques are small, we only use a PI loop.
You’ll find our steering PI loop implementation in latcontrol.py.
ROS 2.0
Let’s take a step back and talk about the architecture of openpilot.
When you look at it, it’s very similar to ROS. But I didn’t like how large ROS was, or the fact that they have a custom message passing protocol and custom typing system.
We use ZMQ publish subscribe for message passing, and cap’n proto (very similar to Protocol Buffers) for typing. Funny enough, that’s what they have planned for ROS 2.0
A Car Abstraction Layer
We want openpilot to be able to talk to many different cars. Our car abstraction layer is defined in car.capnp. When you implement a car interface, you publish a CarState and accept a CarControl.
Adaptive Cruise Control (longitudinal)
We separate the gas/brakes from the steering in openpilot. Longitudinal control is still done in an old school way, without a neural network.
radard is used to parse the messages from the car radar. It does a rudimentary fusion with the vision system, and outputs the positions of up to two “lead cars” at 20hz.
Then adaptivecruise uses the leads and the current speed to determine how fast you should be going.
A Vision System (lateral)
visiond runs the driving neural network. This part of openpilot is closed source for business model reasons. But its API is open.
struct ModelData {
frameId @0 :UInt32;
path @1 :PathData;
leftLane @2 :PathData;
rightLane @3 :PathData;
lead @4 :LeadData;
…It outputs a best guess at the path, where it thinks the left and right lanes are, and where it thinks the lead car is.
pathplanner.py merges these paths together into the path the car should follow. In latcontrol.py, a point is picked along this path as the target point for the car to aim for. Then it follows the arc to that point. There is a lot of room for improvement here with a more sophisticated control strategy.
Putting it all together
manager.py is responsible for starting and stopping the constellation of processes when appropriate. It has two states, car stopped and car started, and runs different processes depending on what state it’s in. See service_list.yaml for a list.
boardd, sensord, and visiond all talk to the outside world. loggerd logs all the data to use for machine learning. plannerd tells the car where to go.
controlsd is the main process that talks to the car. It is started when the car starts. The controlsd_thread function would probably be a good place to start reading the code, it’s the main 100hz control loop.
How you can help
- There’s undoubtedly much more to openpilot than is described here. Dive into the code. If you are feeling really nice, document!
- comma.ai offers bounties for new car ports. Port it to your car!
- If you want to experiment, openpilot runs in simulation on a PC as well. Help out by refactoring and writing tests.
- Contribute good stuff, and we’ll bring you in for an interview. We are particularly looking for an openpilot team lead to handle releases, testing, and documentation.
