A better openpilot, across the boardPermalink

This release brings improvements throughout the openpilot experience, on and off the road.

openpilot 0.11.2 introduces our first chestnut-class driving models, better driver monitoring, Volkswagen ID.4 support as a gold car, new sounds and UI improvements, and new comma connect features such as livestreaming and shareable clips.

chestnut-class driving modelsPermalink

This is our first release with chestnut-class driving models. The new model has about 15× as many parameters as the small model in 0.11.1, and uses about 55× as much computation.

We pair this increase in capacity with a larger world model and a switch to reinforcement learning training. Together, these changes are a big step forward in driving quality and performance, especially in challenging scenarios.

convnext backbonePermalink

We based our initial vision backbone search on the ImageNetV2 Matched Frequency results. ConvNeXt is among the strongest convolutional architectures in that comparison and gives us a straightforward way to scale model capacity by increasing the number of channels and blocks. It also lets us initialize the vision encoder from pretrained weights, adapt them to our camera inputs, and fine-tune on driving data.

We also increased the number of training segments from 500k to 2.5M for the vision backbone (off-policy model).

All these improvements translate to a significant reduction in driving losses.

And better driving losses translate to better driving performance, especially in challenging scenarios.

4B parameter world modelPermalink

-    n_layer = 34,
-    n_head = 20,
-    n_embd = 1280,
-    n_segments = 2.5M,
+    n_layer = 56,
+    n_head = 36,
+    n_embd = 2304,
+    n_segments = 10.0M,

We also scaled up the world model training and enjoyed a very predictable improvement in video generation quality. We fit a scaling law similar to Kaplan et al. 2020 to predict the performance of the full scale run using smaller scale experiments.

The fit $12.65 \cdot F^{-0.063}$ was produced BEFORE the full scale run and predicted a loss very close to the achieved loss.

New desire stackPermalink

We also updated the desire labeling stack to use offline map data and a simpler lane change detection algorithm #39047. Doing this improved recall across the board especially for turns, exits, and complex lane changes.

Reinforcement Learning post trainingPermalink

Previous models learned to imitate the actions suggested by the world model at each step during simulation episodes. The new models are trained with a reinforcement learning (RL) objective, learning which actions produce the highest cumulative reward in the episode.

Imitation learning gives the policy a target action at each step, but matching that action does not necessarily produce the desired motion long term. RL evaluates the consequences of the policy’s actions over time. This lets the policy learn corrections that account for future outcomes. The world model still supplies the driving reference, but the new objective allows the policy to deviate from it when it is advantageous to do so in the long term.

Reward and objectivePermalink

At each step, the reward compares the car’s achieved lateral and longitudinal acceleration with the accelerations predicted from the world model an earlier step. We additionally use smoothness terms in the actor loss to penalize abrupt changes in steering and acceleration.

Residual policy and criticsPermalink

We use the residual learning approach from ResFiT. The pre-trained base policy $\pi_{\mathrm{base}}$ is frozen, and a separate residual temporal policy $\pi_{\mathrm{res}}$ learns a correction. If $o_t$ denotes the policy’s observation history the commanded lateral and longitudinal accelerations $a_t$ are:

The training algorithm builds on TD3. Each critic learns from a five-step return, reducing the number of bootstrapping steps through which value-estimation errors can accumulate, as discussed in Q-learning is not yet scalable.

Full frame copyPermalink

We now copy the full frames from both road cameras over USB to the chestnut and perform the image transformation and model work there #38684. Moving the preprocessing onto the external GPU prepares the input pipeline for full-frame driving models in future releases.

Pre-compiled models on development branchesPermalink

We now ship the big driving model precompiled on development branches as well #38930. Updating the model no longer requires compiling the driving network on the device, making it faster to test new models in cars.

ONNX versions of the models will be available at hf.co/commaai/openpilot_driving_models.

chestnutPermalink

We are introducing chestnut, a PCIe-to-USB bridge that brings desktop-class GPUs to comma four and closes the compute gap to Tesla.

It provides a PCIe x16 slot with four PCIe lanes, a USB3 or USB4 connection to the host, and a 12V GPU power stage. On comma four, it connects over USB3 through the AUX port. Power is supplied over an XT60 connector from the car’s cigarette adapter or a standard ATX power supply.

chestnut is built around the ASM2464PD, a USB4 to PCIe bridge chip. We replaced the stock firmware with our own custom reverse-engineered firmware for the chip’s 8051 core.

The firmware exposes PCIe over USB and uses the bridge’s 512 kB SRAM for bulk DMA transfers. It supports direct PCIe reads and writes, and transfers data at 700 MB/s over a USB3 10 Gbit link.

The 12V GPU rail uses an LM51770 buck-boost converter and an INA231 to monitor the rail. A separate debug USB port uses an FT230X for UART and reset.

For a reliable USB3 connection over a 3 m cable, the link was tuned on both ends. On the ASM2464 side the SerDes parameters were adjusted in the custom firmware. The SDM845 receive equalization and transmit de-emphasis were then tuned in AGNOS.

To test hardware reliability and improve the USB3 link, we built a rack of 10 comma four and chestnuts. The systems were tested under sustained load and repeated model loading.

Driver MonitoringPermalink

Face Detection FixPermalink

The driver camera frame is distortion-corrected when the face detection and pose ground truths are generated. In training the previous release model, the frames from comma four data were rectified with an incorrectly scaled distortion matrix. This made the face detector input more zoomed out, so labeled faces became smaller and offset on comma four segments, but head pose remains unaffected since it uses a crop. Some of these smaller faces are mistaken as backseat passengers and thus discarded.

It affected about 1.6% of comma four frames, so in 0.11.1 about 0.32% of all training frames received an incorrect negative face label. The small amount of junk labels were enough to cause model false negatives when the driver leans back. This bug made up ~75% of undetected faces and was fixed in Zoom Zoom Model.

Escalation Policy UpdatePermalink

Following recommendations in UN Regulation No. 171, we revised DM escalation and tweaked alert timings, focusing on making the severity of different DM alerts clearer. Notably, the 3-second text-only alert was removed, and the first alert at 5 seconds was given a new sound, softening the first transition into audible alerts.

Lockout times are now more forgiving on initial entries, starting at 1 minute instead of lasting until the car ignition is restarted. Repeated lockouts increase to 5, 15, and then 30 minutes, penalizing repeatedly distracted driving while reducing the impact of an accidental lockout during normal driving.

Gold CarPermalink

A gold car has accurate enough controls that every driving failure is a model failure, not a controls failure. If openpilot fails to stay within the lane, takes a turn poorly, or stops too late, that is a model failure.

Until now, the Tesla Model 3 and Model Y were the only cars we could reliably trust to actuate what the driving model planned. Having a diverse fleet of gold cars helps with driving model development, as the controls are no longer the limiting factor. In this release, we expand that fleet with the Volkswagen ID.4, our first car on Volkswagen’s MEB platform.

How the ID.4 ComparesPermalink

In the lateral 0.5 Hz sine maneuver at 30 mph, the ID.4 reaches the 50% threshold in 350 ms while the Tesla Model Y takes 423 ms. This allows the car to follow the model’s plan with less actuator delay and more accuracy.

Good longitudinal control comes down to how fast the car reaches the requested acceleration and how closely it maintains it without over or undershooting. The ID.4 and the Tesla platforms manage this well enough that openpilot sends the feedforward request from the model straight to the car. To contrast with an example, Toyota vehicles need a PID loop, pitch compensation, and anti-windup code to correct for error in its powertrain control module. The Model Y reacts faster to changes, but the ID.4 tracks a changing request more tightly.

Take a look at the full reports behind these plots at commaai/opendbc-data.

Native Curvature ControlPermalink

The MEB steering rack is not torque controlled like previous Volkswagen platforms. It takes a target curvature and a power percentage.

To allow a consistent lateral jerk and acceleration at any speed, we improved our existing curvature safety limits (#3405) by operating directly in acceleration space. Previously, 3 coarse speed breakpoints were applied in curvature space, which resulted in a different jerk at every speed. Now, low-speed jerk does not collapse to zero, allowing the car to wind up and down for turns much faster. Ford is a curvature-controlled car, it gained this improvement as well.

The port is based on community groundwork from lukasloetkolben for the harness, and daHansi and jyoung8607 for the port.

The MEB platform is used across the Volkswagen group, including the ID.3, ID.5 and ID.7, the Škoda Enyaq and Elroq, the CUPRA Tavascan, and the Audi Q4 e-tron. Support for these models will follow.

Simpler cruise speed control logicPermalink

As the models become more capable in controlling the acceleration and braking of the car, we can remove constraints on the longitudinal action the existed to make the acceleration and braking smoother. With this PR the cruise acceleration limits are simplified such that in experimental mode openpilot can accelerate as much as the model wants. This means in experimental mode openpilot will now accelerate faster in some cases.

New SoundsPermalink

This update introduces a new family of sounds, designed for comma four.

Audible feedback is one of the most important interfaces between a driver and openpilot. It conveys relevant information immediately, and defines the personality of the device. The philosophy behind comma four has always been one of restraint; it should be just noticeable, just audible, and only when necessary. The previous sound pack was too harsh and raw, we wanted to make sure this new family of sounds was coherent, harmonious, and modern.

The optimal playback zone of comma four’s speaker sits between 700-1200Hz where the distortion is lowest and the volume is highest. We determined that a simple sine-like tone reinforced with a round thumpy bass produced a sound that was rich and audible above a wide range of background noises, while retaining a modern and clean characteristic.

The structure of the sounds are simple so they cannot be confused with one another despite being written in the same key.

Basic system feedback is centered around the neutral note, A5. All sounds relevant to the engagement state start here. Uniquely, these sounds (engage, disengage, and refuse) include two tones so they are far less likely to be confused, even if they are partially obscured. Furthermore, they are designed to be understood without context. Engage and disengage are based on the universal “mhm” (affirmative) and “mm-mm” (negative) noises found evolved into cultures and languages across the world. Additionally, their timings were tuned based on samples from various physical cruise control buttons and lever detents from a wide array of manufacturers. This way they can feel like a natural extension of your car’s haptics, which immediately connects the driver’s action to the audible feedback from comma four.

Alerts escalate in frequency and shift to a minor key to give an “off-putting” sense to a driver and grab their attention from the more typical recurring sounds like the engage and disengage tones. They also move upwards along the octave until the very highest pitch maximum alerts. This not only allows us to truly max out the speaker when needed, it also works physiologically; the human ear picks out higher pitches quicker and easier, especially in the expected noise environment, the inside of a moving vehicle, where most sounds are low spectrum bass noises.

UI UpdatesPermalink

Driving ViewPermalink

Alerts now include more text or updated text to be more helpful. Previously, some alerts shared headlines that notified the user what action to take, but not why they needed to take action. This is a step to better flesh out the alert and notification experience on comma four.

The bookmark gesture has been updated for more visual consistency with the rest of the UI. The button now has a pressed state when activated, making it more clear when a bookmark will be created or when it can be canceled.

Driver monitor icon head-tracking arc position is corrected in RHD. This was a visual bug.

HomescreenPermalink

A notification pill has been added to the bottom right of the homescreen. It only shows when there are active notifications.

For chestnut users, a chestnut status icon now shows in the tray. Green indicates a working chestnut, pulsing white means the big model is loading, and orange means there is some issue preventing the chestnut from working. All issues have corresponding notifications to help diagnose and fix the problem. Additionally at the start of driving, the chestnut icon appears in the driving view until the big model is loaded, then fades out. This is synced with the homescreen icon so the chestnut status is clear from either view.

A generic USB icon was added to indicate a connection detected in the AUX port.

SettingsPermalink

Software updates and management have been moved out of the device menu into a new software menu. This is part of a broader series of ongoing changes to better semantically organize the various settings.

A new comma connect and prime status menu has been added. This makes it easier to check which account is paired to your device, and see your prime status on device.

eSIM management options have been added to the network menu.

Some buttons can now be long-pressed to view deeper explanations.

body & comma connectPermalink

Over the past few months, we have been busy working on reviving the comma body, and this release marks its first major update in a long time. We are working toward body v2, and while we do this, we get the chance to ship features and improvements not only for body v1, but also for cars (keep reading to learn more).

Remote Teleoperation & LivestreamingPermalink

A big problem with the body in the past was that it was too hard to use and actually get any use out of. That’s why, for our first major comma body release, we are announcing remote teleoperation in connect! If you are a comma body v1 owner, all you need to do is update your comma 3X (or four!) device and connect it to your comma body, pair it to comma connect with your phone, and then you can drive it around from anywhere in the world.

Here’s one of our Discord community members teleoperating the device from a different city during one of our dev calls:

Along with this big customer-facing feature, we have also taken the time to clear the cobwebs from the body infrastructure in openpilot, and are laying the groundwork for more features, more community involvement, and the release of body v2 in the future.

We have added new C4 head mounts for the comma body v1, so you can now 3D print a mount for your new comma four and use it with your body v1 as well.

body v1 firmware also has some quality-of-life updates. Visit the body firmware repository to install them.

A corollary to remote teleoperation—and probably the feature that most people will actually care about—is the new car livestreaming feature! Since we developed the whole stack for remote teleoperation, we realized it would take minimal effort to port it to cars and replace our old screenshot feature.

Now, with an internet connection—whether through your home Wi-Fi or comma prime LTE—and an installed comma power, you can livestream your road and driver cameras to see what is happening around your vehicle while it’s parked. Keep a constant eye on your belongings and who is around your car!

We would like to add more features like this to make our own version of Sentry Mode, but for the car you already own!

comma body UIPermalink

It was also time to give the body UI an update. Since we moved to raylib, we thought it would be only natural to also write the body UI in raylib! Apart from raylib being awesome, it also allows us to make more dynamic animations for the body face. It makes it a lot easier for you as a developer to create your own faces and expressions with the FaceAnimator class.

The class is a lightweight animation player built around a simple idea: a 16-by-8 grid of dots, each of which you can turn on or off.

A single frame is just a pattern of dots on that grid, so drawing a face is as simple as picking which dots to light up:

String a sequence of those frames together and you’ve got an animation.

Since these animations are programmatically designed, you can also change them and their colors on the fly to react to control inputs or things in the environment.

connect ClipsPermalink

Although we talked about connect in the last section, this feature is big enough to warrant its own section. Introducing… clips in connect!

Now in connect, you can click the clips button on a drive to generate a clip of a selected portion of your drive. Your comma device encodes that section of the drive into a user-friendly format that you can download and share anywhere you would like. Here is a quick video demo of how it works:

On each device’s dashboard, you can click the clips icon to see all the clips you have made on your device, and they will be cached in your browser or app for you to view later.

comma normally records video in HEVC (H.265), but its fixed-length segments are inconvenient to share. Now, you can choose any part of a drive and your comma will stitch and re-encode it as a widely supported MP4 (H.264). Leave the device powered while it works, then return to connect to download your clip!

CarsPermalink

Bug fixesPermalink

• Nissan Altima: fix fingerprinting without comma power (opendbc#3418)
• Nissan: fix driver override faults by winding down torque more (opendbc#3407)
• Rivian: fix radar lateral position (opendbc#3511)
• Toyota: remove faulty radar data from lead training stack (opendbc#3452)
• paramsd: stop learning vehicle parameters while in reverse (#38573)

EnhancementsPermalink

• CI / testing: replace hypothesis fuzzer (~1MB wheel) with faster 122 line file (opendbc#3582)
• Nissan: higher precision steering angle and smoother steering (opendbc#3423, opendbc#3415)
• Nissan: allow engaging in low gear (opendbc#3416)

Car portsPermalink

• Volkswagen ID.4 2021-2025 support thanks to DaHansi!
• CUPRA Born 2021-2023 support thanks to DaHansi! (opendbc#3451)