So for context, I'm in automotive on the FAANG side and work closely with people at traditional companies.
There's some misunderstandings here. For one thing, there are obviously skilled people working at every salary level. What you're missing is that this is largely irrelevant. Relative compensation (in software) is a shockingly good proxy for all the associated support and people investments to support them. Employees that cost less are less supported and thus less enabled to produce high quality work than their better-paid counterparts, as a very rough rule of thumb.
It's my experience that OEMs are significantly less invested in their developers and the quality of the software they produce than any FAANG. They don't consider software to be a core competence and they rarely invest in the technical expertise to understand quality or push it internally.
The scale of dev costs is also pretty minor for vehicles as a whole. Amortized labor costs are typically on the order of 10-20% per vehicle. Devs are a small portion (~100s-1000s) of the tens of thousands of engineers that work at major OEMs and their suppliers. An extreme 2-3x wage increase would end up as a few percent increase, but the truth is there are cheaper ways to do that than salary. "All" they have to do is care about the software components the way many do about the mechanical components. It's a cultural problem rather than a strictly monetary one.
> OEMs are significantly less invested in their developers and the quality of the software they produce than any FAANG.
We must be using different FAANG software because the ones I use on a daily basis are slow and buggy as all hell. The last few versions of macOS/iOS have been atrocious. Every version of CarPlay I've used has bugs. Amazon apps are full of glitches. Compare that to developing engine ECU software which I think is a far more difficult engineering task that the average overpaid UI JavaScript slinger could not handle. You can't just update ECU software every week because you sloppily overlooked something. Unless you're Tesla of course.
I don't deny that there are many outstanding, technically excellent engineers at FAANG that deserve every penny they make. But the sad reality is many of them are just TC-chasing posers who got lucky and have subsequently deluded themselves into believing that whatever narrow expertise they now possess is worth far more than it is on the open market.
Want to know something terrifying?
ECU code is often nearly as sloppily written as that JavaScript. They follow MISRA and usually have a couple "QA" teams to catch the most egregious errors of course, but the overall quality is eerily similar to what you'd get if half the team came straight out of a time machine from 1995. I've had to give informational sessions on why boot systems should have redundancy and what undefined behavior is, for example.
And yet the ECU shipped with nearly every car for the past few decades almost never fails through millions of cycles.
I think all this "ECU code is bad" stuff is just people who don't like the style of the produced code. Are you sure those edge cases aren't accounted for in other ways? Maybe they decided that everything should be "best case" code, and any error just resets the ECU which boots up before the next spark event needs to be triggered.
I just cannot square how much people complain about the code in ECUs with how utterly reliable they are. Consider the group that investigated VW cars cheating on emissions. The system was complex and obfuscated, but also powerful, reliable, and configurable by the manufacturer.
A similar argument was made by Toyota about their systems during the lead-up to the unintended acceleration debacle. I recommend Koopman's talk [1] on the subject for details, but long story short the Toyota ETC was apparently utterly reliable and installed in some of the most popular vehicles on the market at the time. However, detailed investigation of the systems and code behind the few publicized incidents revealed basic system issues like the watchdog failed to detect task crashes, rampant memory safety issues, and even a failure to follow their own internal code guidelines. All of that was manifesting in people's vehicles, but it was only rarely turning lethal and even those cases went mostly unrecognized.
Things have improved significantly since the Toyota acceleration issue, in large part because all those details came out. Model-based design is now basically standard for most ECUs, which eliminates large amounts of buggy human-written code. Formal methods tools that don't entirely suck exist. However, most of the issues Koopman points out (especially memory safety) still exist in many places on modern vehicles and all of his points about the issues with relying on testing to surface quality problems remain relevant.
I think you’re conflating the UI of Amazon with the more difficult problem of getting a large distributed system to work at scale.
Conversely, if your ECU has minor glitches, will you notice? Unless it results in catastrophic failure, you will not even know if happens to result in the occasional 0.1 mpg increase in consumption.
And I’m sure VW and the other companies put a lot of resources into software engineering the exhaust emissions defeat workaround.
>> They don't consider software to be a core competence
Because it isn't for OEMs, theirbequipment is their core competency and aoftware, especially the entertainment side of things for automotive OEMs, is just part of that.
Would you argue that software isn't a core competency for Apple or Samsung because they sell phones? Same situation.
Modern vehicles are just distributed computers with tires, and self driving vehicles are worse. Virtually every feature is either massively supported or entirely enabled by software.
There's some misunderstandings here. For one thing, there are obviously skilled people working at every salary level. What you're missing is that this is largely irrelevant. Relative compensation (in software) is a shockingly good proxy for all the associated support and people investments to support them. Employees that cost less are less supported and thus less enabled to produce high quality work than their better-paid counterparts, as a very rough rule of thumb.
It's my experience that OEMs are significantly less invested in their developers and the quality of the software they produce than any FAANG. They don't consider software to be a core competence and they rarely invest in the technical expertise to understand quality or push it internally.
The scale of dev costs is also pretty minor for vehicles as a whole. Amortized labor costs are typically on the order of 10-20% per vehicle. Devs are a small portion (~100s-1000s) of the tens of thousands of engineers that work at major OEMs and their suppliers. An extreme 2-3x wage increase would end up as a few percent increase, but the truth is there are cheaper ways to do that than salary. "All" they have to do is care about the software components the way many do about the mechanical components. It's a cultural problem rather than a strictly monetary one.