Arm is launching an automotive grade ISP called the Mali-C71 and it is not what you expect. SemiAccurate thinks this one is going to be a real change from what people were expecting, it isn’t your average phone derivative.
Updated April 25, 2017@5:30am: Changed Mali-G71 to Mali-C71 in title and elsewhere.
Lets just say that the things the Mali-C71 has in common with a consumer ISP block can be summed up in the name. Yes the term ISP is about all the C71 and other parts have in common, that and the basic use case, image/signal processing. Other than that, forget what you know about consumer-class ISPs when you think about the Mali-C71.
If you are thinking automotive ISP more or less equals self-driving cars, you would be right. It also comes into play for things like backup cameras, video rear-view mirrors, and similar use cases, all in cars of course. Those camera heavy use cases will be common in mid-range cars in short order and self-driving will trickle down the cost slope in the same way too. All of these use cases are mission critical and also very prone to flocks of lawyers gathering at any hiccup. ASIL-D is just the beginning.
With consumer video blocks, pixel-perfect reliability is nice but not critical. A good example of this is if your ISP was really bad and was guaranteed to screw up one pixel per frame, would you notice in a video stream? If the signal path from the sensor was equally wonky and screwed up a second pixel, would you notice? How about if there was a third one, again guaranteed bad, from the rest of the image path? A 1080p screen is ~2M pixels so the answer is very likely not, especially at 60FPS much less 4K/120FPS.
Now how about if you are a car hurtling down the road at 100MPH with the cameras controlling this 2+ ton hunk of metal, glass, and squishy bits in the center. How much does a single pixel error matter? Is that a headlight in the distance or a whoopsie? Think it matters now? How about if you are a lawyer, fresh from eating puppies and now suing a car maker for something an autonomous driving vehicle did ‘wrong’. Even if you are not the squishy center of the car in question, you could be the squishy bits it swerves off the road into. If you don’t think every pixel matters, you are wrong.
That brings us to the basis of the Mali-C71 ISP. It starts out with reliability baked in on a level you probably have never seen in a device before unless you are into making safety certified hardware. Every pixel is reliable by design, and based on what ARM told SemiAccurate, we think they mean it. Every pixel is tagged individually. There are a claimed 300+ dedicated fault detection circuits. There is a fault interrupt controller and the system constantly runs built-in tests. All data paths are CRC checked (duh) and each pixel is also tagged to the data plane it came from. With this and a lot more you get the idea that ARM is serious about reliability of their ISP IP blocks in the automotive world.
From there things get a little more interesting. The needs for computer vision differ a great deal from the needs of a human eye. If you are using a camera for a backup sensor and a rearview mirror, do you optimize for the person looking at the image or the AI scanning the sensor? The aim of the Mali-C71 is to do both at the same time, where needed, and do it in an interesting way.
To start off the C71 has a very wide dynamic range, 24 stops or ~144db. The best modern SLRs are in the 16 stop/80-90db range so that should give you a good idea of what ARM/Apical means by ‘wide’. This isn’t to say there are sensors out there that can deliver that data to the ISP yet, but when they do come out the Mali-C71 will work with it. On top of this the C71 can inherently understand multi-image HDR natively in 2-4:1 exposure formats.
When it gets this information there is the issue of processing it in a way that can be displayed. A 4:1 HDR frame from a high sensitivity sensor will have all the needed data, but how do show it to a driver? Got a screen that will display that many bits? How about an eye that can process that, bet you don’t have one of those if you have a super-screen. In any case ARM gets creative here.
The idea is local tone mapping, not a new idea but one that is quite useful. Like many other ISPs out there, the Mali-C71 can do local tone mapping so all those blown out highs and dark shadows have perceivable detail either to the driving computer or the squishy bits. The scenario ARM uses for this is a backup camera looking into the sun with a person in the shadows behind the car. Without the dynamic range of the C71 and the local tone mapping, the person disappears. Things like this are the reason most cars still have inefficient wing mirrors instead of cameras.
Going back to reliability for a second, we come to another use of pixel tagging. If an ISP modifies an image before that image is fed into an AI system for driving the car, how should the AI deal with tweaked pixels? Is that single hot pixel that was mapped out a sensor error or a distant but fast closing headlight? The answer is that the C71 tags modified data like this, and can also provide a copy of the unmodified image as well if needed. Since AIs used in autonomous driving are based on probabilities, consider this an added weighting factor.
That brings us to the ISP itself, the block can handle four direct sensor feeds or up to 16 pulled from a frame buffer. The former is a lower count but the latter adds latency, pick your poison. One interesting aside is that multiple Mali-C71 blocks can’t be connected directly to sync more than four cameras directly. If you want to keep tight locks between multiple C71 blocks you need to do it in software yourself. This is unlikely to be a big impediment to anyone implementing this block in silicon but it is also something we would expect to be added in the next iteration of the IP.
Speaking of software, ARM doesn’t leave this bit out, it is probably the most complex, time-consuming, and problematic part of implementing the ISP. Throw in ASIL-D’s traceability requirements and you have a nightmarish task ahead of you before you can implement a certified basic video stream. One you start talking about multiple cameras feeding AI’s that drive cars, well, it only gets harder, slower, and much more costly.
ARM has two ways to address this problem starting out with hardware. With 4/16 cameras feeding a single C71, all frames can be kept in tight sync so you remove the issue of when frames were taken, the ISP knows. They are also tagged and the data paths are error checked as well. Doing all the processing in one unit removes a whole class of potential problems, at least in theory. A single ISP can also lower BoM costs as well.
More important is the software stack which ARM helpfully provides, it is a full reference platform out of the box. If you want to do basic image processing, auto white balance, and auto exposure, it is there. ARM also provides a “Roadmap to complete automotive software designed for ASIL compliance”. This may not sound like a big deal but if you understand how ASIL-D, ISO 26262, SIL 3, IEC 61508, and similar standards work for certification and related paperwork, this is a BIG deal. ARM providing a pre-rolled package for these standards is an incredible cost savings.
On top of this all, ARM provides a service for sensor tuning and calibration. They will work with vendors to bring up, tune, and calibrate various sensors so chips bearing a C71 ISP should just work out of the box with most imaging solutions. Image formats are not a problem either, a few of the sensor types supported are RGBIr, RGGB, RCCC(clear), and RCCB, plus multiple other combinations. The C71 can also be sliced into ‘virtual’ ISPs and output various planes as outputs so you can have a human viewable image and one suitable for computational imaging from a single sensor. If you start thinking about the possibilities this brings to the table, well things get wired in a hurry.
SemiAccurate is a chip site so we would be remiss if we didn’t bring you a block diagram of the Mali-C71. Not wanting to be remiss in our duties, much, again today, we bring you a block diagram of the ARM Mali-C71. Enjoy.
Block diagram of the C71
So next time you are looking for a ASIL-D certified single ISP for multiple cameras, all with security and reliability baked in from the beginning, ARM now has an IP block for you. Throw in very wide dynamic range, local tone mapping, and related image enhancements and ARM still has an automotive ISP block for you. If you want a complete reference software stack with reliability and traceability certs, well this joke is getting old already but it is still called the ARM Mali-C71.S|A