Emulex is extending Fibre Channel with NVMe to get NVMf. SemiAccurate has the details for you about this interesting development in storage fabrics.
As you might expect from an acronym about NVMe over Fibre Channel called NVMf, the f probably stands for Fibre. That of course would be wrong, the f stands for fabric because NVMf is NVMe over fabrics, the protocol isn’t specific to Fibre Channel adapters.
If you want to use something over than Emulex Fibre Channel cards, it could also work over Ethernet, Infiniband, OmniPath, or any RFC 2549 compliant transport. (Note: Since QoS is a fundamental part of the system, RFC 1149 compliant devices won’t work.)
The NVMf stack
Getting back to the point of NVMe over FC or fabrics in general, the real question is why bother? The answer to that is flash memory or more to the point flash arrays. FC used to connect banks of storage to banks of compute, the classic SAN. IOPS were in the hundreds per drive spindle and latencies were, lets just say big by compute standards. With the SAN market moving to flash instead of spinning media, everything has changed and IOPS can be in the millions.
Instead of the bottleneck in the storage chain being the drive itself, physical components can only move so fast, flash meant almost instant response. Latencies of the system dropped by several orders of magnitude and the bottleneck of the chain moved elsewhere. Where that elsewhere is depends on the system, it could be the storage array, the network interconnect, the software running the requesting program, or something else.
Optimization of all the components became paramount, there was no longer a single mechanical choke point to beat for the rest of the chain. Everything mattered, and part of this everything is the SCSI/SAS stack. It works, it can be horribly complex but it is very good at mitigating the problems and quirks of spinning disks. Flash is not spinning disks and most of the arrays in question are bespoke architectures, not SAN boxes with SSDs replacing SAS drives. Decades of SCSI/SAS evolution and code are now a latency inducing pile of cruft but one that works reliably while running much of the world behind the scenes.
Emulex claims that the attach rate for their cards is about 90% flash at the moment, a growing market while HDD based SANS are declining. In this arena the bottleneck is a moving target and complex driver stacks don’t help in that regard. For the same complexity, latency, and speed reasons that NVMe is obliterating SATA and SAS on the PC side of the world, it is doing the same on the SAN front. This is why Emulex needed to make FC-NVMe/NVMf, latency would be pretty painful otherwise.
If you have a current Emulex SAN the idea of buying a new one based on a new paradigm isn’t all that enticing. SANS are complex, expensive, and need trained personnel to configure and administer. On the up side they are very reliable, fast, have low latencies, and are physically separated from the rest of your network for security. The protocol itself is a lossless design, something that has a different meaning in this context.
FC is a credit based system meaning anything that sends data has a guaranteed free path to the receiver so collisions or packet losses, there can’t be conflicts by design. No collisions means no retries and resulting retry packet storms on congested networks, at least in theory you can utilize 100% of an FC network. Think of it as QoS with guaranteed delivery at high loads rather than the crash and burn packet storms of Ethernet.
Since flash will probably max out any SAN interconnect like it did with SATA, congestion and high loads is an issue. It probably isn’t enough of an issue to cause an enterprise to rip out what they have and replace them with a new generation of Emulex FC cards. Luckily for them the beauty of NVMf is that you don’t have to rip and replace, it is pure software. If you know your networking stacks think layer 3 not layer 2. Update your firmware, add new drivers and off you go, no new hardware, no new wiring, and no new management tools needed.
The Emulex FC-NVMe stack
If you look at the box above, it intones that you can run both the SCSI and NVMe/NVMf protocols on the same card and over the same SAN. You can and it should do the proverbial ‘just work’ thing. Better yet Emulex is claiming that it will halve the latency vs a SCSI stack on the same hardware, a massive gain by any measure. For the foreseeable future the entire FC chain will have to run dual stacks but it won’t be long before SCSI is dead in this market, the latency of the stack makes that a certainty.
So that is what Emulex is delivering, not a new set of cards or a new infrastructure upgrade, just a new driver set that can halve latency on your existing network. For free, you can get the FC-NVMe stack for Linux on Sorceforge right now, the links to Broadcom’s page seem a little wonky though. In any case there isn’t much heavy lifting to do for this upgrade. Fast, simple, reliable, and standards driven, what’s not to like?S|A