Back at IDF we got to see another really neat technology from Avago called ExpressFabric. In a 48U rack with 2U servers there are a total of 24 servers. Each one of these servers has its own network interface controller and other assorted I/O chips that draw between 5 and 8 Watts of power per server. Avago’s ExpressFabric solution leverages their recent acquisition of PLX to replace these NICs with ~1 Watt PCI-E re-timer chips that are connected with copper or optical cables to a PLX supplied switching chip where all the traffic from the rack is sent to a single NIC, HBA, or other I/O interface chip residing in that top of rack server. Essentially ExpressFabric is an I/O disaggregation technology designed to work within the confines of a rack.
There are a couple big advantages to using Avago’s ExpressFabric namely latency, cost, and power. According to Avago their ExpressFabric costs half as much to implement and draws half the power of a conventional I/O solution thanks to not paying for all the NICs and HBAs you no longer need as well as their power draw. One of the biggest benefits is faster communication between the servers within the rack because they’re all connected to one another via PCI-E rather than through NICs. ExpressFabric is a lot faster than using a 10 Gb/s NIC because 4 lanes of PCI-E 3.0 are good for up to 32 Gb/s of bandwidth and can reliably provide more than 20 Gb/s of bandwidth.
Here’s the PCI-E re-timer add in board.
To be clear ExpressFabric isn’t designed to compete with Infiniband which is for rack to rack connections or applications that need a much higher bandwidth level but it is designed for HPC and micro server applications.
With Avago’s ExpressFabric you have a clever solution that reduces capital and operating cost, improves intra-rack performance, and leverages industry standards to do so.S|A