A Look at Avago’s ExpressFabric

Using PCI-E to ditch NICs…

Sep 23, 2014 by

IDF (29 of 33)

Back at IDF we got to see another really neat technology from Avago called ExpressFabric. In a 48U rack with 2U servers there are a total of 24 servers. Each one of these servers has its own network interface controller and other assorted I/O chips that draw between 5 and 8 Watts of power per server. Avago’s ExpressFabric solution leverages their recent acquisition of PLX to replace these NICs with ~1 Watt PCI-E re-timer chips that are connected with copper or optical cables to a PLX supplied switching chip where all the traffic from the rack is sent to a single NIC, HBA, or other I/O interface chip residing in that top of rack server. Essentially ExpressFabric is an I/O disaggregation technology designed to work within the confines of a rack.

IDF (30 of 33)

There are a couple big advantages to using Avago’s ExpressFabric namely latency, cost, and power. According to Avago their ExpressFabric costs half as much to implement and draws half the power of a conventional I/O solution thanks to not paying for all the NICs and HBAs you no longer need as well as their power draw. One of the biggest benefits is faster communication between the servers within the rack because they’re all connected to one another via PCI-E rather than through NICs. ExpressFabric is a lot faster than using a 10 Gb/s NIC because 4 lanes of PCI-E 3.0 are good for up to 32 Gb/s of bandwidth and can reliably provide more than 20 Gb/s of bandwidth.IDF (26 of 33)

Here’s the PCI-E re-timer add in board.

To be clear ExpressFabric isn’t designed to compete with Infiniband which is for rack to rack connections or applications that need a much higher bandwidth level but it is designed for HPC and micro server applications.

IDF (31 of 33)

With Avago’s ExpressFabric you have a clever solution that reduces capital and operating cost, improves intra-rack performance, and leverages industry standards to do so.S|A

The following two tabs change content below.
My Twitter profileMy Google+ profile

Thomas Ryan

Author at SemiAccurate
Thomas Ryan is a freelance technology writer and photographer from Seattle, living in Austin. You can also find his work on SemiAccurate and PCWorld. He has a BA in Geography from the University of Washington with a minor in Urban Design and Planning and specializes in geospatial data science. If you have a hardware performance question or an interesting data set Thomas has you covered.