Broadcom Tomahawk 2 switch support 64 100Gbps ports

Once chip with 6.4Tbps of bandwidth with deterministic latency

Broadcom logoBroadcom upped the ante to 64 100Gbps ports with the new Tomahawk 2 switch silicon. SemiAccurate feels this single chip now has the bandwidth and some really interesting advanced features to satisfy most home networking needs.

OK that was a joke, the new Broadcom Tomahawk 2 switch doesn’t have Wi-Fi so it isn’t good for home networking, just advanced data center spine duties, advanced top of rack (ToR), and cloud scaling. That is a joke too, the first part anyway, if your home network really needs 64 100Gbps ports, well consider us impressed. Even if you do, the advanced feature set of the Tomahawk 2 switch are much more than you need without a pretty large data center setup. Lets take a look at what this new device packs.

Broadcom Tomahawk 2 block diagram

The block diagram of Tomahawk 2

As the name suggests, Tomahawk 2 replaces the 2014 high-end 32 100Gbps port Tomahawk. Not only does it double the port count but it also doubles the route capacity, packet storage, and FlexGS scale. In short it isn’t just more ports it backs that all up with support and some new features too. Better yet the Tomahawk family which supplants the Trident family before it is also built on the same Broadcom SDK so code that doesn’t carry over directly should be an easy port between generations.

Broadcom Tomahawk 2 tier collapse

Tier collapse with more ports

The Broadcom BCM56970 as the Tomahawk 2 is now on the sales sheets can support 128 logical ports so 64 x 100Gbps or 128 x 40/50Gbps, take your pick. Although there are 256 25Gbps SerDes the rest of the chip doesn’t support 256 x 25Gbps so that configuration is a no go. There is also a 65/129th 1/10Gbps port for management and PCIe lanes for control plane integration. This high port count allows the example ~2k node networks above to lose a full tier going from Tomahawk to Tomahawk 2, a pretty major cost savings by any measure.

Broadly Broadcom is lumping the benefits, other than double the bandwidth, into three main categories, consistent programming model, deterministic performance, and enhanced visibility. The consistent programming model is the Broadcom SDK above, your software should be an easy port from one generation to the other, then it is just putting in new value added features the silicon supports. Technically Tomahawk 2 is the 8th generation of the family so this isn’t an off-handed or new claim, it goes back quite a ways.

Broadcom Tomahawk 2 automated load balancing

Load balancing can now be automated

One of the features SemiAccurate found interesting is automated load balancing. Each switch obviously needs to know the network topology either through ECMP pre-programming or the old-fashioned manual way. With Tomahawk 2 each node can determine which path to the target port will be least congested and do it automagically on the fly. This feature takes advantage of the fact that any decent modern datacenter will have dual paths to anything worth keeping up for redundancy so if it is there, why not use it for latency and bandwidth reductions? Quite the clever hack.

A lot of this is made possible by what Broadcom calls FleXGS or the ability to define flows, work on them, redirect them, and do it all on the fly. This user configurable engine makes a lot of advanced features if not easy, at least possible. There are other devices that can do the same but they tend to add one other less advantageous feature, time, and usually crater peak throughput/bandwidth.

That brings us to the next bullet point of the Tomahawk 2 switch, deterministic latency. Broadcom claims sub-400ns latencies when traversing the chip, not a bad number to start off with. Better yet you get the same bandwidth no matter how many features you apply to one of these flows. You may have noticed that we didn’t claim that latencies would stay the same when going from packet switching to turning on all the advanced features, they will obviously go up. What won’t go up, or in this case won’t go down is the bandwidth, that 6.4Tbps number is claimed to be unchanged with all or no features applied. To us anyway, this is an impressive accomplishment.

The last major feature of Tomahawk 2 is enhanced visibility something any network manager large or small craves. This time around Broadcom added packetized statistics, network latency analysis, and event-driven packet analysis. All three together make a pretty powerful tool set for switch makers to use in their quest to solve problems, an ongoing battle that just keeps getting harder.

Packetization is the interesting one and it tends to be used for longer term analysis. Instead of pushing out bits as they happen, Tomahawk 2 can gather them up and send them in packets to a central database, usually a central controller for a datacenter. This can really lower chatty traffic and sends groupings of states more intelligently than pushing registers and counters across the wire as they happen. Info comes as discrete pre-packaged state chunks, not random data points that need to be assembled again later to glean higher level knowledge from.

Event driven packet analysis is similar to modern code debuggers, you set an event or breakpoint and when that trigger happens, the switch reports it to the main console or whatnot. This is useful in and of itself but doesn’t give you insight into what lead up to the whoopsie so Broadcom fixed that. Instead of sending the registers when the fault happened, Tomahawk 2 buffers the relevant data all the time and reports events prior to and after the breakpoint with the trigger data. Can you say really useful?

Both of these features and more add up to the network latency analysis part, but that isn’t directly on the silicon. With the ability for a switch to know the datacenter topology, its state, triggers, and more, if you packetize all of that and send it to a central repository for analysis, you have a powerful network-wide visibility tool. Obviously this doesn’t run on the switch but it can’t be sanely done without some of the new features of the switch.

Broadcom Tomahawk 2 ecosystem

Broadcom Tomahawk 2 ecosystem

That brings us to how you get the Tomahawk 2 or a device bearing one. There is a large ecosystem of hardware players from brand names to ODMs that make boxes with this switch silicon inside. From there you have many more who write OSes for Tomahawk 2 and yet more writing software to add features and value to said boxes. Some of these are vertically integrated, others offer white boxes and nearly turnkey software, but all are based on the same Broadcom silicon.

Depending on your needs, Broadcom can supply you with chips, reference designs, software, tools, and APIs to do what you need. If Broadcom doesn’t supply it, a third-party vendor probably will. More interesting are the hooks in Tomahawk 2, low-level hooks to allow a control plane OS to do some low-level accesses to the data plane side. If a host/control plane OS can peer at low-level counters and tweak things like FleXGS flows on the fly, a lot can be done for real-time monitoring and optimization. This is where the 3rd party software and tools comes into play putting the value in value added.

Last up we have the physical side of the chip, those SerDes themselves. They can support backplane or copper cables directly, fiber with an external transceiver. As we mentioned before you can have up to 128 ports of 40/50Gbps or 64 100Gbps ports all off of one chip. The port count can collapse datacenter tiers which drops cost, complexity, and latency, things which are also addressed in different ways in the Tomahawk 2 silicon itself. Throw in the better reporting and visibility and you have a very useful chip for large networking installations.S|A

The following two tabs change content below.

Charlie Demerjian

Roving engine of chaos and snide remarks at SemiAccurate
Charlie Demerjian is the founder of Stone Arch Networking Services and SemiAccurate.com. SemiAccurate.com is a technology news site; addressing hardware design, software selection, customization, securing and maintenance, with over one million views per month. He is a technologist and analyst specializing in semiconductors, system and network architecture. As head writer of SemiAccurate.com, he regularly advises writers, analysts, and industry executives on technical matters and long lead industry trends. Charlie is also a council member with Gerson Lehman Group. FullyAccurate