First a little bit of history, the SeaMicro SM10000 upended the rack density paradigm when they packed 512 Atoms in to a single 10U box. If you look at the board at the bottom of the linked article, you can see there are 8 single core Atoms and 8 chipsets. When Intel decided to un-fuse off a few of the goodies in the Atom cores and call it the N570, SeaMicro updated their servers.
The key take home here is that the N570 is a dual core CPU with a full speed PCIe2 1x slot instead of a half speed one, and 4GB of memory. Net effect, the SeaMicro motherboard/cards went from 8 socket to 4, but per core bandwidth, memory, and most everything else stayed the same. Power uses did go down as well, something very important at these density levels.
Drives in front, CPUs on the sides
That brings us to the new box, the SeaMicro SM10000-64HD. This one takes the mobo card and adds two more N570s to it for a total of six. The chassis is still the same, as is everything else physical, they just filled in the blank space on the dual core/second gen SM10000 with more sockets.
Note the blank space by the logo
Everything else stays the same, 4GB memory per socket, 1.6GHz Atoms with 64-bit addressing not fused off for stupid marketing reasons, 64 HDs, and 16 10GbE ports. It is now arranged in 2 core * 6 socket boards, 128 per rack, arranged in 4 banks of 32. It adds up to 768 cores, really, and 1.5TB of DRAM too, while taking only 3.5KW ‘typical’. You can buy them now starting at only $237,000, get two before the price goes up, it is never too early to shop for Christmas.
There are a few interesting things left unsaid about the SM10000-64HD however. First is that SeaMicro has a fully functional 8 socket board with their first iteration of the SM10000 mobo. It went to four with the second generation, and is now back up to six. There is space for eight, and very likely power for eight, especially once the 32nm Atoms are fully debugged.
There is also PCIe bandwidth available, the 4 SeaMicro ASICs on each card support 6 1x sockets, so each chip is capable of at least PCIe2 2x. While SeaMicro would not say how much each one can support, PCIe2 tends to come in 1x, 4x, 8x, and 16x widths, 2x is entirely possible, but very uncommon. Given the two PCIe connectors on the top of the cards, I would bet each chip can support at least a 4x connection, possibly even 8x.
If Intel ever makes a real Atom SoC, putting 16 of those on a SM10000 card seems eminently possible. Then again, one bullet point in the SeaMicro slide deck points out that the ASICs at the heart of the system connect to the CPUs via plain old vanilla PCIe2. They do this in an odd way however, they say, “CPU independent; X86, ARM, etc.” That kind of makes you wonder a bit, doesn’t it?
Mind the gap
Last up is an interesting aside, the interconnect for the SM10000 box. It is all based on a single chip, currently an FPGA, that sits a little bit above the Ethernet ports on the back side. The fabric is a bunch of toruses, and runs in multiple dimensions. Luckily, the boards are all passive, you can see them on the sides and down the center of the unit.
If you see a SM10000 open, you will notice how much empty space there is inside, as well as how empty the boards that occupy it are. It doesn’t take a genius to figure out that you could probably add 50% more CPU cards to the system. If you think about it, 192 16 socket boards filled with the upcoming 22nm 8 core ‘server’ Atoms, means that you are well over 2000 cores per U. Lets see anyone else top that without pulling a GTX590.S|A