Ten Gigabit Switching Thoughts

As mentioned in a previous post, I recently attended the Gestalt IT-organized Net Tech Field Day in San Jose, CA. This event brought me back in contact with a former colleague (Terry Slattery) and a number of my podcast/blogging friends (Ethan Banks, Greg Ferro, Brandon Carroll and Ivan Pepelnjak). In addition, I met in person for the first time many of the network-related blog authors and tweeters I follow (Jeremy Gaddis, , Josh Horton, Jennifer Huber, Steve Rossen and Jeremy Stretch). Oh, and Bob Plankers was there too, but he is just a server guy ;)

Thank you all for your company and your contributions to the many technical and non-technical discussions. And a special thank you to Stephen Foskett and Claire Chaplais for organizing this event. It was an amazing feat of logistics and vendor management. I am in awe of how smoothly the event went. I often run into more trouble during my daily commute, and I work from home!

During the planning of this event, Stephen Foskett asked the attendees what they were most interested in hearing about. The plurality of the responses, including my own, mentioned Data Center technologies. The vendors did not disappoint, as no fewer than five of the seven participants focused on this area. We received briefings on data center switching technologies from Hewlett-Packard, Force10, Juniper and Arista Networks.

The goal of this post is to compare/contrast the 10gb switch offerings of these vendors. Also, because the presentations/discussions made it clear that these vendors measure themselves against Cisco Systems in both market share and feature parity, I’ll include Cisco as well. I regret not including Foundry, Extreme, Oracle/Sun and anyone else, but I do not have any firsthand knowledge of their offerings. Any mention I would make of them would be strictly web-based research. You can do that yourself. :)

(Note… I decided to delay my review of the vendors’ chassis aggregation technologies for another blog post. This one was getting too long even without it.)

Fixed Configuration 10gb Switches

In short, they all have them. Arista sort of “ups the ante” in terms of advertised performance with their 7148SX switch. It is advertised as a low-latency device suitable for High Performance Computing (HPC) and High Frequency Trading (HFT) needs. Force10 also competes in this space with their S2410 device, which is promoted as a component of the New York Stock Exchange’s network. Our Arista contact made the point of saying that their switch is not eligible to be deployed at the NYSE because their company is not listed on that exchange. Abner Germanow (@abnerg) of Juniper mentioned that their devices were used in stock exchanges as well. Most of the supplied documentation mentions Juniper's M-Series routing platforms. At least one link (http://bloga.tw/a648bU) mentions the use of Juniper's EX-series LAN switches. I also noticed the Juniper routers included in the 60 Minutes feature on HFT. Cisco and HP do not appear to be competitive in the HPC/HFT arena, although I may have missed something in my research.

Another important item to note is that each of the presenting switch vendors at Net Tech Field Day has a 10gb, fixed configuration Layer-3 switch in their portfolio. Cisco (who did not present at the event) does not yet have this available. In a meeting with Jim Capobianco of Cisco last week, I learned that the upcoming Nexus 5548 & 5596 switches will eventually have this capability. It will require the installation of a Layer-3 Forwarding Engine, and will not be available until Q1 CY011. I am surprised they’re taking so long to deliver this, as it must be the cause of lost sales opportunities. The 4900M has something to offer in this space, but it is clearly not an integrated part of the new Nexus DC approach.

Lack of Innovation in the Space

I was struck by the similarities of all the switching vendors. Perhaps it was the tight timing constraints of the Net Tech Field Day sessions (most were 2 – 4 hours, with hard stops at the end), but with the exception of Arista Network’s offering (which I’ll discuss below), all of the vendors had very similar stories. This has been noted by several of my fellow attendees (Most notably Ivan). I don’t mean to pile on the criticism, as I’m sure developing these products is very difficult in itself, but I would love to see a significant differentiator from each vendor.

Juniper’s One OS

Aside from Arista, Juniper probably did the best job of differentiation with their “One OS” discussion. Their claim is that having a unified OS across multiple switching and routing platforms reduces the OPEX. Support for this claim include:

1. Network engineers only need to learn a single CLI
2. Commands are common to all platforms, allowing for better configuration standardization
3. Feature parity across all devices

Counterpoints include:

1. Not everything is standardized in JUNOS, such as hardware-based QoS configuration
2. Feature-set differences negate part of the feature parity claim (for example, no MPLS on LAN switches)
3. Cisco’s assertion that purpose-built OSs are better suited for unique environments

I’m not yet sure how I feel about this one. In all honesty, I’ve not had significant trouble learning new CLIs. The feature parity argument carries a bit more weight with me, especially considering my challenges with implementing features across IOS and NX-OS. I am also sympathetic to the OS-sprawl argument, best described by Mike Morris on his Network World blog. I suppose I’ll let the industry sort this one out without my input. I’m sure they’ll manage :)

Arista… Finally Something New

I was clued into Arista Networks about a year ago, when I read that Jayshree Ullal (and later Doug Gourlay) jumped ship from Cisco to join the 10gb switching startup. An industry friend of mine also highlighted their offering to me a few months later. At some point, I got added to a sales list for the company, and for the last few months I’ve received occasional marketing materials via email (thanks Alicia!). For me, the buildup to Arista Network’s presentation was quite extensive. I did not want this influence my fellow attendees’ perceptions, so I kept my thoughts to myself during the event.

Doug Gourlay of Arista scored a lot of points with the audience when he quickly explained that Arista Networks builds Data Center switches, and nothing else. It is clear that Arista is not attempting to be all things to all people. Their switches’ TCAM has room for 16K routes, 16K MAC addresses and 16K ARP entries. They are not going to be able to hold the Internet routing table, nor is Arista attempting to sell products that could do that. Doug was quite blunt when he said that “Arista is selling to companies where IT makes money.” Such businesses include Wall Street firms, HPC opportunities (Bio-Tech and other sciences) and social media websites.

So what’s new? For one thing, the switch runs a nearly-standard version of Linux, Fedora Core 12, kernel 2.6.31 (thank you to Doug for the correction) . According to the company, only about 750 lines were changed in the kernel to support the movement of device interrupts from system space to user space. This facilitates the starting and restarting of device drivers, and protects those processes from affecting the stability of the overall system. End users can build FreeBSD-compatible programs and run them in user space within the OS. EOS, Arista’s switch operating system, normally only requires 10% of one CPU core. On their dual-core switches, this leaves 95% of the processor power to custom-written applications. We were assured that EOS receives priority, so it is unlikely that a user application would affect the stability of the switching function. This capability is a standard feature of the Arista platform, unlike the additional cost of Cisco’s NM-based machine.

A second compelling EOS feature is VM Tracer. This allows a network admin to determine what device is attached to a particular port. If it is an ESX/ESXi server, it can query the server using VMWare’s API to determine which VMs are running on it. If a VM is VMotion’d to another ESX host, the switch can detect this and move the port-profile to the new location. It would be interesting to see exactly how this feature stacks up against Cisco’s NX-OS capabilities.

Summary

There are plenty of options available for 10gb data center network builds. Cisco is likely the safe option, although I do not see that they have any compelling features that would preclude me from choosing another vendor. Long ago I learned that one of the best negotiation tactics is to find two (or more) solutions you would be happy to deploy, then let both vendors know it. This will often get you the lowest price for your project. For the last few years, this has been difficult to do, since Cisco has done a relatively good job of innovating in the data center space. My recent Net Field Tech Day experience has shown me that there are other options that meet or exceed Cisco’s performance specs, so maybe it is time to search for competitive bids.

The Arista Networks presentation also demonstrates that other vendors are not standing still. Arista appears to be in a unique position to be able to price their devices at a premium. As a market strategy, Cisco and the other switch vendors need to begin innovating to put themselves in a similar position. (Hint… FCOE is not the answer). For Cisco, UCS is a potential driver of network equipment sales, but what about the other vendors?

(Disclaimer – Arista Networks, Force10, Hewlett Packard, and Juniper were sponsoring organizations of this event. There is no obligations for me to write anything about these companies or the other participants in this event. So while these musings came out of a sponsored trip, they are assuredly my own thoughts.)