Ten Gigabit Switching Thoughts

As mentioned in a previous post, I recently attended the Gestalt IT-organized Net Tech Field Day in San Jose, CA. This event brought me back in contact with a former colleague (Terry Slattery) and a number of my podcast/blogging friends (Ethan Banks, Greg Ferro, Brandon Carroll and Ivan Pepelnjak). In addition, I met in person for the first time many of the network-related blog authors and tweeters I follow (Jeremy Gaddis, , Josh Horton, Jennifer Huber, Steve Rossen and Jeremy Stretch). Oh, and Bob Plankers was there too, but he is just a server guy ;)

Thank you all for your company and your contributions to the many technical and non-technical discussions. And a special thank you to Stephen Foskett and Claire Chaplais for organizing this event. It was an amazing feat of logistics and vendor management. I am in awe of how smoothly the event went. I often run into more trouble during my daily commute, and I work from home!

During the planning of this event, Stephen Foskett asked the attendees what they were most interested in hearing about. The plurality of the responses, including my own, mentioned Data Center technologies. The vendors did not disappoint, as no fewer than five of the seven participants focused on this area. We received briefings on data center switching technologies from Hewlett-Packard, Force10, Juniper and Arista Networks.

The goal of this post is to compare/contrast the 10gb switch offerings of these vendors. Also, because the presentations/discussions made it clear that these vendors measure themselves against Cisco Systems in both market share and feature parity, I’ll include Cisco as well. I regret not including Foundry, Extreme, Oracle/Sun and anyone else, but I do not have any firsthand knowledge of their offerings. Any mention I would make of them would be strictly web-based research. You can do that yourself. :)

(Note… I decided to delay my review of the vendors’ chassis aggregation technologies for another blog post. This one was getting too long even without it.)

Fixed Configuration 10gb Switches

In short, they all have them. Arista sort of “ups the ante” in terms of advertised performance with their 7148SX switch. It is advertised as a low-latency device suitable for High Performance Computing (HPC) and High Frequency Trading (HFT) needs. Force10 also competes in this space with their S2410 device, which is promoted as a component of the New York Stock Exchange’s network. Our Arista contact made the point of saying that their switch is not eligible to be deployed at the NYSE because their company is not listed on that exchange. Abner Germanow (@abnerg) of Juniper mentioned that their devices were used in stock exchanges as well. Most of the supplied documentation mentions Juniper's M-Series routing platforms. At least one link (http://bloga.tw/a648bU) mentions the use of Juniper's EX-series LAN switches. I also noticed the Juniper routers included in the 60 Minutes feature on HFT. Cisco and HP do not appear to be competitive in the HPC/HFT arena, although I may have missed something in my research.

Another important item to note is that each of the presenting switch vendors at Net Tech Field Day has a 10gb, fixed configuration Layer-3 switch in their portfolio. Cisco (who did not present at the event) does not yet have this available. In a meeting with Jim Capobianco of Cisco last week, I learned that the upcoming Nexus 5548 & 5596 switches will eventually have this capability. It will require the installation of a Layer-3 Forwarding Engine, and will not be available until Q1 CY011. I am surprised they’re taking so long to deliver this, as it must be the cause of lost sales opportunities. The 4900M has something to offer in this space, but it is clearly not an integrated part of the new Nexus DC approach.

Lack of Innovation in the Space

I was struck by the similarities of all the switching vendors. Perhaps it was the tight timing constraints of the Net Tech Field Day sessions (most were 2 – 4 hours, with hard stops at the end), but with the exception of Arista Network’s offering (which I’ll discuss below), all of the vendors had very similar stories. This has been noted by several of my fellow attendees (Most notably Ivan). I don’t mean to pile on the criticism, as I’m sure developing these products is very difficult in itself, but I would love to see a significant differentiator from each vendor.

Juniper’s One OS

Aside from Arista, Juniper probably did the best job of differentiation with their “One OS” discussion. Their claim is that having a unified OS across multiple switching and routing platforms reduces the OPEX. Support for this claim include:

1. Network engineers only need to learn a single CLI
2. Commands are common to all platforms, allowing for better configuration standardization
3. Feature parity across all devices

Counterpoints include:

1. Not everything is standardized in JUNOS, such as hardware-based QoS configuration
2. Feature-set differences negate part of the feature parity claim (for example, no MPLS on LAN switches)
3. Cisco’s assertion that purpose-built OSs are better suited for unique environments

I’m not yet sure how I feel about this one. In all honesty, I’ve not had significant trouble learning new CLIs. The feature parity argument carries a bit more weight with me, especially considering my challenges with implementing features across IOS and NX-OS. I am also sympathetic to the OS-sprawl argument, best described by Mike Morris on his Network World blog. I suppose I’ll let the industry sort this one out without my input. I’m sure they’ll manage :)

Arista… Finally Something New

I was clued into Arista Networks about a year ago, when I read that Jayshree Ullal (and later Doug Gourlay) jumped ship from Cisco to join the 10gb switching startup. An industry friend of mine also highlighted their offering to me a few months later. At some point, I got added to a sales list for the company, and for the last few months I’ve received occasional marketing materials via email (thanks Alicia!). For me, the buildup to Arista Network’s presentation was quite extensive. I did not want this influence my fellow attendees’ perceptions, so I kept my thoughts to myself during the event.

Doug Gourlay of Arista scored a lot of points with the audience when he quickly explained that Arista Networks builds Data Center switches, and nothing else. It is clear that Arista is not attempting to be all things to all people. Their switches’ TCAM has room for 16K routes, 16K MAC addresses and 16K ARP entries. They are not going to be able to hold the Internet routing table, nor is Arista attempting to sell products that could do that. Doug was quite blunt when he said that “Arista is selling to companies where IT makes money.” Such businesses include Wall Street firms, HPC opportunities (Bio-Tech and other sciences) and social media websites.

So what’s new? For one thing, the switch runs a nearly-standard version of Linux, Fedora Core 12, kernel 2.6.31 (thank you to Doug for the correction) . According to the company, only about 750 lines were changed in the kernel to support the movement of device interrupts from system space to user space. This facilitates the starting and restarting of device drivers, and protects those processes from affecting the stability of the overall system. End users can build FreeBSD-compatible programs and run them in user space within the OS. EOS, Arista’s switch operating system, normally only requires 10% of one CPU core. On their dual-core switches, this leaves 95% of the processor power to custom-written applications. We were assured that EOS receives priority, so it is unlikely that a user application would affect the stability of the switching function. This capability is a standard feature of the Arista platform, unlike the additional cost of Cisco’s NM-based machine.

A second compelling EOS feature is VM Tracer. This allows a network admin to determine what device is attached to a particular port. If it is an ESX/ESXi server, it can query the server using VMWare’s API to determine which VMs are running on it. If a VM is VMotion’d to another ESX host, the switch can detect this and move the port-profile to the new location. It would be interesting to see exactly how this feature stacks up against Cisco’s NX-OS capabilities.

Summary

There are plenty of options available for 10gb data center network builds. Cisco is likely the safe option, although I do not see that they have any compelling features that would preclude me from choosing another vendor. Long ago I learned that one of the best negotiation tactics is to find two (or more) solutions you would be happy to deploy, then let both vendors know it. This will often get you the lowest price for your project. For the last few years, this has been difficult to do, since Cisco has done a relatively good job of innovating in the data center space. My recent Net Field Tech Day experience has shown me that there are other options that meet or exceed Cisco’s performance specs, so maybe it is time to search for competitive bids.

The Arista Networks presentation also demonstrates that other vendors are not standing still. Arista appears to be in a unique position to be able to price their devices at a premium. As a market strategy, Cisco and the other switch vendors need to begin innovating to put themselves in a similar position. (Hint… FCOE is not the answer). For Cisco, UCS is a potential driver of network equipment sales, but what about the other vendors?

(Disclaimer – Arista Networks, Force10, Hewlett Packard, and Juniper were sponsoring organizations of this event. There is no obligations for me to write anything about these companies or the other participants in this event. So while these musings came out of a sponsored trip, they are assuredly my own thoughts.)

The Ascendency of Hewlett Packard?

On Thursday I sat through several hours of Hewlett Packard presentations as part of the Net Tech Field Day program. Is there finally a viable competitor to Cisco across multiple product lines?

(Disclaimer – Hewlett Packard is one of the sponsoring organizations of this event. There is no obligations for me to write anything about HP, or the other participants in this event. So while these musings came out of a sponsored trip, they are assuredly my own thoughts)

Through its recent acquisition of 3Com, Hewlett Packard has acquired some fairly impressive networking technology. The A-Series switch line stacks up well to Cisco’s Nexus product line. If you read the following marketing description, can you determine if this is a description of the Nexus 7K platform or of the HP A-12500? I don’t think I could:

Next-generation, large core/data center switching platforms with innovative Intelligent Resilient Framework (IRF) technology. 18- and 6-slot switches with up to 6.6 Tbps performance and up to 128 (1:1) or 512 (4:1) 10GbE ports, and 864 GbE ports. Non-blocking, zero service interruption design, and architecture support for 40 GbE and 100 GbE. Wire speed L2/IPv4/IPv6/MPLS.

Except for the mentioning of IRF and the slot count, it sounds like the Nexus 7K to me. (Reference http://h10144.www1.hp.com/products/switches/index.aspx#A12500).

Unfortunately HP didn’t provide any tangible information about how this switch looks and feels in a predominantly Cisco network. The Tech Field Day participants pleaded for some white papers and/or deployment guides, but none are available at this time. In a somewhat ominous sign, the A-Series presenter mentioned that a design guide had been written, but HP has not yet determined if it would be treated as proprietary information for HP’s services organization.

Takeaways

I have two main takeaways from the HP portion of Tech Field Day:

1) Cisco may have a viable competitor across their networking product line.

If the industry rejects new Cisco-proprietary protocols in the Data Center (L2MP, VNTag), HP and other DC switching vendors (Arista, Force10, Juniper) will be able to compete directly on “speeds and feeds.” This would not be good for Cisco. Once a company goes multi-vendor, Cisco will lose its pricing power across the majority of its product lines. This is a good thing for the industry. Buying 3Com makes that equipment line a viable solution for businesses. I did not have enough faith in 3Com as a standalone company to recommend their gear for my network. (As an aside, I do not have faith in Foundry, even after being acquired by Brocade)

What also must be said is that HP is taking a huge risk here. My organization is building a greenfield data center soon. Historically we’ve gone with HP for the majority of our compute power (C-class blade chassis) and Cisco for most of the network components. If HP wants an opportunity to bid for the network side of this project, we’ll also need to open the compute side up to Cisco. All’s fair, right? The issue with this is that the network opportunity is approximately one-third the size of the compute opportunity. HP is effectively risking X for a shot at X * 1.33, while Cisco is risking Y for a shot at 4 * Y.

2) HP’s large services organization could significantly hinder their ability to gain traction in the marketplace.

There are basically two types of resellers in the network marketplace. There are those that compete on price and attempt to shave a few percentage points of margin off every sale, and there are those who make their margins based on consulting or management services that are sold along with networking equipment. My organization generally prefers the former, as we are comfortable with our ability deploy complex networking solutions. Price is usually our deciding factor for equipment purchases, although we have made exceptions for new technology, such as WAN acceleration and Cisco’s ACNS video platform. The problem with utilizing the cheapest provider is that the purchaser and networking equipment vendor has to take responsibility for the network architecture, product selection and installation.

For organizations that have less network know-how, purchasing from resellers that can deliver the installation and management services required is a more attractive option. They can provide network designs and product selection advice. If the reseller has agreements with multiple equipment vendors, they can help determine which vendors' equipment best fits the customer’s needs. As a network engineer, these are the resellers I want to see in the marketplace, because they offer interesting jobs.

So what does this have to do with HP? As mentioned above, HP isn’t sure whether their deployment guides will be released to the general public, or for that matter, even to their resellers. This is ridiculous. If HP wants the world to adopt their product line, they need to do everything possible to educate the engineers responsible for deployments. If HP’s plan is to have their own services organization take the bulk of HP network deployments, they’ll lose all the value-added resellers. Those are the resellers they need the most, since they are the ‘trusted advisors’ for businesses. Without their support, HP will be forced to identify and win every networking deal on their own.