Building An All-Flash Racing Cluster With Intel and VSAN and more...

Building An All-Flash Racing Cluster With Intel and VSAN

PatG_VSAN_IntelHyperconverged is the latest justifiable buzz-worthy topic in IT.

VMware’s software-led approach supports multiple hardware consumption options: from mild to wild — you’re not limited to someone else’s idea of an appliance if that’s not your thing.

And I’m really enjoying seeing the creativity from our hardware partners in coming up with clever and unique configurations.  The more the better!

At EMC World, the Intel team certainly raised the bar on impressive off-the-shelf Virtual SAN configuration — a 32-node all-flash NVMe-capable VSAN configuration that delivers both outrageous performance and substantial capacity in a slick single-rack footprint.

Better yet, they splurged for some very cool custom bezel graphics.  After all, it’s all about how your equipment looks, right?

I tweeted out a picture and a brief description from the show floor.  My twitter gang went so crazy commenting and retweeting, I thought I should loop back and share more detail.

In particular, I wanted to interview John Hubbard — the cool cat at Intel that put this impressive rig together in very short order.

John, tell a bit about yourself and what you do …

John_HubbardI’m a network engineer that was asked to test SSDs.  My first impression was that it sounded boring, but it turned out to be the best career choice I’ve made.  I’ve done network engineering work at Kaiser and other places, and then this spot opened up at Intel on the SSD side.

Having a solid network engineering background is very helpful when working with SSD designs, as it turns out.  As flash technology moves closer to the bus and CPU, it’s often the network design that becomes more important.

How did this all come about?

 VMware asked Intel about 6 weeks ahead of EMC World if we would be interested in building a cool all-flash VSAN configuration.

We thought that was a great idea to showcase our technology.  But six weeks!!  A big shout-out to our Intel team who made all this happen.  It couldn’t have been done without them, so a big thanks!!

To meet the timeframe, I managed the project end-to-end — I knew there would be a lot of helpful opinions, but we didn’t have much time to debate and discuss.  We ended up with a quick design, review and approval in record time.

Other than getting the hardware delivered, what was the hardest part of setting this up?

Intel_VSAN_32nodeMost of it was pretty straightforward.  Automation was key, as we’re talking 32 nodes here.  I used a lot of PowerCLI across 32 nodes which saved a lot of time: to clone VMs, configure static IPs, and minimized all the grunt work.

Let’s run through the configuration details for everyone …

Sure. It starts with the Intel ® S2600TT Server Board in a 1U package.

We have 36 cores per node, 2 processors each, in this configuration.  We’ve configured 128GB of RAM per node, more is possible.

Each node uses the Intel Solid-State Drive (SSD) Data Center P3700 Series NVMe for write-intensive caching, and the new Intel SSD DC S3510 Series 1.6TB drive for persistent storage.  

We've set up two disk groups in each node: one caching and two capacity drives.  It's a simple, straightforward design.

Everything is connected with dual 10Gbase-T.  

I should point out that – from a network perspective -- 10Gb is plentiful if your payload is small, e.g. small block sizes like the 4K we tested for here.  Larger block sizes, I’d consider 40Gb — especially as the cluster gets bigger.

And, of course, we used the standard VSAN 6.0 distribution, which now supports all-flash configurations such as this one.

You found some time to do some initial performance runs, what did you find?

We spun up 3200 VMs, each running IOmeter against a single 7.5 Gb VMDK – nothing fancy.

Out-of-the box, we were getting 3.25 million IOPS random 4K reads.  Not bad.  No tuning or optimization required.

Better yet, when we switched to 70r/30w 4K random mixes, we were seeing ~2 million IOPS.  Pretty impressive performance and capacity in a very dense package.

It’s early days for NVMe, are we fully exploiting it in this configuration?

Intel and VMware have certified for ESX, and we are in the process of certifying for VSAN.

VSAN_colorsOne of the things we’re waiting for is a new NVMe virtual adapter from VMware to come out.  That will deliver a much more efficient IO path than the current driver.  

But the current hardware configuration is ready when it becomes available, and it’s plenty fast in its present state.

Performance should bump considerably as a result when the new virtual adapter becomes available. I’m looking forward to testing that when it’s ready.

Is all of this hardware generally available today?

Yes – all the hardware is available today. The Intel DC S3510 enterprise capacity tier drive just launched, and that’s what we’re using in this configuration for persistent storage.  

The Intel DC P3700 we use for caching has been out for a while.  The boot drive is an Intel DC S3710 and can double for host swap if needed.

What’s the deal behind the cool bezel LED graphics?

Great, huh?  

I know a guy who likes doing this stuff.  His company, PCJunkieMods, did a great job for us.

In addition to the great stencil work, the LEDs can be made to flash, change colors, etc.  Big fun.

Just the thing for your next data center tour.

I know we’ll be seeing you and your team at VMworld — what’s the plan?

64 nodes, baby!  Our session has been submitted, and it’s titled “Will You Still Love Me When I’m 64?

Pat_and_JohnIn this session, we’re going to share all the details: how to design, optimize and tune and everything else you need to know.

Seriously, though, we’re seeing a ton of interest in bigger VSAN all-flash configurations and we want to show people what Intel and VMware can do together.

Any final comments?

It’s been pretty amazing.  This project has generated the most buzz and foot traffic than anything I’ve ever been involved with.  

People love the idea of all-flash hyperconverged clusters powered by Intel and VMware technology.  It’s been a blast.

And it’s nice to see all those drive bays finally get used …


 Like this post?  Why not subscribe via email?




The Split In Software Defined Storage

SplitYou just have to love enterprise IT tech.  

While much of it can be safely identified as “settled science”, interesting portions predictably morph and evolve quickly at the edges.

In my cozy little world, it’s clear that one category is showing obvious signs of splitting cleanly into two completely different beasts that share the same name — software-defined storage.

In many ways, this split has very little to do with vendors and technology — and everything to do with how customers are putting the technology to work.

The Basics Of SDS

DynamicI’ve written a lot on the topic — maybe too much! — but I’ve also gotten good at boiling things down to its fundamentals: in this case, the ability to dynamically compose storage services using an API.  

That's what makes it "software defined".


Today, most storage exposes static service levels, and does so on convenient storage boundaries, e.g. a LUN.  Under SDS, storage services can be dynamically requested, and are provided on convenient application boundaries, e.g. a VM.

That's the big idea, in a nutshell.

Note that this definition has little to do with how storage is actually implemented: external arrays, or using software running on servers.  

And that’s where it gets interesting.

Reinventing The Past?

Best-storage-certsOne core audience for software-defined storage is — obviously — storage professionals.  

Thankfully, they are acquainted with just how rigid and inflexible external arrays can be, so the notion of software-defined can be appealing.

The idea of providing dynamic services based on application policy — without manual intervention on each and every request — can be a huge win: happier internal customers, less over provisioning waste, and more time to focus on issues that really matter.

In the VMware portfolio, that’s where VVOLs come in — a control plane for storage that moves towards this dynamic services consumption model.  Arrays advertise their capabilities, applications express their preferences via VASA, and VVOLs are returned: customized storage containers that align to application boundaries.

The world of external storage arrays will be with us for many, many years to come.  These portfolios will have to be evolved: not only core technology, but also how they’re consumed.  VVOLs helps to do that.

But storage folks aren’t the only audience for software-defined storage, it turns out.

Building A New Future?

The second audience for software-defined storage isn’t really interested in storage as a standalone topic.  These aren’t storage people.  They don't want to be storage people.

Vcap5-dcdThis second group wants to collapse and integrate storage functions into their compute environment. They don’t want to use external storage arrays, or dedicated servers running storage software.

For them, it’s not about migrating the legacy models forward, it’s about establishing a new model.

They don’t want to acquire and manage storage separately.  They don’t want to invest in a storage team, or specialized storage expertise.  They want storage functions to be integrated into the same exact tools and workflows they use for server virtualization.

I’ll call this group the “hyperconverged” folks for lack of a better term :)

When you ask them about their motivations, the answer is almost always the same: operational efficiency.  Rather than trying to build better bridges between two worlds, they want to combine those worlds into the exact same thing.

The precise details of the storage technology become less important than how well it fits in with their desired model going forward.   It's all about the operational model.  

And that’s a very different agenda than the first crowd.

Differences In Perspective

PerceptionWhen speaking to the storage-centric folks about software-defined storage, their interests are usually the same: what’s the storage management console look like, what features do you support and how do you support them, how do they compare with the arrays we know, etc.?

It’s a very familiar storage conversation, only this time we’re talking about storage software running on servers.  Same familiar model, different tech.

The hyperconverged customer conversation is quite different.  

They’re not looking for a separate storage management console; they want storage that’s deeply integrated into their virtualization environment, most usually vSphere. As a group, they are usually less concerned with individual point features and implementation details.

Their measurement system is simple: does this new technology let me get my work done more efficiently?  In the VMware world, that’s VSAN.  It wasn't build for storage people -- it was built for virtualization people.

Note that with regards to the original definition — dynamic service levels from storage — can be achieved either way by either group.  

One industry concept, two very different measurement systems.  A split appears to be inevitable, especially when two very important audiences want two very different things.

Who’s In Control?

Behind the scenes, there are two distinct organizational models that are quietly competing.

ControlThe first is driven by the view that storage professionals should make storage decisions, and run the storage environment.  

All storage should be controlled by storage professionals.

The second is driven by the observation that dedicated storage professionals maybe aren’t needed for everything, and there’s value to be gained by consolidating portions of the storage function with the virtualization administrators.

Not all storage needs to be controlled by storage professionals.

Note that this isn’t an either/or discussion in most cases.  There are plenty of good reasons to continue to use external storage arrays for some workloads, and thus invest in the dedicated expertise needed to run them effectively.

Nor can it be stated that each and every workload always demands an external storage array, and its required dedicated expertise.  It’s about finding the right mix.

What’s Really Important To This Second Crowd?

In a phrase — deep integration.

IntTheir starting point is their hypervisor, most often vSphere.  

Everything — including software-defined storage — gets evaluated first and foremost by how deeply it integrates: technology, management, operations, workflows, support — you name it.  They want their storage technology built in, not bolted on.

Why?  It's all about the new operational model: a simpler, easier approach that enables fewer people to get more work done, faster and with less friction.

The Split Is Already Here

If you take a quick survey of software-based storage products in the marketplace today, you’ll see confirming signs.  There’s clearly one class of product aimed at storage professionals, and another class aimed at the all-in-one virtualization administrator.

They want very different things.  They have good reasons for doing so.

And — as a result — we’ll end up with two categories to debate about going forward :)


Like this post?  Why not subscribe via email?



The Cults Among Us

Leave_the_cultOver the past few years, news in the United States has extensively featured the activities of ISIS (ISIL?) — which I perceive as a cult brutalizing innocents in the name of religion.

Very nasty stuff, to be sure.

But the exact same human behavior is extensively quite visibly on the internet every day — albeit without guns and terrorism involved.

Political cults.  Religious cults.  Climate change cults.  And so on and so on.

Even in our cozy IT world, cults are among us.

And it’s not a good thing — for us as individuals, and us as an industry.

Cultism In IT?

CultsJust to be clear, I fully respect anyone with strong opinions and reasonable arguments to back them up.

I often don’t agree (that’s to be expected!), but I fully value their contribution to the discussion, and I fully welcome any opportunity to intelligently debate what's at hand.

But — very often — a line is crossed. “You don’t believe”. “You’re not one of us”. “You just don’t get it”.  Sometimes, it devolves into personal attacks, questioning motivations, and devaluing individuals just because they don’t agree with you.

This is cultism.  It is not a pretty thing.  It discourages intelligent stakeholders from participating. It stifles discussion.  It holds our industry — and our customers — back.

Why Cultism?

I am not a trained psychologist.  If I were, I could probably offer a scholarly perspective as to why human beings tend to behave like this — even when there are negative outcomes.

PrayingIf someone were to ask my naive opinion, I’d point to a couple of things relevant to cultism in IT.

One thing is insecurity.  If you are insecure of your place in life, your position in the industry, your future prospects, etc. — you are a candidate for IT cultism.  You can fill this void and gain confidence from being joined with other like-minded people, and collectively attacking non-believers.

Startups are historical breeding grounds for IT cultism.  There needs to be a dramatic, motivating storyline — changing the world, defeating the evil incumbents, creating a strong internal culture, etc.  Standard playbook stuff.

Want_to_joinBut at some point, a strong startup culture edges into cultism, and I would argue that’s not a good thing.

Groups of people who are heavily invested in a particular technology from a career perspective can be targets for spontaneous cultism — almost defending their prior choices, and refusing to acknowledge the inevitable and rapid shifts in technology.  

They feel threatened, they bind together, and start attacking the interlopers -- a cult is formed.

Why This Is Bad For IT Users — And The Industry

As a young pup, one of the reasons I liked the IT industry is that it was vibrant and dynamic.  Lots of smart people, lots of innovation and lots of healthy discussion and debate.  Fun, stimulating and ultimately value-creating.

DiscussionCultism smothers this behavior.  Who wants to get ripped up by some yahoo who has joined an IT cult, and considers it their moral duty to attack anyone who threatens their belief system?

In my travels, I meet all sorts of really bright people with really bright ideas, and I ask them “why don’t you speak up more publicly?”  You can guess what I hear in response.  They don’t want to subject themselves to the inevitable abuse — from IT cultists.

Some of us — myself included — have developed incredibly thick skins.  We think it’s our duty to speak up regardless of the abuse hurled at us.  But not everyone is willing to make that choice.

And that’s unfortunate.

What Can We Do?

I care about this industry, and I care about the customers that use technology to build a better world.  I think we deserve an environment where we can politely discuss and debate everything around us — and not fear the inevitable cultish tirade.

Better_worldCall me naive, but I think it’s something we should strive for.

So I’d ask you to consider two propositions.

First, when you see cultism, be prepared to call it out.  If you see a group of people banding together, espousing an isolated point-of-view, and jumping all over anyone who disagrees with them, name them and shame them.  You'll be doing a public service for the community

Second, if you’ve got strong opinions and good arguments, share them.  Be prepared for occasional bouts of cultism and online bullying.  The industry needs to hear your voice.  No matter what anyone else has to say.

It’s our chosen profession — our community — our career choice.

Let’s make this a better place, shall we?


Like this post?  Why not subscribe?







Life Beyond Hyperconverged

Wasp_nestWell, I certainly gave the hornet’s nest a good, healthy smack with my recent post (“Ten Reasons Why VMware Is Leading The Hyperconverged Market”).  

Never underestimate the power of a well-written blog post to shake things up :)

In addition to hearing from dozens of enraged Nutanix employees, the usual round of pundits are now weighing in with their perspectives as well.  Everyone is entitled to their opinions, and there seems to be no shortage of those.  

Although I do find it interesting that no one has yet attempted to refute any of the facts behind the ten arguments I presented.  That's typical in these situations: lots of passionate emotions, very little discussion of the underlying facts.

The good news: I did have a chance to have a few great conversations with intelligent, non-partisan folks who said they were thinking differently after reading my thoughts.

Thinking differently is always a good thing :)

BridgeThe observation: most strategic technologies in IT put their users on a pathway to something bigger and better.  That’s what makes them strategic, no?

The basic question: if hyperconverged is truly “strategic”, what bigger and better world should it lead to?   Or, is it an end unto itself in its present form, and thus not strategic?

Clearly, this is a great question, and worth a blog post or two to discuss …

The Fuss Over Hyperconverged

THE_HYPE_MAGAZINESome times new buzz-worthy IT concepts resonate and endure, sometimes they don’t. This one appears to be doing quite well at present, and I think I understand why that is.

The first wave of hyperconverged was presented as an appliance that didn’t need an external storage array.  The value proposition was heavily weighted towards “convenient consumption”.

Bring it in, rack it up, connect the network and presto!  Given the typical complex state of affairs in standing up IT infrastructure, it seems almost as easy as calling up your service provider and having them fire up a few new instances on your behalf.  And I’ve learned to never underestimate the appeal of convenient consumption.  VMware's EVO:RAIL hyperconverged offering clearly targets this model.

Now_and_laterI am arguing that we’re moving into the next wave of how people see hyperconverged.  

Yes, there are still IT shops that certainly prefer that “convenient consumption” benefit, but I see a growing number now view the potential to do more with the technology: both now and in the future.

But, as a result, the criteria changes in their mind — less weighted towards “the box” and immediate gratification, and more weighted towards “the strategy” of how their short-term choices play into the broader evolution of the IT landscape.

How Strategic Technology Usually Plays Out In Enterprise IT

StrategyAfter decades of being an armchair observer of enterprise IT adoption patterns, I’d argue that a common pattern is the “two-fer”.  As in "two for the price of one".

The new technology is brought in to ostensibly solve an immediate short-term requirement with obvious justification.  But at the same time, there is full awareness that this same technology has the potential to play a broader and more transformative role changing the way things are done in IT.

A familiar example?

VMware virtualization got its start by solving an immediate data center problem: rampant server overprovisioning.  The pitch at the time was deadly simple: save money with virtualization.  However, over time, people realized that — once virtualized — vSphere could fundamentally change the way IT was done from an operational perspective: provisioning, management and more.  And that was a really big deal.

The answer to a tactical problem built the foundation for a great strategic outcome.  And it wasn't dumb luck on the part of IT shops, either.  They saw what we saw.

Another example from the storage world?

Flash_storageWhen flash was introduced, it was seen as the solution to a very narrow but very demanding set of workloads, e.g. databases with very high transactional rates — a tactical solution to a specific pain point.

As prices dropped, many shops have now decided on a ‘flash first’ strategy — use it just about anywhere that performance could potentially be an issue.  

The result was that users got spectacular performance, and IT could get out of the storage-performance-problem-resolution business — arguably transformative in its own way.

Words Fail Me

So if we’re going to think of hyperconverged as one of these “two-fers” (tactical today, strategic tomorrow) what does the longer term picture look like?

At VMware, we describe future state data center architectures as “software defined”, e.g. the software-defined data center or SDDC.  Other labels also get used around similar concepts: devops, cloud, software-define infrastructure, etc.  

Software-defined.DC (3)I’m not here to debate labels, though.

Why?  The core technology ideas behind each are similar: heavy use of commodity technologies, everything programmable and thus able to be automated by software, driven by application-centric policies, dynamic and flexible, ease of consumption for end users, an enterprise-class operational model, etc.

Here’s the observation: with this perspective, well-considered hyperconverged solutions can easily deliver a “two-fer” for enterprise IT.

The tactical problem they solve is a cost-effective solution for an immediate infrastructure requirement.  The strategic benefit is that they can potentially create is a pathway to SDDC or whatever you’d prefer to call your next-generation environment.

But if we’re going to want to exploit that second part, our evaluation criteria may have to evolve.

Getting To SDDC

If you’ve ever sat down with a customer responsible for a large, complex enterprise IT environment, pitching the attractiveness of something like SDDC isn’t hard.  On paper, it’s not hard to get agreement that it’s a great future vision of how IT ought to work.

The fun part starts in putting in place a realistic plan to get there.

Moving_partsNot surprisingly, there are a *lot* of moving parts that are highly resistant to change.  Legacy investments and legacy vendors.  Operational models and processes that have existed for perhaps decades.  Entrenched organizations complete with factions, tribes and internal politics.  

And, of course, precious little time between firefighting episodes to actually work on anything.

Much as we’d like to believe that all it takes is magic software and a quick implementation plan, the reality is usually quite different.

Can Hyperconverged Be A Short-Cut To A Better Place?

Let’s say you’d like to introduce SDDC-like concepts quickly into your data center, but do so with a minimum of cost, hassle and inevitable organizational impact.  It’d be hard to imagine an easier or more powerful way of doing so than standing up a modest vSphere+VSAN hyperconverged environment.  Or maybe an EVO:RAIL if you're looking for something even simpler.  Same basic software technology in both.

You’d get the very best in hyperconverged software technology: vSphere, VSAN, vRealize Automation Suite, vRealize Operations, NSX, etc. etc.   You’d have vSphere admins on your staff who didn’t need a long learning curve.   You’d already have support relationships in place, etc.

You could evaluate for yourself — and quickly — what the new technology can offer.  With almost no downside.

Aircraft_carrierAnd, here’s the best part: if you decided you liked what you saw, the entire environment could be seen as a working scale model of what you’d eventually like to build:

- proven functionality, processes and tools that can scale to truly large enterprises
- the ability to accommodate and leverage existing infrastructure choices (servers, storage, network)
- the ability to extend in new directions, like OpenStack, containers or whatever new thing comes along next that looks attractive
- and with well-supported interoperability between all essential components

Let’s Turn It Around

OK, so I get involved in a lot of VSAN sales calls.  Let me share something interesting that I’m finding as a result.

Right now, it’s breaking 50/50 between people who are clearly looking for a tactical solution, and people who clearly want to take the first step towards a strategic outcome along the lines I’ve described.

HorizonYou’ll hear things like “we want to change the way we do IT”. Or “we’re trying to introduce a disruptive model into our environment”. Or perhaps “this is part of our cloud vision”.

Yes, there are the inevitable feature/function questions as you’d expect.  But it’s clear to this crowd that they’re not just looking for a like-for-like replacement for a traditional storage array.  No, they see something much more attractive just over the horizon.

They probably don’t use the word “hyperconverged” to describe what their first step is.  I think that’s partly due to the fact that the term has been unfairly chained to a specific piece of hardware and associated consumption, and that’s not what they’re ultimately after.  

They’re looking for a software model that fundamentally changes the way IT is done.

That being said, some of this group are still interested in the ease-of-consumption that comes with an appliance model such as EVO:RAIL — but it’s clear that’s not their desired end state.

Where Does That Leave Us?

ShortcutI think one core question remains — how will the marketplace come to define “hyperconverged”?

Will it continue to be associated with a specific vendor appliance, focusing on ease-of-consumption?

Or will more people realize that hyperconverged is essentially software, representative of a large-scale design pattern made easy to consume, and thus can be an attractive short-cut to a much nicer place?

I know which outcome I’m betting on.


Like this post?  Why not subscribe via email?





Going To EMC World? We'd Like To Have A Chat!

It's that time of year again -- EMC World in Las Vegas May 4-7.  Only a month away!

Emcworld2015In my world, EMC World is absolutely the best show to talk storage in all its different aspects.  You meet some pretty amazing people there.

For the last two years, we've held small non-disclosure sessions with select folks at the show.  

We share some of what we're working on, and we get incredibly valuable feedback on key issues we're debating internally.

We'd like to do it again ... if you're up for it!

So, here's the deal -- if you're way into virtualization and software-defined storage, maybe you'd like to join us?  Space is very limited, though.  Previous attendees to prior sessions get first priority.

This year, we'll be talking about:

  • the current VSAN roadmap 2016 and beyond
  • new proposals on how we manage VSAN ReadyNodes and the VSAN HCL
  • a new software-defined model for data protection and management
  • plans for deeper integration with vCloud Air
  • and a discussion around "cloud native" applications, time permitting

We'll be holding three sessions in a suite at the Venetian, near by:

  • Monday, May 4th 3-5 pm (partners only, please)
  • Tuesday, May 5th 3-5 pm (end users and partners)
  • Wednesday, May 6th 3-5 pm (end users and partners)

If this sounds like something you'd be interested, please drop me an email at  In your email, any information you could provide as to what you do, why you're interested, etc. would be very helpful.


-- Chuck