Hyper-V + VXLAN and more from Tech Ed Europe

If you thought -as I admittedly did- that on-prem Windows Server was being left for dead on the side of the Azure road, then boy were we wrong.

Not sure where to start here, but some incredible announcements from Microsoft in Barcelona, most of which I got from Windows Server MVP reporter Aidan Finn

Among them:

  • VXLAN, NVGRE & Network Controller, courtesy of Azure: This is something I’ve hoped for in the next version of Windows Server: a more compelling SDN story, something more than Network Function Virtualization & NVGRE encapsulation. If bringing the some of the best -and widely supported- bits of the VMware ecosystem to on-prem Hyper-V & System Center isn’t a virtualization engineer’s wet dream, I don’t know what is.
  • VMware meet Azure Site Recovery: Coming soon to a datacenter near you, failover your VMware infrastructure via Azure Site Recovery, the same way Hyper-V shops can

    Not sure what to do with this yet, but gimme!

    Not sure what to do with this yet, but gimme!

  • In-place/rolling upgrades for Hyper-V Clusters: This feature was announced with the release of Windows Server Technical Preview (of course, I only read about it after I wiped out my lab 2012 R2 cluster) but there’s a lot more detail on it from TechEd via Finn:  rebuild physical nodes without evicting them first.You keep the same Cluster Name Object, simply live migrating your VMs off your targeted hosts. Killer.
  • Single cluster node failure: In the old days, I used to lose sleep over clusres.dll, or clussvc.exe, two important pieces in Microsoft Clustering technology. Sure, your VMs will failover & restart on a new host, but that’s no fun.  Ben Armstrong demonstrated how vNext handles node failure by killing the cluster service live during his presentation. Finn says the VMs didn’t failover,but the host was isolated by the other nodes and the cluster simply paused and waited for the node to recovery (up to 4 minutes). Awesome!
  • Azure Witness: Also for clustering fans who are torn (as I am) between selecting file or disk witness for clusters: you will soon be able to add mighty Azure as a witness to your on-prem cluster. Split brain fears no more!
  • More enhancements for Storage QoS: Ensure that your tenant doesn’t rob IOPS from everyone else.
  • The Windows SAN, for real: Yes, we can soon do offsite block-level replication from our on-prem Tiered Storage Spaces servers.
  • New System Center coming next year: So much to unpack here, but I’ll keep it brief. You may love System Center, you may hate it, but it’s not dead. I’m a fan of the big two: VMM, and ConfigMan. OpsMan I’ve had a love/hate relationship with. Well the news out of TechEd Europe is that System Center is still alive, but more integration with Azure + a substantial new release will debut next summer. So the VMM Technical Preview I’m running in the Daisetta Lab (which installs to C:\Program Files\VMM 2012 R2 btw) is not the VMM I was looking for.

Other incredible announcements:

  • Docker, CoreOS & Azure: Integration of the market-leading container technology with Azure is apparently further along than I believed. A demo was shown that hurts my brain to think about: Azure + Docker + CoreOS, the linux OS that has two OS partitions and is fault-tolerant. Wow
  • Enhancements to Rights Management Service: Stop users from CTRL-Cing/CTRL-Ving your company’s data to Twitter
  • Audiocodes announces an on-prem device that appears to bring us one step closer to the dream: Lync for voice, O365 for the PBX, all switched out to the PSTN. I said one step closer!
  • Azure Operational Insights: I’m a fan of the Splunk model (point your firehose of data/logs/events at a server, and let it make sense of it) and it appears Azure Operational Insights is a product that will jump into that space. Screen cap from Finn

This is really exciting stuff.

Commentary

Looking back on the last few years in Microsoft’s history, one thing stands out: the painful change from the old Server 2008R2 model to the new 2012 model was worth it. All of the things I’ve raved about on this blog in Hyper-V (converged network, storage spaces etc) were just teasers -but also important architectural elements- that made the things we see announced today possible.

The overhaul* of Windows Server is paying huge dividends for Microsoft and for IT pros who can adapt & master it. Exciting times.

* unlike the Windows mobile > Windows Phone transition, which was not worth it

More than good hygiene : applying a proper cert to my Nimble array

So one of my main complaints about implementing a cost-effective Nimble Storage array at my last job was this:

Who is Jetty Mortbay and why does he want inside my root CA store?

Who is Jetty Mortbay and why does he want inside my root CA store?

I remarked back in April about this unfortunate problem in a post about an otherwise-flawless & easy Nimble implementation:

The SSL cert situation is embarrassing and I’m glad my former boss hasn’t seen it. Namely that situation is this: you can’t replace the stock cert, which, frankly looks like something I would do while tooling around with OpenSSL in the lab.

I understand this is fixed in the new 2.x OS version but holy shit what a fail.

Well, fail-file no more,  because my new Nimble array at my current job has been measured and validated by the CA Gods:

verified

Green padlocks. I want green padlocks everywhere

Oh yeah baby. Validated in Chrome, Firefox and IE. And it only cost me market rates for a SAN certificate from a respected CA, a few hours back ‘n forth with Nimble, and only a few IT McGuyver-style tricks to get this outcome.

Now look. I know some of my readers are probably seeing this and thinking…”But that proves nothing. A false sense of security you have.”

Maybe you’re right, but consider.

I take a sort of Broken Windows Theory approach to IT. The Broken Windows Theory, if you’re not familiar with it, states that:

Under the broken windows theory, an ordered and clean environment – one which is maintained – sends the signal that the area is monitored and that criminal behavior will not be tolerated. Conversely, a disordered environment – one which is not maintained (broken windows, graffiti, excessive litter) – sends the signal that the area is not monitored and that one can engage in criminal behavior with little risk of detection.

Now I’m not saying that adding a proper certificate to my behind-the-firewall Nimble array so that Chrome shows me Green Padlocks rather than scary warnings is akin to reducing violent crime in urban areas. But I am saying that little details, such as these, ought to be considered and fixed in your environment.

Why? Well, somehow fixing even little things like this amount to something more than just good hygiene, something more than just ‘best practice.’

Ultimately, we infrastructurists are what we build, are we not? Even little ‘security theater’ elements like the one above are a reflection on our attention to detail, a validation of our ability to not only design a resilient infrastructure on paper at the macro level, but to execute on that design to perfection at the micro level.

It shows we’re not lazy as well, that we care to repair the ‘broken windows’ in our environment.

And besides: Google (and Microsoft & Mozilla & Apple) are right to call out untrusted certificates in increasingly disruptive & work-impairing ways.

*If you’re reading this and saying: Why don’t you just access the array via IP address, well, GoFQDNorGoHomeSon.com

Containers! For Windows! Courtesy of Docker

DockerWithWindowsSrvAndLinux-1024x505 (1)

Big news yesterday for fans of agnostic cloud/on-prem computing.

Docker -the application virtualization stack that’s caught on like wildfire among the *nix set- is coming to Windows.

Yeah baby.

Mary Jo with the details:

Under the terms of the agreement announced today, the Docker Engine open source runtime for building, running and orchestrating containers will work with the next version of Windows Server. The Docker Engine for Windows Server will be developed as a Docker open source project, with Microsoft participating as an active community member. Docker Engine images for Windows Server will be available in the Docker Hub. The Docker Hub will also be integrated directly into Azure so that it is accessible through the Azure Management Portal and Azure Gallery. Microsoft also will be contributing to Docker’s open orchestration application programming interfaces (APIs).

When I first heard the news, emotion was mixed.

On the one hand, I love it. Virtualization of all flavors -OS, storage, network, and application- is where I want to be, as a blogger, at home in my lab, and professionally.

Yet, as a Windows guy (I dabble, of course), Docker was just a bit out of reach for me, even with my lab, which is 100% Windows.

On the other hand, I also remembered how dreadful it used to be to run Linux applications on Windows. Installing GTK+ Libraries on Windows isn’t fun, and the end-result often isn’t very attractive. In my world, keeping the two separate on the application & OS side/uniting them via Kerberos and/or https/rest has always been my preference.

But that’s old world thinking, ladies and gentlemen.

Because you see, this announcement from Microsoft & Docker Inc sounds deep, rich, functional. Microsoft’s going to contribute some of its Server code to the Docker folks, and the Docker crew will help build Container tech into Windows Server and Azure. I’m hopeful Docker will just be another Role in Server, and that Jeffrey Snover’s powershell cmdlets will hook deep into the Docker stuff.

This probably marks the death of App-V, which I wrote about in comparison to Docker just last month, but that’s fine with me.

Docker on Windows marks a giant step forward for Agnostic Computing…do we dare imagine a future in which our application stacks are portable? Today I’m running an application in a Docker Container on Azure, and tomorrow I move it to AWS?

Microsoft says that’s exactly the vision:

Docker is an open source engine that automates the deployment of any application as a portable, self-sufficient container that can run almost anywhere. This partnership will enable the Docker client to manage multi-container applications using both Linux and Windows containers, regardless of the hosting environment or cloud provider. This level of interoperability is what we at MS Open Tech strive to deliver through contributions to open source projects such as Docker.

Full announcement.

Microsoft releases new V2V and P2V tool

Do you smell what I smell?

Inhale it boys and girls because what you smell is the sweet aroma of VMware VMs being removed from the vSphere collective and placed into System Center & Hyper-V’s warm embrace.

Microsoft has released version three of its V2V and P2V assimilator tool:

Today we are releasing the Microsoft Virtual Machine Converter (MVMC) 3.0, a supported, freely available solution for converting VMware-based virtual machines and virtual disks to Hyper-V-based virtual machines and virtual hard disks (VHDs).

With the latest release, MVMC 3.0 adds the ability to convert a physical computer running Windows Server 2008 or above, or Windows Vista or above to a virtual machine running on a Hyper-V host (P2V).

This new functionality adds to existing features available including:

• Native Windows PowerShell capability that enables scripting and integration into IT automation workflows.
• Conversion and provisioning of Linux-based guest operating systems from VMware hosts to Hyper-V hosts.
• Conversion of offline virtual machines.
• Conversion of virtual machines from VMware vSphere 5.5, VMware vSphere 5.1, and VMware vSphere 4.1 hosts to Hyper-V virtual machines.

Download available here.

This couldn’t have come at a better time for me. At work -which is keeping me so busy I’ve been neglecting these august pages- my new Hyper-V cluster went Production in mid-September and has been running very well indeed.

But building a durable & performance-oriented virtualization platform for a small to medium enterprise is only 1/10th of the battle.

If I were a consultant, I’d have finished my job weeks ago, saying to the customer:

Right. Here you go lads: your cluster is built, your VMM & SCCM are happy, and the various automation bits ‘n bobs that make life in Modern IT Departments not only bearable, but fun, are complete

But I’m an employee, so much more remains to be done. So among many other things, I now transition from building the base of the stack to moving important workloads to it, namely:

  • Migrating and/or replacing important physical servers to the new stack
  • Shepherding dozens of important production VMs out of some legacy ESXi 5 & 4 hosts and into Hyper-V & System Center and thence onto greatness

So it’s really great to see Microsoft release a new version of its tool.

Going full Windows 10 Server in the Lab, part 1

So many new goodies in Windows Server 10.

So little time to enjoy them.

Highlights so far:

  • Command line transparency is awesome. Want the same in my Powershell windows
  • Digging the flat look of my Windows when they are piled atop one another. THere’s a subtle 3d effect (really muted shadows I think) that helps to highlight Window positions and focus. Nice work UI team
  • Server 10 without Desktop mode looks just about 100% like Server 2012 R2. So yeah, if you’re using your PC as a server, definitely install the Desktop mode

On the agenda for today:

  • Build what has to be one of the few Windows Server 10 Hyper-V clusters
  • Install the new VMM & System Center
  • Testing out the new Network Controller role on a 1U AMD-powered server I’ve had powered-off but ready for just this moment (never got around to building a Server 2012 R2 Network Virtualization Gateway server)
  • Maybe, just maybe, upgrading the two Domain Controllers and raising forest/domain functional level to “Technical Preview”, if it’s even possible.

What won’t be upgraded in the short term:

  • San.daisettalabs.net, the Tiered Storage box that hosts my SMB 3 shares as well as several iSCSI .vhdx drives
  • The VM hosting SQL 2012 SP2, IPAM, and other roles
  • The TV computer, which is running Windows 8.1 Professional with Media Center Edition. Yes, it’s a lab, but even in a lab environment, television access is considered mission critical

More later.

Microsoft to introduce the New Shiny Windows

Devoted readers of Agnostic Computing.com, I write today to implore you to set your powershell scripts to Signed, get your Windows Key + R trigger fingers ready, and prep your forests and domains for a functional upgrade because today ladies and gentlemen, today, we get a new Windows. 

Ahhh yeah.

There’s some excitement in Microsoft Country again.

No one knows what it’ll be called. Windows 9 is the front-runner, but late-breaking rumors say big MS could throw us for a loop too and name it Windows TH (Threshold?!?! the pundits echo) or just plain old Windows.

It's always a good day when a new Windows is detailed

It’s always a good day when a new Windows is detailed

I say they should name it Windows TNS: Windows The New Shiny. Because among the rumors I’ve enjoyed hearing most is the one Microsoft may offer a sort of Windows 365 subscription for fanbois like me, a continuously morphing and changing OS, just like my O365 experience has been. New Shiny Windows every month…well maybe I’d tell ConfigMan to delay updates for a week or so, just to shake the bugs loose. But still. A subscription OS would be great.

But that’s a long-shot and probably not a very strong selling point for today’s event, which is, as everyone has noted, focused entirely on enterprise computing.

You see, Microsoft is trying desperately to court Enterprise IT people, to bring us back into the fold, targeting this entire event today at IT people like me who were aghast & horrified two years ago when they first installed Windows 8 in a VM.

“No. No. To get to start screen, hover your mouse in the lower corner. The lower corner, not the charms bar.There it is. Click that. Ahh shit, you missed it. Try again.” was how the conversation went throughout IT departments in ‘Merica.

As I’ve written before, the experience of Windows 8 & Server 2012 was so shocking and painful, it sent me running and crying into the Mac OS X camp, and then into ChromeBook fantasyland.

But I got over it. I overcame, and I figured out how to move all that nonsense touch stuff away when Windows 8.1/Server 2012 R2 debuted about a year ago.

Apparently other IT pros haven’t, and are still sticking to Windows 7 as if it’s the greatest thing since Active Directory. Thus today’s event.

To them I say: get with the program, or get left behind. Windows 8 did suck, but 8.1 & 2012 R2 were fine recoveries. If you decided to punt on learning about Windows 8.1/2012 R2, you missed a whole bunch of incredible advancements that are only going to improve with Windows TNS. Have fun catching up on this:

  • Baked in Hyper-V. Free on Windows 8.1 Pro and up. A virtual lab on every desktop.
  • Tiered Storage Spaces in Windows server 2012 R2: yet another software abstraction framework, but for your storage! You missed out on this too!
  • An awesome networking stack, totally rewritten: Native support for teaming, network function virtualizations, Layer 3 routing protocols via PowerShell…oh my. I’d hate to be you stuck with a Server 2008 R2 box, running your old tired batch files, your dated vbs scripts and ipconfig. You missed out on some incredible advancements

And the great thing is that all this is going to get better, I think (hope). True, we won’t be learning about Windows Server today (Aidan Finn reckons that + nextgen System Center will be next month) but there will be lots of detail about our next Enterprise desktop product, by which you can bet people like me will make inferences for the next server product.

Things are looking up in Microsoft Country. We’ve a ten year head start on Trustworthy Computing (ShellShock couldn’t have had better timing for MS), a highly-modular & secure OS, a mature cloud stack, a SaaS offering second to-none (O365) and now, today, a new Windows OS.

Good times.

2014 is the Year the Application broke free

It’s only September but I feel like 2014 is the year the Application broke free -was liberated if you will- from the infrastructure beneath it.

“What?!? That’s crazy talk,” the two of you who read this say in response.

Maybe, but consider this.

Untitled pictureDocker is huge and growing in popularity. As the drip-drip of my Enterprise RSS feed attests, interest in Docker has been growing steadily all year long; at times it seems there’s more Docker hype than even SDN hype. People seem as excited about Docker as they were about x86 virtualization, way back in the day. It’s real and it’s something.

But what is it?

The Infrastructurist in me cries a little bit to admit this, but it’s about time the Application was unyoked from the OS, the spindles, the network and the compute beneath it. And that’s what Docker does.

You see, Docker positions itself as a “container” (shipping metaphors abound in this space, which is just great) for your apps; once safely ensconsed in a container, your app, or more properly your entire application supply chain, can be moved from platform to platform with virtually no configuration changes, downtime, or dependency on your infrastructure guy.

That’s right baby. If you’re a Linux guy, no more fiddling around with Ubuntu VMs, BSD jails, standing up dev VMs or any of that nonsense. Docker takes your Ruby/Java/Objective C application, your backend MySQL DB or your NoSQL MongoDB, and your Android / IOS app, containerizes it, and lets you run it on just about anything capable of issuing a ping command. It makes it super-simple to go from dev to test to production, and it’s all free and made of wholesome open source stuff.

Neat bit of tech, wouldn’t you say?

Little boxes, full of apps, little boxes full of streaming apps!

Little boxes, full of apps, little boxes full of streaming apps!

In the Microsoft kingdom, I was doing similar things with App-V about two years ago. App-V is similar in concept to Docker, though admittedly App-V was never built to allow cross-platform application migration, and it’s not as all-encompassing as Docker. Yet, one can’t help but draw comparisons.

App-V is, how shall we say, a bit less elegant. Simple to install and build-out, but a bit clumsy in execution, App-V works by more or less capturing what an application does to an underlying Windows system. Por ejemplo: take your average ordinary WIndows executable (I’ll pick Spotify, as that’s what came up first in Task Manager).

To virtualize Spotify, all one has to do (after standing up a an App-V VM) is tell App-V to capture all the things spotify_installer.exe  does to a Windows system. Once finished, you signal to App-V that it should build a virtualized app for Spotify, and then, voila! You can “stream” the Spotify app -plus all the registry bits ‘n bobs it changed, the dll libraries it created/modified, and all that stuff- down to your Windows clients, where it will have zero interaction with potentially hostile local registry entries or dlls.

Two years ago, this was hot stuff and I was gung-ho on Application Virtualization to the point where I was thinking server virtualization wasn’t long for this world. Who wants to virtualize an OS when you can simply virtualize & delivery the application?

Sadly, App-V seems to have been placed on the “Neglect” shelf at Microsoft. It’s still around (still running on Silverlight too), but you don’t hear about it much anymore. I want to use it, but I can’t commit.

Microsoft’s taking a different approach to de-coupling the app from its OS, it seems to me: Universal binaries. At least that was the pre-Satya Nadella thinking, but it’s very close to realization. Saddle on up next to your favorite Visual Studio developer and ask him to show you how it works: you can build one codebase and compile it for multiple platforms. It’s almost to the point where you do a “Save As IOS App” then a “Save as Android App” and, of course,  “Save as Windows app.” Awesome!

One could argue this is the ultimate app endgame…universal binaries that run on any OS would seem to be a more elegant solution than even Docker’s portability.

And then there’s Google. In the last two weeks, Google has announced that some Android applications can now run on ChromeOS. Sharp devs took very little time to expand the library of Android apps you can run and touch on a standard Chromebook.

Then they took it a step further- some devs have gotten Android apps to run on Chrome within Windows, a feat of software engineering so amazing, I think it deserves one of the same fancy nicknames Google used to employ in its financial engineering efforts (“a reverse double starbuck” comes to mind).

This is an exciting space, way more exciting and real than SDN, I’d reckon. Whether you want to virtualize your application by containerizing its infrastructure, building a binary that runs on everything, or going full Trojan Horse with Browser-as-a-Platform strategy, there’s real movement and change afoot.

And it’s all aimed at making computing more agnostic, if you’ll forgive the self-referential plug.

Forget infrastructure…How long until we stop talking about Operating Systems at all?

Favorite IT metaphors, sayings & aphorisms

In my Linkedin Profile, I write that I’m a fan of “elaborate IT Metaphors” yet, in a very literal way, I’ve never actually written a list of my favorites.

Listing out my favorite IT metaphors, sayings, aphorisms and such is risky. Too much pithiness, and I risk not being taken seriously. Too much cynicism and no one wants to talk to you.

And yet I must take that risk, because if you’re a practitioner of the IT arts as I am, then you’re used to engaging in these sorts of thoughtful/silly/humorous reflections.

Some of the metaphors/thoughts below are funny. Some are so terrible I’m embarassed to write them. Others are good, but could be better. There’s no real organization to them, but I hope you find them useful nonetheless.

Enough. On with the metaphors, sayings & aphorisms!

Metaphor Meaning Origin/Notes
Dark side of the moon Waiting for a host or device to reply to pings after reload/reboot NASA obviously
Eat our own dogfood Applying same policies/tech/experience to IT that apply to users Not sure but heard on TWiT
DNS is like a phonebook Computers speak in numbers humans speak in words My own metaphor to explain DNS problems
Fat finger A stupid mistake in perhaps an otherwise solid plan (eg IP address keyed incorrectly) Former boss/Homer Simpson?
Go FQDN or Go Home Admonishment to correct lazy IT tendency to code/build with IP addresses rather than FQDN My own
Garbage in Garbage Out You get out of a system that which you put in Unknown but s/he was brilliant
This ____ is like a princess A server or service that is high-profile/important but prone to failure and drama without constant attention My own
Cadillac Solution A high-priced solution to a problem that may require only ingenuity/dilligence My own but really…Cadillac…I'm so old
The Sun Never Sets on Infrastructure A reference to the 24/7 nature of Infrastructure stack demand by way of the British Empire I used this metaphor extensively in last job
Infrastructure is Roads/Applications are cars/Users are drivers Reference to the classic split in IT departments Former colleague
Two Houses Both Alike in Dignity Another reference to AppDev & Infrastructure divide in IT My own liberal abuse of Shakespeare's opening line in R&J
Child Partition/Parent Partition Reference to me and my son in light of Hypervisor technology My own
Supervisor Module Reference to wife-as-Sup7203B Linecard in 650xE chassis switch The packets flow where she tells them to flow
Code is poetry There is something more to technology than just making things work Google but adapted by me for scripting and configs
Going full Fibonacci The joy & euphoria inherent in a well-designed subnetting plan wherein octets are harmonized & everything just fits My own abuse of the famed Fibonacci Sequence which honestly has nothing to do with IP subnetting and more to do with Dan Brown. Also applies to MAC address pools because encoding your MAC address pools is fun
Dystopian IT A poor but descriptive term to describe dysfunctional IT departments My own I think
When I was a Child I thought as a Child How I defend poor technical decisions that haunt me years later A (perhaps blasphemous) homage to St. Paul
There are three ____ and the greatest of these is ____ Another St. Paul reference Useful in IT Purchasing decisions
IT White Whale Highly technical problems I haven't solved yet and obsess over Borrowed from Herman Melville's Moby Dick

Ballad of the Converged IT Guy

The inestimable Greg Ferro once said that what’s needed in modern IT are men & women whose skillsets are shaped like a capital “T.”

“You’ve got to have broad experience and familiarity with various technologies, see? That’s the wide part at the top of the T. And then you’ve got to be deep on some things in your portfolio. Maybe it’s storage, or WAN or who the hell knows?. That’s the leg of the T. Right?”

Right you are Greg.

I’m paraphrasing of course, but Ferro’s description of good IT Guys (or Gals) appeals to me because it more or less describes my career in IT: I’ve touched lots of tech and gone deep in a few things.

I have breadth and depth in my portfolio, in other words. Yay me and yay for confirmation bias!

Unfortunately for me, this reality -while good in Greg’s eyes perhaps- usually results in me being labeled with the contemptible catch-all “IT Generalist.”

“You’re sort of an IT Generalist,” the recruiter says. “Is that fair?”

Sigh.

“Yes. I’m a systems guy, but really, an IT Generalist works too,” I reply.

This is how I reluctantly describe myself to others and even on the About the Author Page.

But I hate that term, “Generalist…” it’s too prosaic, too generic, too….general. I want it banished and replaced, and I don’t want to be known as an IT Generalist.

So what to replace it with? Re-writing the About the Author page with “Hi, I’m Jeff Wilson, a T-shaped IT Guy” doesn’t exactly inspire confidence and might make the reader question my sanity. Systems Engineer is nice, but doesn’t hint at my rudimentary skills at herding packets with DSCP values intact across a WAN, does it?

What I need -no, what we Generalists need!- is some sizzle for our T-shaped career story boards. We need to make IT Generalism seem sexy, without using that lame g word. What we need is a way to converge all our skills -broad and deep- into one smart, market-aware, cloud-hip, fully-qualified and routable term that…

hey wait a second.

converge.

Converged IT Guy.

That’s it.

Contains Silicon Valley buzzword? Check.

Easy to remember? Check.

Clever, and only with a little bit of smart-ass spunk? Check.

Descriptive? Not really, but better than Generalist.

Done.

I’m a Converged IT Guy. And this is my ballad.

Ballad of the Converged IT Guy

 I’ve touched lots of tech, from VoIP to SQL,

the LAMP stack & PowerShell

I don’t fear multicast or spanning tree

I once wrote a Valentine’s to LACP 

Yay though, I’m a Converged IT Guy

Block, file, object, LUNs and Vols, NFS,

but wack-wack filesharings the best
seen every file extension from east to west
Kilo, Mega, Giga,Tera 
I dedupe, replicate and compress

Yay though, I’m a Converged IT Guy

Gone deep on storage and virtualization

but change out the tapes from time to time

From Voice and an analog PBX

to Layer 4 Load Balancing and Cisco’s FEX

Yay though, I’m a Converged IT Guy

ITIL, HIPPA, PCI & SOX

Waterfall, Agile and now DevOps

Declarative, Imperative

Concatenate, quiesce, compile

Yay Though, I’m a Converged IT Guy

Lo, the whiteboard is my kryptonite

and IT Siloes are my enemy

Yay Though, I’m a Converged IT Guy

Thoughts on EVO:RAIL

So if you work in IT, and even better, if you’re in the virtualization space of IT as I am, you have to know that VMworld is happening this week.

VMworld is just about the biggest vCelebration of vTechnologies there is. Part trade-show, part pilgrimage, part vLollapalooza, VMworld is where all the sexy new vProducts are announced by VMware, makers of ESXi, vSphere, vCenter, and so many other vThings.

It’s an awesome show…think MacWorld at the height of Steve Jobs but with fewer hipsters and way more virtualization engineers. Awesome.

And I’ve never been :sadface:

And 2014’s VMworld was a doozy. You see, the vGiant announced a new 2U, four node vSphere & vSAN cluster-in-a-box hardware device called EVO:RAIL. I’ve been reading all about EVO:RAIL for the last two days and here’s what I think as your loyal Hyper-V blogger:

  • What’s in a name? Right off the bat, I was struck by the name for this appliance. EVO:RAIL…say what? What’s VMware trying to get across here? Am I to associate EVO with the fast Mitsubishi Lancers of my youth, or is this EVO in the more Manga/Anime sense of the word? Taken together, EVO:RAIL also calls to mind sci-fi, does it not? You could picture Lt. Cmdr Data talking about an EVO:RAIL to Cmdr Riker, as in “The Romulan bird of prey is outfitted with four EVO:RAIL phase cannons, against which the Enterprise’s shields stand no chance.” Speaking of guns: I also thought of the US Navy’s Railguns; long range kinetic weapons designed to destroy the Nutanix/Simplivity the enemy.
  • If you’re selling an appliance, do you need vExperts? One thing that struck me about VMware’s introduction of EVO:RAIL was their emphasis on how simple it is to rack, stack, install, deploy and virtualize. They claim the “hyper-converged” 2U box can be up and running in about 15 minutes; a full rack of these babies could be computing for you in less than 2 hours. evo1They’ve built a sexy HTML 5 GUI to manage the thing, no vSphere console or PowerCLI in sight. It’s all pre-baked, pre-configured, and pre-built for you, the small-to-medium enterprise. It’s so simple a help desk guy could set it up. So with all that said, do I still need to hire vExperts and VCDX pros to build out my virtualization infrastructure? It would appear not. Is that the message VMware is trying to convey here?
  • One SKU for the Win: I can’t be the only one that thinks buying the VMware stack is a complicated & time-consuming affair. Chris Wahl points out that EVO:RAIL is one SKU, one invoice, one price to pay, and VMware’s product page confirms that, saying you can buy a Dell EVO:RAIL or a Fujitsu EVO:RAIL, but whatever you buy, it’ll be one SKU. This is really nice. But why? VMware is famous for licensing its best-in-class features…why mess with something that’s worked so well for them?
    Shades of Azure simplicity here

    Shades of Azure simplicity here

    One could argue that EVO:RAIL is a reaction to simplified pricing structures on rival systems…let’s be honest with ourselves. What’s more complicated: buying a full vSphere and/or vHorizon suite for a new four node cluster, or purchasing the equivalent amount of computing units in Azure/AWS/Google Compute? What model is faster to deploy, from sales call to purchasing to receiving to service? What model probably requires consulting help?

    Don’t get me wrong, I think it’s great. I like simple menus, and whereas buying VMware stuff before was like choosing from a complicated, multi-page, multi-entree menu, now it’s like buying burgers at In ‘n Out. That’s very cool, but it means something has changed in vLand.

  • I love the density: As someone who’s putting the finishing touches on my own new virtualization infrastructure, I love the density in EVO:RAIL. 2 Rack Units with E5-26xx class Xeons packing 6 cores each means you can pack about 48 cores into 2U! Not bad, not bad at all. The product page also says you can have up to 16TB of stroage in those same 2U (courtesy of VSAN) and while you still need a ToR switch to jack into, each node has 2x10GbE SFP+ or Copper. Which is excellent. RAM is the only thing that’s a bit constrained; each node in an EVO:RAIL can only hold 192GB of RAM, a total of 768GB per EVO:RAIL.In comparison, my beloved 2U pizza boxes offer more density in some places, but less overall, given than 1 Pizza Box = one node. In the Supermicros I’m racking up later this week, I can match the core count (4×12 Core E5-46xx), improve upon the RAM (up to 1TB per node) and easily surpass the 16TB of storage. That’s all in 2U and all for about $15-18k.Where the EVO:RAIL appears to really shine is in VM/VDI density. VMware claims a single EVO:RAIL is built to support 100 General Purpose VMs or to support up to 250 VDI sessions, which is f*(*U#$ outstanding.
  • I wonder if I can run Hyper-V on that: Of course I thought that. Because that would really kick ass if I could.

Overall, a mighty impressive showing from VMware this week. Like my VMware colleagues, I pine for an EVO:RAIL in my lab.

I think EVO:RAIL points to something bigger though…This product marks a shift in VMware’s thinking, a strategic reaction to the changes in the marketplace. This is not just a play against Nutranix and other hyper-converged vendors, but against the simplicity and non-specialist nature of cloud Infrastructure as a Service.  This is a play against complexity in other words…this is VMware telling the marketplace that you can have best-in-class virtualization without worst-in-class licensing pain and without hiring vExperts to help you deploy it.

Tales from the Hot Lane

A few brief updates & random thoughts from the last few days on all the stuff I’ve been working on.

Refreshing the Core at work: Summer’s ending, but at work, a new season is advancing, one rack unit at a time. I am gradually racking up & configuring new compute, storage, and network as it arrives; It Is Not About the Hardware™, but since you were wondering: 64 Ivy Bridge cores and about 512GB RAM, 30TB of storage, and Nexus 3k switching.

Cisco_logoAhh, the Nexus line. Never had the privilege to work on such fine switching infrastructure. Long time admirer, first-time NX-OS user. I have a pair of them plus a Layer 3 license so the long-term thinking involves not just connecting my compute to my storage, but connecting this dense stack northbound & out via OSPF or static routes over a fault-tolerant HSRP or VRRP config.

To do that, I need to get familiar with some Nexus-flavored acronyms that aren’t familiar to me: virtual port channels (VPC), Control Plane policy (COPP), VRF, and oh-so-many-more. I’ll also be attempting to answer the question once and for all: what spanning tree mode does one use to connect a Nexus switch to a virtualization host running Hyper-V’s converged switching architecture? I’ve used portfast in the lab on my Catalyst, but the lab switch is five years old, whereas this Nexus is brand new. And portfast never struck me as the right answer, just the easy one.

To answer those questions and more, I have TAC and this excellent tome provided gratis by the awesome VAR who sold us much of the equipment.

Into the vCPU Blender goes Lync: Last Friday, I got a call from my former boss & friend who now heads up a fast-growing IT department on the coast. He’s been busy refreshing & rationalizing much of his infrastructure as well, but as is typical for him, he wants more. He wants total IT transformation, so as he’s built out his infrastructure, he laid the groundwork to go 100% Microsoft Lync 2013 for voice.

Yeah baby. Lync 2013 as your PBX, delivering dial tone to your endpoints, whether they are Bluetooth-connected PC headsets, desk phones, or apps on a mobile.

Forget software-defined networking. This is software-defined voice & video, with no special server hardware, cloud services, or any other the other typical expensive nonsense you’d see in a VoIP implementation.

If Lync 2013 as PBX is not on your IT Bucket List, it should be. It was something my former boss & I never managed to accomplish at our previous employer on Hyper-V.

Now he was doing it alone. On a fast VMware/Nexus/NetApp stack with distributed vSwitches. And he wanted to run something by me.

So you can imagine how pleased I was to have a chat with him about it.

He was facing one problem which threatened his Go Live date: Mean Opinion Score, or MOS, a simple 0-5 score Lync provides to its administrators that summarizes call quality. MOS is a subset of a hugely detailed Media Quality Summary Report, detailed here at TechNet.

thMy friend was scoring a .6 on his MOS. He wanted it to be at 4 or above prior to go-live.

So at first we suspected QoS tags were being stripped somewhere between his endpoint device and the Lync Mediation VM. Sure enough, Wireshark proved that out; a Distributed vSwitch (or was it a Nexus?) wasn’t respecting the tag, resulting in a sort of half-duplex QoS if you will.

He fixed that, ran the test again, and still: .6. Yikes! Two days to go live. He called again.

That’s when I remembered the last time we tried to tackle this together. You see, the Lync Mediation Server is sort of the real PBX component in Lync Enterprise Voice architecture. It handles signalling to your endpoints, interfaces with the PSTN or a SIP trunk, and is the one server workload that, even in 2014, I’d hesitate making virtual.

My boss had three of them. All VMs on three different VMware hosts across two sites.

I dug up a Microsoft whitepaper on virtualizing Lync, something we didn’t have the last time we tried this. While Redmond says Lync Enterprise Voice on top of VMs can work, it’s damned expensive from a virtualization host perspective. MS advises:

  • You should disable hyperthreading on all hosts.
  • Do not use processor oversubscription; maintain a 1:1 ratio of virtual CPU to physical CPU.
  • Make sure your host servers support nested page tables (NPT) and extended page tables (EPT).
  • Disable non-uniform memory access (NUMA) spanning on the hypervisor, as this can reduce guest performance.

Talk about Harshing your vBuzz. Essentially, building Lync out virtually with Enterprise Voice forces you to go sparse on your hosts, which is akin to buying physical servers for Lync. If you don’t, into the vCPU blender goes Lync, and out comes poor voice quality, angry users, bitterness, regret and self-punishment.

Anyway, he did as advised, put some additional vCPU & memory reservations in place on his hosts, and yesterday, whilst I was toiling in the Hot Lane, he called me from Lync via his mobile.

He’s a married man just like me, but I must say his voice sounded damn sexy as it was sliced up into packets, sent over the wire, and converted back to analog on my mobile’s speaker. A virtual chest bump over the phone was next, then we said goodbye.

Another Go Live Victory (by proxy). Sweet.

Azure Outage: Yesterday’s bruising hours-long global Azure outage affected Virtual Machines, storage blobs, web services, database services and HD Insight, Microsoft’s service for big data crunching. As it unfolded, I navel-gazed, when I felt like helping. There was literally nothing I could do. Had I some crucial IaaS or PaaS in the Azure stack, I’d be shit out of luck, just like the rest. I felt quite helpless; refreshing Mary Jo’s pageyellow-exclamation-mark-in-triangle-md and the Azure dashboard didn’t help. I wondered what the problem was; it’s been a difficult week for Microsofties whether on-prem or in Azure. Had to be related to the update cycle, I thought.

On the plus side, Azure Active Directory services never went down, nor did several other services. Office 365 stayed up as well, though it is built atop separate-but-related infrastructure in my understanding.

Lastly, I pondered two thoughts: if you’re thinking of reducing your OpEx by replacing your DR strategy with an Azure Site Recovery strategy, does this change your mind? And if you’re building out Azure as your primary IaaS or PaaS, do you just accept such outages or do you plan a failback strategy?

Labworks : Towards a 100% Windows-defined Daisetta Lab: What’s next for the Daisetta Lab? Well, I have me an AMD Duron CPU, a suitable motherboard, a 1U enclosure with PSU, and three Keepin’ it RealTek NICs. Oh, I also have a case of the envies, envies for the VMware crowd and their VXLAN and NSX and of course VMworld next week. So I’m thinking of building a Network Virtualization Gateway appliance. For those keeping score at home, that would mean from Storage to Compute to Network Edge, I’d have a 100% Windows lab environment, infused with NVGRE which has more use cases than just multi-tenancy as I had thought.

Stack Builders ‘R Us

This is a really lame but (IMHO) effective drawing of what I think of as a modern small/medium business enterprise ‘stack':

stack

As you can see, just about every element of a modern IT is portrayed.

Down at the base of the pyramid, you got your storage. IOPS, RAID, rotational & ssd, snapshots, dedupes, inline compression, site to site storage replication, clones and oh me oh my…all the things we really really love are right here. It’s the Luntastic layer and always will be.

Above that, your compute & Memory. The denser the better, 2U Pizza Boxes don’t grow on trees and the business isn’t going to shell out more $$$ if you get it wrong.

Above that, we have what my networking friends would call the “Underlay network.” Right. Some cat 6, twinax, fiber, whatever. This is where we push some packets, whether to our storage from our compute, northbound out to the world, southbound & down the stack, or east/west across it. Leafs, spines, encapsulation, control & data planes, it’s all here.

And going higher -still in Infrastructure Land mind you- we have the virtualization layer. Yeah baby. This is what it’s all about, this is the layer that saved my career in IT and made things interesting again. This layer is designed to abstract all that is beneath it with two goals in mind: cost savings via efficiency gains & ease of provisioning/use.

And boy,has this layer changed the game, hasn’t it?

So if you’re a virtualization engineer like I am, maybe this is all you care about. I wouldn’t blame you. The infrastructure layer is, after all, the best part of the stack, the only part of the stack that can claim to be #Glorious.

But in my career, I always get roped in (willingly or not) into the upper layers of the stack. And so that is where I shall take you, if you let me.

Next up, the Platform layer. This is the layer where that special DBA in your life likes to live. He optimizes his query plans atop your Infrastructure layer, and though he is old-school in the ways of storage, he’s learned to trust you and your fancy QoS .vhdxs, or your incredibly awesome DRS fault-tolerant vCPUs.

Or maybe you don’t have a DBA in your Valentine’s card rotation. Maybe this is the layer at which the devs in your life, whether they are running Eclipse or Visual Studio, make your life hell. They’re always asking for more x (x= memory, storage, compute, IP), and though they’re highly-technical folks, their eyes kind of glaze over when you bring up NVGRE or VXLAN or Converged/Distributed Switching or whatever tech you heart at the layer below.

Then again, maybe you work in this layer. Maybe you’re responsible for building & maintaining session virtualization tech like RDS or XenApp, or maybe you maintain file shares, web farms, or something else.

Point is, the people at this layer are platform builders. To borrow from the automotive industry, platform guys build the car that travels on the road infrastructure guys build. It does no good for either of us if the road is bumpy or the car isn’t reliable, does it? The user doesn’t distinguish between ‘road’ and ‘car’, do they? They just blame IT.

Next up: software & service layer. Our users exist here, and so do we. Maybe for you this layer is about supporting & deploying Android & iPhone handsets and thinking about MDM. Or maybe you spend your day supporting old-school fat client applications, or pushing them out.

And finally, now we arrive to the top of the pyramid. User-space. The business.

This is where (and the metaphor really fits, doesn’t it?) the rubber meets the road ladies and gentlemen. It’s where the business user drives the car (platform) on the road (infrastructure). This is where we sink or swim, where wins are tallied and heros made, or careers are shattered and the cycle of failure>begets>blame>begets>fear>begets failure begins in earnest.

That’s the stack. And if you’re in IT, you’re in some part of that stack, whether you know it or not.

But the stack is changing. I made a silly graphic for that too. Maybe tomorrow.

Respect my Certificate Authoritah

Fellow #VFD3 Delegate and Chicago-area vExpert Eric Shanks has recently posted two great pieces on how to setup an Active Directory Certificate Authority in your home lab environment.

Say what? Why would you want the pain of standing up some certificate & security infrastructure in your home lab?

Eric explains:

Home Lab SSL Certificates aren’t exactly a high priority for most people, but they are something you might want to play with before you get into a production environment.

Exactly.

Security & Certificate infrastructure are a weak spot in my portfolio so I’ve been practicing/learning in the Daisetta Lab so that I don’t fail at work. Here’s how:

As I was building out my lab, I knew three things: I wanted a routable Fully Qualified Domain Name for my home lab, I was focused on virtualization but should also practice for the cloud and things like ADFS, and I wanted my lab to be as secure as possible (death to port 80 & NTLM!)

With those loose goals in mind, I decided I wanted Daisetta Labs.net to be legit. To have some Certificate Authority bonafides…to get some respect in the strangely federated yet authoritarian world of certificate authorities, browser and OS certificate revocations, and yellow Chrome browser warning screens.

dlabs

Too legit, too legit to quit

So I purchased a real wildcard SSL certificate from a real Certificate Authority back in March. It cost about $96 for one year, and I don’t regret it at all because I’m using it now to secure all manner of things in Active Directory, and I’ll soon be using it as Daisetta Labs.net on-prem begins interfacing with DaisettaLabs.net in Azure (it already is, via Office 365 DirSync, but I need to get to the next level and the clock is ticking on the cert).

Building on Eric’s excellent posts, I suggest to any Microsoft-focused IT Pros that you consider doing what I did. I know it sucks to shell out money for an SSL certificate, but labwork is hard so that work-work isn’t so hard.

So, go follow Eric’s outline, buy a cert, wildcard or otherwise (got mine at Comodo, there’s also an Israeli CA that gives SSL certs for free, but it’s a drawn-out process) and stand up a subordinate CA (as opposed to a on-prem only Root CA) and get your 443 on!

Man it sucks to get something so fundamentally wrong. Reader Chris pointed out a few inaccuracies and mistakes about my post in the comments below.

At first I was indignant, then thoughtful & reflective, and finally resigned. He’s right. I built an AD Root -not a subortinate as that’s absurd- Certificate Authority in the lab.

Admittedly, I’m not strong in this area. Thanks to Chris for his coaching and I regret if I mislead anyone.

#StorageGlory Achieved : 30 Days on a Windows SAN

 Behold, these three remain. File. Block. Object. And the greatest of these is block.  – Sr. Systems Engineer St. Paul, in a letter to confused storage engineers in Thessalonika

Right. So a couple weeks back I teased the hardware specs of the new storage array I built for the Daisetta Lab at home.

Software-defined. x86. File and block. Multipath. Intel. And some Supermicro. Storage utopia up in the Daisetta Lab

Software-defined. x86. File and block. Multipath. Intel. And some Supermicro. Storage utopia up in the Daisetta Lab

My idea was to combine all types of disks -rotational 3.5″ & 2.5″ drives, SSDs, mSATAs, hell, I considered USB- into one tight, well-built storage box for my lab and home data needs. A sort of Storage Ark, if you will; all media types were welcome, but only if they came in twos (for mirroring & Parity sake, of course) and only if they rotated at exactly 7200 RPM and/or leveled their wears evenly across the silica.

And onto this unholy motley crue of hard disks I slapped a software architecture that promised to abstract all the typical storage driver, interface, and controller nonsense away, far, far away in fact, to a land where the storage can be mixed, the controllers diverse, and by virtue of the software-definition bits, network & hypervisor agnostic. In short, I wanted to build an agnostic #StorageGlory box in the Daisetta Lab.

Right. So what did I use to achieve this? ZFS and Zpools?

Hell no, that’s so January.

VSAN? Ha! I’m no Chris Wahl.

I used Windows, naturally.

That’s right. Windows. Server 2012 R2 to be specific, running Core + Infrastructure GUI with 8GB of RAM, and some 17TB of raw disk space available to it. And a little technique developed by the ace Microsoft server team called Tiered Storage Spaces.

Was a #StorageGlory Achievement Unlocked, or was it a dud?

Here’s my review after 30 days on my Windows SAN: san.daisettalabs.net.

The Good

It doesn’t make you pick a side in either storage or storage-networking: Do you like abstracted pools of storage, managed entirely by software? Put another way, do you hate your RAID controller and crush on your old-school NetApp filer, which seemingly could do everything but object storage?

When I say block, do you instinctively say file? Or vice-versa?

Well then my friend, have I got a storage system for your lab (and maybe production!) environment: Windows Storage Spaces (now with Tiering!) offers just about everything guys like you or me need in a storage system for lab & home media environments. I love it not just because it’s Microsoft, but also because it doesn’t make me choose between storage & storage-networking paradigms. It’s perhaps the ultimate agnostic storage technology, and I say that as someone who thinks about agnosticism and storage.

A lot.

You know what I’m talking about. Maybe today, you’ll need some block storage for this VM or that particular job. Maybe you’re in a *nix state of mind and want to fiddle with NFS. Or perhaps you’re feeling bold & courageous and decide to try out VMware again, building some datastores on both iSCSI LUNs and NFS shares. Then again, maybe you want to see what SMB 3.0 3.0 is all about, the MS fanboys sure seem to be talking it up.

The point is this: I don’t care what your storage fancy is, but for lab-work (which makes for excellence in work-work) you need a storage platform that’s flexible and supportive of as many technologies as possible and is, hopefully, software-defined.

And that storage system is -hard to believe I’ll grant you- Windows Server 2012 R2.

I love storage and I can’t think of one other storage system -save for maybe NetApp- that let’s me do crazy things like store .vmdks inside of .vhdxs (oh the vIrony!), use SMB 3 multichannel over the same NICs I’m using for iSCSI traffic, create snapshots & clones just like big filers all while giving me the performance-multiplier benefits of SSDs and caching and a reasonable level of resiliency.

File this one under WackWack\StorageGlory\Achieved\Windows boys and girls.

I can do it all with Storage Spaces in 2012 R2.

As I was thinking about how to write about Storage Spaces, I decided to make a chart, if only to help me keep it straight. It’s rough but maybe you’ll find it useful as you think about storage abstraction/virtualization tech:

Storage-Compared

And yes. Ex post facto dedupe is a made up term. By me. It’s latin for “After the fact, dedupe,” because I always scheduled my dedupes for Saturday night, when the IO load on the filer was low. Ex post facto dedupe is in contrast to some newer storage companies that offer inline compression & dedupe, but none of the ones above offer this, sadly.

It’s easy to build and supports your disks & controllers: This is a Microsoft product. Which means it’s easy to deploy & build for your average server guy. Mine’s running on a very skinny, re-re-purposed SanDisk Ready Cache SSD. With Windows 2012 R2 server running the Infrastructure Management GUI (no explorer.exe, just Server Manager + your favorite snap-ins), it’s using about 6GB of space on the boot drive.

And drivers for the Intel C226 SATA controller, the LSI 9218si SAS card, and the extra ASMedia 1061 controller were all installed automagically by Windows during the build.

The only other system that came close to being this easy to install -as a server product- was Oracle Solaris 11.2 Beta. It found, installed drivers for, and exposed all controllers & disks, so I was well on my way to going the ZFS route again, but figured I’d give Windows a chance this time around.

Nexenta 4, in contrast, never loaded past the Install Community Edition screen.

It’s improved a lot over 2012: Storage Spaces almost two years ago now, and I remember playing with it at work a bit. I found it to be a mind-f*** as it was a radically different approach to storage within the Windows server context.

I also found it to be slow, dreadfully slow even, and not very survivable. Though it did accept any disk disk I gave it, it didn’t exactly like it when I removed a USB drive during an extended write test. And it didn’t take the disk back at the conclusion of the test either.

Like everything else in Microsoft’s current generation, Storage Spaces in 2012 R2 is much better, more configurable, easier to monitor, and more tolerant of disk failures.

It also has something for the IOPS speedfreak inside all of us.

Storage Spaces, abstract this away

Storage Spaces, abstract this away

Tiered Storage Spaces & Adjustable write cache: Coming from ZFS & the Adaptive Replacement Cache, the ZFS Intent Log, the SLOG, and L2ARC, I was kind of hooked on the idea of using massive amounts of my ECC RAM to function as a sort of poor-mans NVRAM.

Windows can’t do that, but with Tiered Storage Spaces, you can at least drop a few SSDs in your array (in my case three x 256GB 840 EVO & one 128GB Samsung 830), mix them into your disk pool, and voila! Fast read-cache, with a Microsoft-flavored MRU/LFU algorithm of some type keeping your hottest data on the fastest disks and your old data on the cheep ‘n deep rotationals.

What’s more, going with Tiered Storage Spaces gives you a modest 1GB write cache, but as I found out, you can increase that up to 10GB.

Which i naturally did while building this guy out. I mean, who wouldn’t want more write-cache?

But there’s a huge gotcha buried in the Technet and blogposts I found about this. I wanted to pool all my disks together into as large of a single virtual disk as possible, then pack iSCSI-connected .vhdxs, SMB 3 shares, and more inside that single, durable & tiered virtual disk.What I didn’t want was several virtual disks (it helped me to think of virtual disks as a sort of Aggregate) with SMB 3 shares and vhdx files stored haphazardly between them.

Which is what you get when you adjust the write-cache size. Recall that I have a capacity of about 17TB raw among all my disks. Building a storage pool, then a virtual disk with a 10GB write cache gave me a tiered virtual disk with a maximum size of about 965GB.  More on that below.

It can be wicked fast, but so is RAID 0: Check out my standard SQLIO benchmark routine, which I run against all storage technologies that come my way. The 1.5 hour test is by no means comprehensive -and I’m not saying the IOPS counter is accurate at all (showing max values across all tests by the way)- but I like this test because it lets me kick the tires on my array, take her out for a spin, and see how she handles.

And with a “Simple” layout (no redundancy, probably equivalent to RAID 0), she handles pretty damn well, but even I’m not crazy enough to run tiered storage spaces in a simple layout config:

storage spaces

These three tests (1.5 hours each, identical setup against multiple configs) were done locally on the array, not over my home network

What’s odd is how poorly the array performed with 10GB of “Write Cache.” Not sure what happened here, but as you can see, latency spiked higher during the 10GB write cache write phase of the test than just about every other test segment.

Something to do with parity no doubt.

For my lab & home storage needs, I settled on a Mirror 2-way parity setup that gives me moderate performance with durability in mind, though not much as you’ll see below.

Making the most of my lab/home network and my NICs: Recall that I have six GbE NICs on this box. Two are built into the Supermicro board itself (Intel), and the other four come by way of a quad-port Intel I350-T4 server NIC.

Anytime you’re planning to do a Microsoft cluster in the 1GbE world, you need lots of NICs. It’s a bit of a crutch in some respects, especially in iSCSI. Typically you VLAN off each iSCSI NIC for your Hyper-V hosts and those NICs do one thing and one thing only: iSCSI, or Live Migration, or CSV etc. Feels wasteful.

But on my new storage box at home, I can use them for double-duty: iSCSI (or LM/CSV) as well as SMB 3. Yes!

Usually I turn off Client for Microsoft Networks (the SMB file sharing toggle in NIC properties) on each dedicated NIC (or vEthernet), but since I want my file cake & my block cake at the same time, I decided to turn SMB on on all iSCSI vEthernet adapters (from the physical & virtual hosts) and leave SMB on the iSCSI NICs on san.daisettalabs.net as well.

The end result? This:

Storage Networking-All of the Above Approach
nic Name VLAN IP Function
1 MGMT 100 192.168.100.15 MGMT & SMB3
2 CLNT 102 192.168.102.15 Home net & SMB3
3 iSCSI-10 10 172.16.10.x iSCSI & SMB3
4 iSCSI-11 10 172.16.11.x iSCSI & SMB3
5 iSCSI-12 10 172.16.12.x iSCSI & SMB3

That’s five, count ‘em five NICs (or discrete channels, more specifically) I can use to fully soak in the goodness that is SMB 3 multichannel, with the cost of only a slightly unsettling epistemological question about whether iSCSI NICs are truly iSCSI if they’re doing file storage protocols.

Now SMB 3 is so transparent (on by default) you almost forget that you can configure it, but there’s quite a few ways to adjust file share performance. Aidan Finn argues for constraining SMB 3 to certain NICs, while Jose Barreto details how multichannel works on standalone physical NICs, a pair in a team, and multiple teams of NICs.

I haven’t decided which model to follow (though on san.daisettalabs.net, I’m not going to change anything or use Converged switching…it’s just storage), but SMB 3 is really exciting and it’s great that with Storage Spaces, you can have high performance file & block storage. I’ve hit 420MB/sec on synchronous file copies from san to host and back again. Outstanding!

I Finally got iSNS to work and it’s…meh: One nice thing about san.daisettalabs.net is that that’s all you need to know…the FQDN is now the resident iSCSI Name Server, meaning it’s all I need to set on an MS iSCSI Initiator. It’s a nice feature to have, but probably wasn’t worth the 30 minutes I spent getting it to work (hint: run set-wmiinstance before you run iSNS cmdlets in powershell!) as iSNS isn’t so great when you have…

SMI-S, which is awesome for Virtual Machine Manager fans: SMI-S, you’re thinking, what the the hell is that? Well, it’s a standardized framework for communicating block storage information between your storage array and whatever interface you use to manage & deploy resources on your array. Developed by no less an august body than the Storage Networking Industry Association (SNIA), it’s one of those “standards” that seem like a good idea, but you can’t find it much in the wild as it were. I’ve used SMI-S against a NetApp Filer (in the Classic DoT days, not sure if it works against cDoT) but your Nimbles, your Pures, and other new players in the market get the same funny look on their face when you ask them if they support SMI-S.

“Is that a vCenter thing?” they ask.

Sigh.

Microsoft, to its credit, does. Right on Windows Server. It’s a simple feature you install and two or three powershell commands later, you can point Virtual Machine Manager at it and voila! Provision, delete, resize, and classify iSCSI LUNS on your Windows SAN, just like the big boys do (probably) in Azure, only here, we’re totally enjoying the use of our corpulent.vhdx drives, whereas in Azure, for some reason, they’re still stuck on .vhds like rookies. Haha!

Single Pane o' glass in VMM with SMI-S for the Hyper-V set

Single Pane o’ glass in VMM with SMI-S, GUIDs galore and more for the Hyper-V set

It’s a very stable storage platform for Microsoft Clustering: I’ve built a lot of Microsoft Hyper-V clusters. A lot. More than half a dozen in production, and probably three times that in dev or lab environments, so it’s like second nature to me. Stable storage & networking are not just important factors in Microsoft clusters, they are the only factors.

So how is it building out a Hyper-V cluster atop a Windows SAN? It’s the same, and different at the same time, but, unlike so many other cluster builds, I passed the validation test on the first attempt with green check marks everywhere. And weeks have gone by without a single error in the Failover Clustering snap-in; it’s great.

The Bad

It’s expensive and seemingly not as redundant as other storage tech: When you build your storage pool out of offlined disks, your first choice is going to involve (just like other storage abstraction platforms) disk redundancy. Microsoft makes it simple, but doesn’t really tell you the cost of that redundancy until later in the process.

Recall that I have 17TB of raw storage on san.daisettalabs.net, organized as follows:

Disk Type Quantity Size Format Speed Function
WD Red 2.5" with NASWARE 6 1TB 4KB AF SATA 3 5400RPM  Cheep 'n deep
Samsung 840 EVO SSD 3 256GB 512byte 250MB/read  Tiers not fears
Samsung 830 SSD 1 128GB 512byte 250MB/read  Tiers not fears
HGST 3.5" Momentus 6 2TB 512byte 105MB/r/w Cheep 'n deep

Now, according to my trusty IOPS Excel calculator, if I were to use traditional RAID 5 or RAID 6 on that set of spinners, I’d get about 16.5TB usable in the former, 15TB usable in the latter (assuming RAID penalty of 5 & 6, respectively)

For much of the last year, I’ve been using ZFS & RAIDZ2 on the set of six WD Red 2.5″ drives. Those have a raw capacity of 6TB. In RAIDZ2 (roughly analogous to RAID 6), I recall getting about 4.2TB usable.

All in all, traditional RAID & ZFS’ RAIDZ cost me between 12% and  35% of my capacity respectively.

So how much does Windows Storage Spaces resiliency model (Mirrored, 2-way parity) cost me? A lot. We’re in RAID-DP territory here people:

 

storagespaces5

Ack! With 17TB of raw storage, I get about 5.7TB usable, a cost of about 66%!

And for that, what kind of resieliency do I get?

I sure as hell can’t pull two disks simultaneously, as I did live during prod in my ZFS box. I can suffer the loss of only a single disk. And even then, other Windows bloggers point to some pain as the array tries to adjust.

Now, I’m not the brightest on RAID & parity and such, so perhaps there’s a more resilient, less costly way to use Storage Spaces with Tiering, but wow…this strikes me as a lot of wasted disk.

Not as easy to de-abstract the storage: When a disk array is under load, one of my favorite things to do is watch how the IO hits the physical elements in the array. Modern disk arrays make what your disks are doing abstract, almost invisible, but to truly understand how these things work, sometimes you just want the modern equivalent of lun stats.

In ZFS, I loved just letting gstat run, which showed me the load my IO was placing on the ARC, the L2ARC and finally, the disks. Awesome stuff:

In this Gifcam, watch ada0-6 as they struggle under load with the "Always Sync" option enabled.

In this Gifcam, watch ada0-6 as they struggle under load with the “Always Sync” option enabled.

As best as I can tell, there’s no live powershell equivalent to gstat for Storage Spaces. There are teases though; you can query your disks, get their SMART vitals, and more, but peeling away the onion layers and actually watching how Windows handles your IO would make Storage Spaces the total package.

Bottom line

So that’s about it: this is the best storage box I’ve built in the Daisetta Lab. No regrets going with Windows. The platform is mature, stable, offers very good performance, and decent resiliency, if at a high disk cost.

I’m so impressed I’ve checked my Windows SAN skepticism at the door and would run this in a production environment at a small/medium business (clustered, in the Scaled Out File Server role). Cost-wise, it’s a bargain. Check out this array: it’s the same exact Hardware a certain upstart Storage vendor I like (that rhymes with Gymbal Porridge) sells, but for a lot less!

#StorageGlory achieved. At home. In my garage.

Meet my new Storage Array

So the three of you who read this blog might be wondering why I haven’t been posting much lately.

Where’s Jeff, the cloud praxis guy & Hyper-V fanboy, who says IT pros should practice their cloud skills? you might have asked.

Well, I’ll tell you where I’ve been. One, I’ve been working my tail off at my new job where Cloud Praxis is Cloud Game Time, and two, the Child Partition, as adorable and fun as he is, is now 19 months old, and when he’s not gone down for a maintenance cycle in the crib, he’s running Parent Partition and Supervisor Module spouse ragged, consuming all CPU resources in the cluster. Wow that kid has some energy!

Yet despite that (or perhaps because of that), I found some time to re-think my storage strategy for the Daisetta Lab.

Recall that for months I’ve been running a ZFS array atop a simple NAS4Free instance, using the AMD-powered box as a multi-path iSCSI target for Cluster Shared Volumes. But continuing kernel-on-iscsi-target-service homicides, a desire to combine all my spare drives & resources into a new array, and a vacation-time cash-infusion following my exit from the last job lead me to build this for only about $600 all-in:

Software-defined. x86. File and block. Multipath. Intel. And some Supermicro. Storage utopia up in the Daisetta Lab

Software-defined. x86. File and block. Multipath. Intel. And some Supermicro. There’s some serious storage utopia up in the Daisetta Lab

Here are some superlatives and other interesting curios about this new box:

  • WP_20140705_01_19_31_ProIt was born on the 4th of July, just like ‘Merica and is as big, loud, ostentatious and overbearing as ‘Merica itself
  • I would name it ‘Merica.daisettalabs.net if the OS would accept it
  • It’s a real server. With a real Supermicro X10SAT server/workstation board. No more hacking Intel .inf files to get server-quality drivers
  • It has a real server SAS card, an LSI 9218i something or other with SAS-SATA breakout cables
  • It doesn’t make me choose between file or block storage, and is object-storage curious. It can even do NFS or SMB 3…at the same time.
  • It does ex post facto dedupe -the old model- rather than the new hot model of inline dedupe and/or compression, which makes me resent it, but only a little bit
  • It’s combining three storage chipsets -the LSI card, the Supermicro’s Intel C226, and ASMedia 1061- into one software-defined logical system. It’s abstracting all that hardware away using pools, similar to ZFS, but in a different, more sublime & elegant way.
  • It doesn’t have the ARC -ie RAM AS STORAGE- which makes me really resent it, but on the plus side, I’m only giving it 12GB of RAM and now have 16GB left for other uses.
  • It has 16 Disks : 12 rotational drives (6x1TB 5400 RPM & 6x2TB 7200RPM) and four SSDs (3x256GB Samsung 840 EVO & 1x128GB Samsung 830) and one boot drive (1x32GB SanDisk ReadyCache drive re-purposed as general SSD)
  • Total capacity RAW: nearly 19TB. Usable? I’ll let you know. Asking
    “Do I need that much?” is like asking “Does ‘Merica need to stretch from Sea to Shining Sea?” No I don’t, but yes ‘Merica does. But I had these drives in stock, as it were, so why not?
  • It uses so much energy & power that it has, in just a few days, erased any greenhouse gas savings I’ve made driving a hybrid for one year. Sorry Mother Earth, looks like I’m in your debt again
  • But seriously, under load, it’s hitting about 310 watts. At idle, 150w. Not bad all things considered. Haswell + full C states & PCIe power management work.
  • It’s built as veritable wind-tunnel as it lives the garage. In Southern California. And it’s summer. Under load, the CPU is hitting about 65C and the south-bridge flirts with 80c, but it’s stable.
  • It has six, yes, six, 1GbE Intel NICs. Two are on the motherboard, and I’m using a 4 port PCIe 2 card. And of course, I’ve enabled Jumbo Frames. I mean do you have to even ask at this point?
  • It uses virtual disks. Into which you can put other virtual disks. And even more virtual disks inside those virtual disks. It’s like Christopher Nolan designed this storage archetype while he wrote Inception…virtual disk within virtual disk within virtual disk. Sounds dangerous, but in the Daisetta Lab, Who Dares Wins!

So yeah. That’s what I’ve been up to. Geeking out a little bit like a gamer, but simultaneously taking the next step in my understanding, mastery & skilled manipulation of a critical next-gen storage technology I’ll be using at work soon.

Can you guess what that is?

Stay tuned. Full reveal & some benchmarks/thoughts tomorrow.