“Assume Breach” not just at work, but at home too

Security has been on my mind lately. I think that in the Spring of 2015, we’re in a new landscape regarding security, one that is much more sinister, serious and threatening than it was in years past. I used to think anonymity was enough, that there was saftey in the herd. But the rules & landscape have changed, and it’s different now than it was just 12 or 24 months ago. So, let’s do an exercise, let’s suppose for the sake of this post that the following are true:

  • Your credit history and your identity are objects in the marketplace that have value and thus are bought and sold between certain agents freely
  • These things are also true of your spouse or significant other’s credit history & identity, and even your child’s
  • Because these things are true, they are also true for malefactors (literally, bad actors) just like any other object that has value and can be traded
  • There is no legal structure in America aside from power of attorney that allows a single member of a family to protect the identity and credit history of another member of his/her family.
  • The same market forces that create innovation in enterprise technology are now increasing the potency of weaponized malware systems, that is to say that financial success attracts talent which begets better results which begets more financial success.
  • The engineers who build malware are probably better than you are at defending against them, and what’s more,they are largely beyond the reach of local, state, or national law enforcement agencies.1
  • The data breaches and mass identity theft of 2014 & 2015 are similar somehwat to a classic market failure, but no cure for this will be forthcoming from Washington, and the trial attorneys & courts who usually play a role in correcting market failures have determined your identity & credit history are worth about $0.14 (($10 million settlement for the 70 million victims of Target breach = $0.14))
  • Generally speaking most IT departments are bad and suffer from poor leadership, poorly-motivated staff, conflicting directions from the business, an inability to meet the business’ demands, or lack of C-level support. IT is Broken, in other words
  • All of this means it’s open season on you and your family’s identity & credit history, which we have to assume rest unencrypted on unpatched SQL servers behind an ASA with a list of unmitigated CVEs maintained by some guys in an IT department who hate their job
Don't be like these people. Secure your online identity now

Don’t be like these people. Secure your online identity now

There it is. That’s the state of personal identity & credit security in 2015 in America, in my view.

And worst of all, it’s not going to get better as every company in America with your data has done the math from the Target settlement and the beancounters have realized one thing: it’s cheaper to settle than to secure your information.

Assume breach at home

If this is truly the state of play -and I think it is- then you as an interested father/mother husband/wife need to take action. I suggest an approach in which you:

  1.  Own your Identity online by taking SMTP back: Your SMTP address is to the online world what your birth certificate and/or social security number is to the meatspace world: everything. Your SMTP address is the de facto unique identifier for you online2 , which begs the question: why are you still using some hippy-dippy free email account you signed up for in college, and why are you letting disinterested third party companies host & mine something for free that is so vital to your identity? Own your identity and your personal security by owning and manipulating SMTP like corporations do: buy a domain, find a hosting service you like, and pay them to host your email. It doesn’t cost much, and besides, you should pay for that which you value. And owning your email has value in abundance: with your own domain, you can make alias SMTP addresses for each of the following things: social media, financial, shopping, food, bills, bulk and direct your accounts to them as appropriate. This works especially well in a family context, where you can point various monthly recurring accounts at a single SMTP address that you can redistribute via other methods and burn/kill as needed.3
  2. Proxy your financial accounts wherever possible: Mask your finances behind a useful proxy, like Paypal, perhaps even Mint. The idea here is to put a buffer between your financial accounts and the services, people, and corporations that want access to them and probably don’t give two shits about protecting your identity or vetting their own IT systems properly. Whenever possible, I buy things online/pay people/services via Paypal or other tools so that use of my real accounts is minimized. Paypal even offers a business credit card backed by the Visa logo, which means you can use it in brick ‘n mortar stores like Target, where the infosec is as fast and loose as the sales and food quality.
  3. Filter the net at home and wherever else you can: Spyware, malware and viruses used to be an annoyance, the result of a global dick-measuring contest for geeks and nerds who liked to tinker and brag. But no more; today’s malware systems are weaponized and potent, and that puts you and your family at a huge disadvantage as it’s difficult to secure all the devices creeping into your life, let alone worry about the bad IT departments stewarding your sosh, DOB, mother’s maiden name and home address at RetailCo. I suggest a heavy filtering strategy by whatever means you can employ: employ whitelist javascript filtering on Windows PCs, use and pay for OpenDNS malware filtering, or buy something like ITUS Networks or even a ZyXel like the one I have. Get to know Privoxy well as I think filtering ads from websites is even fair now as the major ad agencies apparently can’t prevent malware from creeping into them. Finally invest some time and study into certificates and periodically review their use, as there are Certificate Authorities out there that you should not trust.
  4. Use Burner Numbers: Similar to SMTP, your standard US 10 digit POTS/Mobile phone is a kind of unique identifier to companies, existing somewhere in a unsecured table no doubt. Use burners where you can as your 10 digit mobile is important as  a unique identifier and an off-net secondary notification/authentication channel.  If Google Voice is to be killed off, as it appears to be, consider Ooma, where for $100/year, you can spawn burner numbers and use them in the same way you use SMTP. Else, use the app on your phone for quick burner numbers.
  5. Consider Power of Attorney or Incorporation: This is admittedly a little crazy, but words can’t describe how furious you’ll be when a family member’s identity has been stolen and some scummy organization that calls itself a bank is calling to verify that you’ve purchased $1000 in Old Navy gift certificates in Texas -something completely out-of-sync with your credit history- but they refuse to stop the theft because it’s happening to your wife, not you, and your wife can’t come to the phone right now.  The solution to this problem is beyond me, but probably involves a “You can’t beat ‘em, join ‘em” approach coupled with an attorney’s threatening letter.
  6. Learn to Love Sandboxing: Microsoft has a free and incredibly powerful tool called Enhanced Mitigation Experience Tool, or EMET, which allows you to select applications and essentially sandbox them so that they can’t pwn your entire operating system. Learn to use and love it. But the idea here goes beyond Win32 to the heart of what we should be doing as IT Pros: standing-up and tearing-down instances of environments, whether those environments are Docker containers, Windows VMs, jails in BSD, or KVM virtual machines. Such techniques are useful beyond devops, they are also useful as operational security techniques at home in my view.
  7. Go with local rather than national financial institutions: Where possible, consider joining a local credit union, where infosec practices might not be state of the art, but your family’s finances have more influence and weight than they do at a Bank of America.

I am not a security expert, but that’s how I see it. If we IT pros are to assume breach at work, as many experts advise us to, we should assume breach at home too, where our identities and those of our loved ones are even more vulnerable and even more valuable.


  1. Supposing that your local Sheriff’s Department even has the in-house know-how to handle security breaches, they lack jurisdiction in Ukraine 

  2. By virtue of the fact that these two things are true of SMTP but are not true of rival identity systems, like Facebook or Google profiles: 1) Your SMTP address is required to transact business or utilize services online or is required at some point in a chain of identity systems and 2) SMTP is accepted by all systems and services as prima facie evidence of your identity because of its uniqueness & global acceptance and rival systems are not 

  3. Pretty soon, you and your loved ones will get the hang of it, and you and your family will be handing out food@domain.com to the grocery store checkout person, retail@domain.com for receipts, shopping@domain.com for the ‘etailers’ and apple@domain.com for the two iPhones & three other Apple devices you own. 

How to Superfish Your Users : SSL Proxy in a Windows Network

When in the course of IT events it becomes necessary to inspect all traffic that hits your user’s PCs, there is but one thing you can do in 2015: get a proxy server or service, deploy a certificate to your trusted root store, and direct all traffic through the proxy.

Why would you do what amounts to a Man in the Middle Attack on your users as a responsible & honest IT Pro? Why Superfish your users?1

Numbers are hard to pin down,2 but it seems an increasing amount of virulent & potent malware is arriving at your edge encrypted. Because those packets are encrypted, you essentially can’t scan the contents. All you get is source/destination IP address, some other IP header information, and that’s about it.

No bueno.

One option, really your only option at that point, is to crack open those packets and inspect them. Here’s how.

1.You need a proxy server or service that does security inspection. 

I’ve seen ZScaler used at several firms. ZScaler dubs itself the premiere cloud-based, SaaS proxy service, and it’s quite a nifty service.

For a fee per user, ZScaler will proxy most if not all of your internet traffic from several datacenters around the globe, sort of like how CloudFlare protects your websites.

The service scans all that http and https traffic, filters out the bad and malicious stuff, blocks access to sites you tell it to, and sends inspected http/https content to your users, wherever they are, on-prem or connected to the unsecured Starbucks access point.

2. You need to bundle those proxy settings up into a .pac file

Getting the service is one thing, you still need to direct your users and computers through it. The easiest way is via Group Policy & what’s called a .pac file.

A .pac file is a settings file generated by ZScaler that contains your preferences, settings, and lists of sites you’d prefer bypass the filter. It looks like this:


function FindProxyForURL(url, host)
{
    var resolved_host_ip = dnsResolve(host);

    if (!isResolvable("gateway.zscaler.net"))
        return "DIRECT";

    if (url.substring(0, 4) == "ftp:")
        return "DIRECT";

    // If the requested website is hosted within the internal network, send direct
    if (isPlainHostName(host) ||
        isInNet(resolved_host_ip, "1.1.1.1", "255.0.0.0") ||
        return "DIRECT";

    // If the requested website is SSL and associated with Microsoft O365, send direct
    return "DIRECT";

3. Deploy the .pac file via Group Policy to Users

Next, you need to pick your favorite deployment tool to push the .pac file out and set Windows via IE to proxy through ZScaler. We’ll use Group Policy because it’s fast and easy.

Under User Configuration > Policies > Windows Settings > Internet Explorer Maintenance > Connection / Automatic Browser Configuration, select Enable.

Then point the Auto-proxy URL to your Zscaler .pac file URL. It looks like this:

grouppolicy

Keep Group Policy open, because we’re not done quite yet.

4. Download the ZScaler Root CA certificates

You’ll find the certs in the administration control screen of ZScaler. There are two:

  • ZScaler Root Certificate -2048.crt
  • ZScalerRoot Certificate -2048-SHA256.crt

The two certificates are scoped similarly, the only difference seems to be SHA1 or SHA256 encoding.

Double-click the certificate you prefer to use, and notice that Windows alerts you to the fact that it’s not trusted. Good on ya Microsoft, you’re right.

To validate this setup, you’ll probably want to test before you deploy. So select Install Certificate, select your Computer (not user) and navigate to the Trusted Root CA Store:

rootca

or you can do it via powershell:


PS C:\daisettalabs.net\Import-Certificate -FilePath C:\users\jeff\Downloads\ZscalerRootCerts\ZscalerRootCertificate-2048-SHA256.crt -CertStoreLocation Cert:\LocalMachine\Root
Directory: Microsoft.PowerShell.Security\Certificate::LocalMachine\Root
Thumbprint Subject
---------- -------
thumbprint E=support@company.com, CN=Zed, OU=Zed Inc, O=Zed's Head, L=The CPT, S=CaliforniaLove, C=USA 

4. Verify that the .pac file is in use

Now that you’ve installed the .pac file and the certificate, ensure that IE (and thus Chrome, but not necessarily Firefox) have been set to proxy through Zscaler:

Your settings will differ no doubt from my screenshot

5. SSL Proxy Achievement Unlocked: 

Go to Google or any SSL/TLS encrypted site and check the Certificate in your browser.

You should see something like this:

googlewithz

 

6. You can now deploy that same certificate via Group Policy to your Computers.

It’s trivial at this point to deploy the ZScaler certificates to end-user PCs via Group Policy. You’ll want to use Computer Preferences.

Once deployed, you’ll get comprehensive scanning, blocking and reporting on your users http/https use. You can of course exempt certain sites from being scanned3


  1. IT Shakespeare put it like this:

    To proxy SSL or not to proxy, that is the question

    whether ’tis nobler in the mind to suffer

    the breaches and theft of outrageous malware

    or to take Arms against a sea of digital foes

    and by opposing, only mitigate the threat.

    To protect via decrypt ; Aye there’s the rub

    Thus Conscience does make Cowards of us all

    and lose the name of Action 

  2. I am not a security expert, and though I checked sources I respect like the Norse IP Viking security blog, Malwarebytes Unpacked blog, SearchSecurity.com etc, I found very few sources that a percentage on how much malware is encrypted and thus difficult to detect. This NSS Labs report from summer 2013 comparing Next Gen Firewall SSL Decryption performance, for instance, says that “the average proportion of SSL traffic within a typical enterprise is 25-35%”  and that only ~1% of malware is encrypted. A GWU security researcher named Andre DiMino has a series of good blog posts on the topic, showing what SSL-encrypted malware looks like in WireShark. Team CYMURU’s TotalHash database is probably the most comprehensive open DB of malware samples, but I don’t feel qualified to search it frankly 

  3. Before you do this, make sure you get your Legal department or corporate controller’s sign-off on this. Your company needs to understand exactly what SSL Proxy means, and the Gordian Knot of encryption.

    By making all SSL traffic visible to your proxy service,  you may gain some ability to prevent potent malware attacks, but at the cost of your user’s privacy. When a user transacts business with their bank, their session will be secured, but only between the ZScaler cloud and the bank’s webserver. The same is true of Facebook or personal email sites.

    By doing this, you’re placing an immense amount of trust in the proxy server/service of your choice. You’re trusting that they know what they’re doing with Certificates, that they didn’t use a weak password. You’re trusting that they have their act together, and you’re doing this on behalf of all your users who trust you. This is not to be taken lightly, so run it up the legal/HR flagpole before you do this.  

Convergence 2015 – Microsoft

Microsoft’s business technology conference, Convergence 2015, is live-streaming today from Atlanta Georgia.

It’s a special event for me because I’m consulting for the IT team at WASH Multifamily Laundry, which will be featured as a “Visionary” business during today’s presentation.

“Big data,” “actionable intelligence,” “the cloud,” and such might be buzzwords to some, or on the roadmap for others, but here at WASH, these things are the real deal and they are empowering the business. This company services several hundred thousand coin-operated washing machines & driers throughout North America, and what they’ve done to instrument, analyze and harness the way their machines are utilized is amazing.

In the lobby of the office where I work, we show it off: PowerBI dashboards like this, showing a heatmap of our washer/dryers by various metrics and categories. We can tell you with certainty the volume of service calls per manufacturer of washing machine.  We can see gross profit per  machine across the entire fleet:

IMG_20150316_081725002

and even crosstab Census, demographic and economic data to figure out which markets to expand to next.

Another cool thing we’re doing with Microsoft: taking a firehose of instrumentation data from our several hundred thousand machines and pointing it at Azure Machine Learning. What’s it going to tell us about the way our machines operate, how they’re used? I’m not sure, but I work in the building where some bright devs are much closer to answering those questions today than they were a year or six months ago.

As for me, I’m just an infrastructure guy who likes IT, has had a lot of success with the Microsoft stack and is happy to play my small part in WASH’s success.

To the clouds!

Microsoft’s commitment to open initiatives & the riddle of whitebox networking

On Tuesday Microsoft surprised me by announcing an open switching/networking plan in partnership with Mellanox and as part of the Open Compute initiative.

Wait, what?

Microsoft’s building a switch?

Not quite, but before we get into that, some background on Microsoft’s participation in what I call OpenMania: the cloud & enterprise technology vendor tendency to prefix any standards-ish cooperative work effort with the word Open.

Microsoft’s participating in several OpenMania efforts, but I only really care about these two because they highlight something neat about Microsoft and apply or will soon apply to me, the Converged IT Guy.

Open Compute, or OCP, is the Facebook-led initiative to build agnostic hardware platforms on x86 for the datacenter. I like to think of OCP as a ground-up re-imagining of hardware systems by guys who do software systems.

As part of their participation in OCP, Microsoft is devoting engineering resources and talent into building out specifications, blueprints and full hardware designs for things like this, a 12U converged chassis comprised of storage and compute resources.

ocs

Are those brown Zunes in the blades?

 

Then there’s Open Management Infrastructure (OMI), an initiative of the The Open Group (TOG). Microsoft joined OMI almost three years ago to align & position Windows to share common management frameworks across disparate hardware & software systems.

That’s a lot of words with little meaning, so let me break it down for the Windows guys and gals reading this. The promise of Microsoft’s OMI participation is this: you can configure other people’s hardware and software via the same frameworks your Windows Server runs on (CIM, the next-gen WMI) using the same techniques and tooling you manage other things with: Powershell.

All your management constructs are belong to CIM

All your management constructs are belong to CIM

I’ve been keenly interested in Microsoft & their OMI push because it’s an awesome vision, and it’s real, or real-close at any rate: SMI-S, for instance, is gaining traction as a management play on other people’s hardware/software storage systems1 , and is already baked-into Windows server as a feature you can install and use to manage Windows Storage Spaces, which itself is a first-class citizen of CIMville.

All your CIM classes -running as part of Windows or not- manipulated & managed via Powershell, the same ISE you and I use to deploy Hyper-V hosts, spin-up VMs, manage our tenants in Office 365, fiddle around in Azure, and make each day at work a little better and a little more automated than the last.

That’s the promised land right there, ladies and gentlemen.

Except for networking, the last stubborn holdout in my fevered powershell dream.

Jeff Snover, the architect of the vision, teases me with Powershell Leaf Spine Tweets like this:

but  I have yet to replace Putty with Powershell, I still have to do show int status rather than show-interface -status “connected” on my switch because I don’t have an Arista or N7K, and few other switches vendors seem to be getting the OMI religion.

All of which makes Microsoft’s Tuesday announcement that it is extending its commitment to OCP’s whitebox switching development really odd yet worthy of more consideration:

The Switch Abstraction Interface (SAI) team at Microsoft is excited to announce that we will showcase our first implementations of the specification at the Open Compute Project Summit, which kicks off today at the San Jose Convention Center. SAI is a specification by the OCP that provides a consistent programming interface for common networking functions implemented by network switch ASIC’s. In addition, SAI allows network switch vendors to continue to build innovative features through extensions.

The SAI v0.92 introduces numerous proposals including:

Access Control Lists (ACL)
Equal Cost Multi Path (ECMP)
Forwarding Data Base (FDB, MAC address table)
Host Interface
Neighbor database, Next hop and next hop groups
Port management
Quality of Service (QoS)
Route, router, and router interfaces

At first glance, I wouldn’t blame you if you thought that this thing, this SAI, means OMI is dead in networking, that managing route/switch via Powershell is gone.

But looking deeper, this development speaks to Microsoft’s unique position in the market (all markets, really!)

  1. SAI is probably more about low-level interaction with Broadcom’s Trident II2 and Microsoft’s participation in this is more about Azure and less about managing networking stuff w/Powershell
  2. But this is also perhaps Microsoft acknowledging that Linux-powered whitebox switching is really enjoying some momentum, and Microsoft needs to have something in this space

So, let’s review: Microsoft has embraced Open Compute & Open Management. It breaks down like this:

  • Microsoft + OCP =  Contributions of hardware blueprints but also low-level software code for things like ASIC interaction
  • Microsoft + OMI = A long-term strategic push to manage x86 hardware & software systems that may run Windows, but likely run something Linuxy yet

In a perfect world, OCP and OMI would just join forces and be followed by all the web-scale players, the enterprise technology vendors, the storage guys & packet pushers. All would gather together under a banner singing kumbaya and praising agnostic open hardware managed via a common, well-defined framework named CIM that you can plug into any front-end GUI or CLI construct you like.

Alas, it’s not a perfect world and OCP & OMI are different things. In the real world, you still need a proprietary API to manage a storage system, or a costly license to utilize another switchport. And worst of all, in this world, Powershell is not my interface to everything, it is not yet the answer to all IT questions.

Yet Microsoft, by virtue of its position in so many different markets, is very close now to creating its own perfect world. If they find some traction with SAI, I’m certain it won’t be long before you can manage an open Microsoft-designed switch that’s a first-class OMI citizen and gets along famously with Powershell!3">Cloud Platform System program))


  1. cf NIMBLE STORAGE NOW INTEGRATES WITH SCVMM 

  2. At least that’s my read on the Github repo material 

  3. Or buy one, as you can buy the Azure-in-a-box which is simply the OCP blueprint via Dell/Microsoft

The Value of Community Editions

I was excited to hear on the In Tech We Trust podcast this week that the godfather of all the hyperconverged things -Nutanix- may release a community edition of their infrastructure software this year.

That. Would. Be. Amazing.

I’ve crossed paths with Nutanix a few times in my career, but they’ve always remained just a bit out of reach in my various infrastructure projects. Getting some hands-on experience with the Google-inspired infrastructure system in my lab at home would be most excellent, not just for me, but for them, as I like to recommend product stacks I’ve touched above ones I haven’t.

Take Nexenta as an example. As Hans D. pointed out on the show, aside from downloading & running Oracle Solaris 12, Nexenta’s just about the only way one can experience a mature & enterprise-focused implementation of ZFS. I had a blast testing Nexenta out in my lab in 2014 and though I can’t say my posts on ZFS helped them move copies of NexentaStore, it surely didn’t hurt in my view.

VEEAM is also big in the community space, and though I’ve not tested their various products, I have used their awesome stencil collection.

Lest you think storage & hyperconvergence vendors are the only ones thinking ‘community, today my favorite yellow load balancer Kemp announced in effect a community edition of their L4/L7 Loadmaster vAppliance. Kemp holds a special place in the hearts of Hyper-V guys; as long as I can remember, yes even back in the dark days of 2008 R2, they’ve always released a Loadmaster that’s just about on-par with what they offer to VMware shops. In 2015 that support is paying off I think; Kemp’s best-in-class for Microsoft shops running Hyper-V or building out Azure, and with the announcement you can now stress a Kemp at home in your lab or in Azure with your MSDN sub. Excellent.

Speaking of Microsoft, I’d be remiss if I didn’t mention Visual Studio 2013, which got a community edition last fall.

I’d love to see more community editions, namely:

  • Nimble Storage: I’ve had a lot of success in the last 18 months racking/stacking Nimble arrays in environments with older, riskier storage. I must not be the only one;  the company recently celebrated its 5,000th customer. Yet, Nimble’s rapid evolution from storage startup with potential to serious storage player is somewhat bittersweet for me as I no longer work at the places I’ve installed Nimble arrays and can’t tinker with their rapidly-evolving features & support. Come on guys, just give me the CASL caching system in download form and let me evaluate your Fiber Channel support and test out your support for System Center
  • NetApp: A community release of Clustered Data OnTAP 8.2x would accomplish something few NetApp products have accomplished in the last few years: create some genuine excitement about the big blocky blue N. I’m certain they’ve got a software-only release in-house as they’ve already got an appliance for vSphere and I heard rumors about this from channel sources for years. So what are you waiting for NetApp? Let us build-out, support, and get excited about cDOT community-style since it’s been too hard to see past the 7-mode–>clustered mode transition pain in production.

On his Graybeards on Storage podcast, Howard Marks once reminisced about his time testing real enterprise technology products in a magazine’s tech lab. His observations became a column, printed on paper in an old-school pulp magazine which was shipped to readers. This was beneficial relationship for all.

Those days may be gone but thanks to scalable software infrastructure systems, the agnostic properties of x86, bloggers & community edition software, perhaps they’re back!

© 2015

Theme by Anders NorenUp ↑