San Francisco – Queen of the Pacific Basin

The ultimate proof of Brechin’s thesis can be had in today’s Washington Post which describes the phenomenal, dazzling wealth in the City by the Bay, named after the humble Francis of Assisi. We see via this piece that the Great Queen of the Pacific Basin, the Colossus, home of “tech”, presses on to the profitable colonization of the globe and enslaves humanity to the production of valuable bits, for which no recompense is returned to the producers and deception reigns supreme.

The Bay Area is home to more billionaires per capita than anywhere on Earth, one out of every 11,600 residents, according to Vox. The entire region, as far as two hours away, has been affected by spiraling real estate prices. Venture capitalist John Doerr has claimed that the area’s economic growth is “the greatest legal accumulation of wealth in history.”

Read more

But today’s Bay Area is much much more powerful than it was in the period Brechin describes.

Figure 4. “Man’s Great Storehouse of Wealth.” A graphic in the Hearst newspapers celebrates mining as a violent assault upon the “Beautiful Planet Devoted to His Use.” San Francisco Examiner, February 8, 1907. -Brechin, Imperial San Francisco, 2006, UC Berkeley Press

Continue reading “San Francisco – Queen of the Pacific Basin”

California Technology Exports

Check out this sentence. I’ll reveal who wrote it later:

…the American West had been the most fertile field for technical innovation…California engineers exported their technology to the rest of the world and improved on that which they imported from everywhere else.

Interesting sentence, right? The author is making the point that California, particularly the Bay Area in this case, is a hub of technical innovation and engineering prowess.

And indeed it is. I mean just look all around us. Silicon Valley companies dominate the world. Three of the top five technology companies (Google, Facebook, Apple) are headquartered there, and the other two, Microsoft & Amazon, have significant presence in Silicon Valley.

Consider those five companies and what they’ve done. Just as the author alleges, those five companies have found a formula for success; they’ve “imported from everywhere else” elemental technology primitives, things like standardized and open protocols built by academics and expert committees in the IETF, IEEE and other standards bodies. These companies have taken those elemental primitives and packaged them up into new exciting innovations and won dominance in the marketplace with them. How much dominance?

Look at this chart I made in Excel. $3.5+ trillion of market dominance, that’s how much dominance. And notice how few they actually employ compared to other titans of the marketplace. They’re massively efficient. That’s the whole point. That’s why capital is so excited about the Big 5.

marketcap
Numbers are out of date reflecting 2017 LTM Revenue & employment numbers but you get the idea

All around the world, people have tried but largely failed to replicate the supposed success of this vibrant hive of technical & engineering prowess. I hear it all the time on podcasts, I read it on Twitter, I read it in blogs. Everyone wants to be Silicon Valley, to be the exciting hub of innovation. Indeed, they want to be the next Silicon Valley, as if this is a repeatable formula there for the taking, as if you could just divine it out of the ether and bam, the next Silicon Valley. 

You see the big 5 marketed endlessly by the apostles of the Disruption Gospel, by the trade press, by us, even when we just think we’re talking about a new device or service. Oh yeah, I love this new feature on my Android. Oh Instagram is introducing end-to-end encryption & direct messaging. People love the products they’re using from these big five companies, and some study them so much they’ve launched ancillary careers just by studying how they work.  I’ve mentioned it before how I admire Ben Thompson, of stratechery.com for the one-man punditry business he’s built atop what he calls Aggregation Theory.

And the founders! We construct mythologies about them too. We build them up into icons. They collectively have more money than God or the tycoons of old.

Now circle your mind back to the quoted sentence. That’s it. Now let’s zoom out:

By 1893, the renowned Canadian mining operator James Douglas could claim that the American West had been the most fertile field for technical innovation in the development of hardware, techniques, and chemistry. California engineers exported their technology to the rest of the world and improved on that which they imported from everywhere else.

The quoted passage is from Dr Gray Brechin’s masterpiece polemic, Imperial San Francisco:Urban Power, Earthly Ruinpublished by University of California Press in 1999, revised in 2006.

Brechin, is, in the words of people I follow on Twitter, my spirit animal. He’s a Geographic Historian who lectures at Berkeley and other universities in the Mountain West. His book -which invokes huge themes about mining, agriculture, cities vs rural areas, and what he terms the Anglo-Aryan race- is all about the conquest of the frontier, and how that conquest was directed by a cartel of mining interests in San Francisco just after the start of the Gold Rush. If you’re interested in Manifest Destiny, you can’t miss this book.

Throughout his polemic, Brechin details the ruthlessness of the early titans of gold & silver mining in and around San Francisco. How they pushed out or simply killed natives. How President Polk, on discovery of gold in California, sparked a war with Mexico and ultimately won control of the west for America. How the early miners scooped up and collected the easy gold first, then pitched a false vision of California to the rest of America and got suckers to move out west for cheap & easy gold. How the miners & miner interests leveled entire forests in the Sierra Nevada, changed the course of rivers, dynamited and blasted their way deep into the scarred earth. And how, once the great con was over, they set their eyes westward again, to spreading the Anglo-Aryan race across the Pacific Basin from the mouth of the Golden Gate.

It’s really a yarn, quite the page turner I tell you. Definitely a great purchase, especially if you’re interested in place and history. Brechin even links the mining & mineral themes almost up to the present day, with the founding of Lawrence Livermore Labs in the east Bay, and its work on developing nuclear weapons.

We see all the time in technology commentary people invoking the same themes Brechin masterfully describes. They talk of atoms versus bits, as in the mining of precious metal atoms vs the mining of non-physical bits, or elements of technology. We ourselves call the titans of bit-mining today founders, and we all listen to the founders as they pitch a vision that, like the mining cartels and newspaper barons before them, results in more wealth accruing to them, and, like the rubes we are, only marginal value for the rest of us*.

It is hardly surprising that the bronze men at the prow of the Pioneer Monument were gold panners working the Sierra placers. California artists almost always depicted the Western miners as free men working under friendly Western skies—not underground,not for others, and not in squalor of their own creation. Such hardy individuals quickly came to symbolize Western opportunity itself, for they were the first to tap untouched bonanzas amid then-unspoiled scenery, and they remain the most enduring agents in the legend of entrepreneurial independence and of he-men living close to nature’s ample bosom.

ibid, Chapter 1, A Promised Land Plundered

And just as the gold miners of the 19th century externalized costs onto society, the environment, indigenous peoples, the Chinese,so too do the mining titans of the 21st Century externalize their costs onto our society. 

These founders, and the people working to sell the vision have, like the mining cartels before them, become digital prophets and invoke almost with religious intensity the themes of the frontier, the very words & phrases of Manifest Destiny.Simon Wardley, for instance, has built another business atop bits and bit mining. He calls them Wardley Maps, and they offer strategic advice and interesting mapping techniques to software engineers & technology companies. Wardley consistently uses the words pioneers, settlers, town planners and ‘uncharted’ as if there’s still more frontier left to exploit.

Untitled pictureThe founders in charge of today’s mining cartels have been using these words and phrases for more than a decade. I just don’t think we realized they actually meant what they were saying.  I think we all got confused by the razzle dazzle of what we saw on our screens, and so we listened to and trusted the razzle dazzle prophets and founders. In short order, we’ve all adopted the language of this new frontier. We’ve all taken Manifest Destiny a step further, even if we’d object to the old Manifest Destiny in principal if not in our history. Because we don’t see the metaphors the founders use for what they truly are: actual frontier-speak. 

The founders’ conquests are occurring in and around San Francisco, where the last frontier closed a little over a century ago. It’s a place that, on the surface, looks much different than the one Brechin details in his polemic. Yes, there is chronic homelessness and skyrocketing rents on the surface, but no one could claim San Francisco or the Bay Area is uncivilized, that it is not a world class city, that most people feel safe there.

But San Francisco -and the Bay Area- always looked beautiful. It’s a beautiful and lovely place. As beautiful as it was in 1898 to be sure, probably more so. But that’s just the surface. You’ve got to dig deeper, you’ve got to peer across whatever industry vertical you work in in 2019 to see the real costs. To see the con and misdirection. Until you do that, you’ll miss the externalized costs and exploitation of the 21st century mining cartels. You need to look at the razzle dazzle on your screen and realize the words you’re seeing are deceptive, that the metaphors have been used to misdirect you, to create a ‘smoky hall of mirrors’ effect, as I called it in an earlier essay. And then you’ve got to read the news and study it and think about it: Rohingya violence, violence in India, the amplification of bad information, anti-vaxxer ads, measles cases soaring, the flat earth, white supremacy on the march, and so much more. All of it organized, spread, and amplified at lightning speed with tooling created by the founders, their cartels, and the engineering prowess of the Bay Area.

tahoeAs Brechin would point out, the costs of the first mining cartels were hidden from the eyes of the wealthy urbanites in San Francisco as they extracted value out of people and the land far away.  They never saw the destruction of old growth Sierra Nevada forests because they didn’t want to see it. They never saw the Chinese Coolies -practically slave labor- herded into railcars and dispatched post-haste once the mining was done and the railroads were built. They never saw the mud and floods as millions of metric tons of toxins and earth flowed down the Central Valley and into the Bay itself. They never saw any of the costs because those costs were intentionally remote.

But in our age, we do see the costs. The exploitation. We see the costs all the time and everyday on our screens, if we just flip the script and study a little bit. You see the costs and you even think about the costs in the privacy of your own home, with yesterday’s Momo freakout. You see the costs but you don’t conceive of them as costs on you or your loved ones. You think of them as social media problems or platform abuse. 

Zoom out a bit, and the vista becomes clear. You see that the founders imported the elemental primitives of 20th Century standards bodies -things like TCP/IP, SMTP, and DNS, the WWW, and packet-switched networking- and got busy constructing and exporting Manifest Destiny 2.0 with those elements. And they’ve been telling us what they’ve been doing the whole time, we just didn’t realize it.

*I have noted in a previous essay how wonderful these technologies have been for women, People of Color and LGBTQ folks. I celebrate their agenda and the fact that they are seizing real political power long denied to them in the old, physical world. The value & benefit to them is immense, and I acknowledge that, and I want to ally with them in my politics. But this essay explores the costs side of the equation.

On the Advocacy & Entitlement of tech workers

Google backs out of Pentagon cloud contract after workers protest20,000 Googlers walk-out to protest sexual harassment and workplace discrimination….Microsoft workers protest use of Halolens by US ArmyGooglers protests AI board..Google closes AI board..Microsoft workers stand-up for Chinese tech workers….Googlers claim retaliation for walkout….

On and on over the last 18 months we’ve seen headlines and stories like this, stories about political advocacy at what are technically -and legally- private workplaces, but which, in reality, function differently. We see these stories on our screens, and we read about the workers and their workplaces, but what are all these stories really about? Are we seeing the birth of a proto-labor movement, or is this something else? Why did I feel support & solidarity with Googlers walking out following sexual harassment at their workplace, while secretly resenting their ability to organize & protest?

Their Workplace and Our Workplace

These are workplaces where all the familiar trappings of American workplaces are present. Por ejemplo, if a Google workplace is in California, I’m 100% certain there’s a “Your rights & responsibilities” placard in the facility that’s meant to inform workers of their rights. There’s likely OSHA placards too. Information about worker’s comp. Minimum wage notices. Exit signage. Fire & building regulations. There’s probably compliance hotlines for employees to dime on bad or unethical behavior they see at their employer. All the legal trappings and rights that labor won for us politically in the 20th century, all those things are at Google, at Microsoft, at Amazon, just like they are at your workplace & my workplace, no matter how big or small it is.

Untitled
A Googler is entitled to complain publicly about workplace politics and decisions and to even hire counsel to assist in their complaint

And yet, step back, and these tech workers enjoy much more liberty -indeed, are entitled to more rights- at their workplaces than we are at our’s. So much liberty, in fact, they can be themselves on our screens. During work hours even. Rarely do they have to say, “Thoughts & Opinions expressed here are mine nad mine alone and do not represent those of my employer, ” like I have to say. They can even show off the gifts they’ve received on social media -no need to disclose how or why they are receiving compensation from other companies- they can just show it off.

They are evidently entitled to so many rights within their workplaces, they can even criticize their superiors in public, by name. Wild!

Meanwhile, the rest of us don’t have that power. We’d be fired *instantly* if we did a 10th of what the tech workers do.

The Tech Workers’ Entitlement to Rights We Don’t Have

This then, is their entitlement, their privilege. They are entitled to lobby and organize and effect political change at their workplace -which intersects with our workplaces & homes by virtue of the internet, our screens and their products-  during work hours, while we cannot do the same at our’s. What explains that and what are their goals?

Well, their entitlement to rights we don’t have springs from their ability to code a world -a frontier- that achieves returns in an effecient manner for the shareholders. They are, in effect, hired guns building out a frontier that ultimately will deliver a reliable annuity to their sponsors, the bank*. Let’s take a look at this chart I made in Excel to understand how this works.

marketcap

Particularly pay attention to the revenue per employee number & market cap/employee. Those numbers are the Rosetta Stone for divining the political power of these tech workers, and indeed, the economic model that SiliconValley itself uses in its conquest of the world, and of the 20th Century’s verticals.

Notice that, for instance, every Google employee generates almost $1.4 million worth of revenue for the company. A Costco employee, on the other hand, generates only about half that much. Notice too that the Googler’s share of market cap (this chart was done on 2017 numbers but generally is accurate) is almost $9 million, 18x that of the Costco employee’s share of that corporation’s market cap.

Apple’s revenue/employee figure is off the charts too. Nearly $2m of revenue is generated by each Apple employee**. Microsoft & Amazon’s are more modest -particularly Amazon’s due to the larger number of people they employ- but even they are able to protest and organize politically at their workplaces, though their employees seem to make fewer waves as compared to Google’s.

Facebook employees, in contrast, are the great outlier. What little we hear from inside Facebook comes from ex-employees. The firm, like Apple, seems to have a stronger management culture than Google, but a less politically aware one. But notice that Facebook -whose revenue per employee numbers are very high- outsources the dirty work of managing its frontier to third party firms. We recently heard from one company’s employees about the work they do to moderate content on the Facebook frontier. It was not a happy story.

Now look at Walmart: Walmart employs almost 3 million people yet its revenue per employee is the lowest of the bunch, as is its market cap/employee figure. How often do we hear from Walmart employees?

The Political Objectives of some Big Tech workers

Every political movement needs to unite disparate and sovereign peoples under a banner of change and a well-understood set of objectives. Typically, we calll this an “agenda” or even a manifesto. As best I can tell, the tech workers’ agenda is this:

  • To have a say in how AI Systems are built and work, such that these products don’t discriminate or target marginalized groups inside the company and outside the company
  • To be free to not work on products that will be used by the US military, public sector governments, or even cities and municipalities
  • To blunt or evade the power and sovereignty of foreign nations like China by delivering products that guarantee anonymity, encryption, and privacy on the internet in opposition to China’s or another nation’s laws
  • To freedom at their workplaces from harassment & discrimination
  • To not be censored at work, and to not have their external communities be censored within the products they build

Notice, this is very much a political agenda.  These -apart from harassment & discrimination- are not things most people working in most workplaces expect or demand to have a say in. I certainly don’t at my work place. We use our government for that. We go to the polls for that. We pester our representatives in government to write laws for this kind of stuff.

But on the frontier, there is no government and there is no law.

Notice too that these employees’ demands are not material. That is to say, they aren’t about compensation, a shorter work week, more time off, or a greater share of revenue. They are not about the relationship of the employee with the employer, largely. They are more about product development, about the next stage of the conquest of the frontier.

Notice too that these employees are free to leave the employ of their workplaces, just like you and I are. But they largely….don’t.

And notice that largely, these employees are looking out for their own cohorts’ interests in the development of those products. Not your’s necessarily, nor mine necessarily.

And unlike my workplace, or your workplace, the work these employees do at their workplace intersects and impacts us at our homes and work. Daily. Globally. Their work impacts you and me, and our loved ones, and people as far away as Myanmar, South Africa, and New Zealand.

Mostly, I like their agenda. But it’s inherently a non-democratic agenda because I have no say in how the products are being developed. It’s an agenda that includes some laudable aims & goals -especially as it pertains to empowering marginalized groups- but it’s still an agenda that’s predicated upon their employers & sponsors conquering what little remains of Hannah Arendt’s ‘public’ and all the institutions thereof. It’s an agenda not necessarily at odds with Zuboff’s surveillance capitalism, so long as their cohorts are protected -favored even- on the frontier.

And I’m not sure I’m okay with that.

*I call this arrangement Capitaltech, and you can see how it works here, in this chart which I made in Visio, based largely off Everett Rogers’ Theory of Diffusion of Innovation with some bits added on from Wardley & Brechin.

**Notice Apple still primarily designs, builds and sells tangible products to customers. The business is therefore different & relationship between buyer & seller is transparent and easy to understand

Enterprise Secrets + Privileged Account Management | CyberArk at #XFD1

Managing Enterprise Secrets & Privileged accounts has to be one of the most difficult jobs in Information Technology today, and one of the least transparent to the business. Bad guys have painted a target on admin’s backs, regulators are chomping at the bit as more consumer data is lost online, and Compliance officers are scrambling to understand the landscape and adapt to new rules from overseas. And yet the business may not even realize that unsung heroes in IT are still managing a stack of hardware & software designed to fulfill 1990s-era security models.

Take it from me: I know this pain well. Even if you do have an internal identity system, say Active Directory, it can be difficult to get all the bits from your Storage, Network, Compute & cloud systems to run a proper AAA model against your AD Forest. Even more difficult: figuring out how to audit the records of Active Directory (or NPS/RADIUS or ADFS or OAuth2/SAML glues) to present to your Compliance officers.

Yet in the background, a constant churn of news that only raises the pessimism bar higher: Target. Anthem. Maersk. Equifax. Facebook. Marriot. The goddamned CIA and the f****** National Security Agency. I made a Visio Timeline because I was having difficulty tracking all the breaches, and I’ve run out of room! And let’s not forget the business and your user colleagues’ need for secrets too as consumer technology continues to eat away at the Enterprise and as more of the economy is digitized. By 5pm most days, IT admins are just hoping to make it to retirement in 10 years without their orgs getting popped by a black hat.

cyberark-logoEnter CyberArk. This Silicon Valley company was founded in 1999, which is impressive to me. It’s not often you’ll find a company that’s been selling a product that handles Enterprise secrets + PAM for 20 years, at least a decade longer by my count than the popular consumer password management companies that are now sashaying their way into your Enterprise, as if they understand the challenge you’re facing. At Security Field Day 1 (#XFD1), CyberArk’s maturity & comprehension of the challenge of securing the enterprise really showed.

CyberArk’s Privileged Access Security Suite is a mature & fully-featured secrets + PAM tool. I was super-impressed with the demo their Global Director of Systems Engineering, Brandon Traffanstedt, gave us back in December 2018 in sunny San Jose. I came prepared to endure a boring password management demo; I left impressed at what I had seen, with only a single caveat.

Not only was CyberArk’s product comprehensive, it was bad-ass, with one exception. I saw:

  •  An SSH session opened to a network device’s command line, with a second factor prompt before access was granted
  • Full auditing + screen recordings of a Privileged Account accessing a protected server, just the kind of thing that reassures the business that you, as an admin, have nothing to hide, are not an ‘insider threat’ and are 100% transparent in your work.
  • Deep integration into Windows’ Win32 API, hooking into parts of the OS I’d not seen before outside of Microsoft products, including Credential Management
  • Full integration & support for MacOS
  • OAUTH2/SAML support and full support for your ADFS infrastructure
  • Cloud secrets & PAM management across AWS (and soon) Azure
  • Full support for your RADIUS infrastructure & 802.11x, whether via Microsoft’s NPS or some other solution
  • Automated credential rotation so that you don’t have to scramble when a fellow admin changes jobs, is fired for negligence, or joins Edward Snowden in Moscow
  • Secure sharing of secrets among your privileged IT colleagues
  • An offline, secured, and high-entropy password in a sealed envelope you can hand to the business for peace of mind

I’ve been working in IT for about as long as CyberArk’s been pounding the pavement and trying to convince IT Teams to invest in Enterprise Secrets & PAM software. I was impressed…..particularly because CyberArk scratches an itch that many IT Teams don’t know they have: the security costs & technical debt that a legacy of tactical, rather than strategic, investments that tend to leave an org arrears in 2019’s security landscape.

Por ejemplo: say you’re a mid-market SMB IT shop in the healthcare sector that’s experienced a lot of turnover among its IT admin staff through the years. If you’re the business, you’ve watched as IT Admins come and go, and listened as they’ve pitched tactical solutions to various challenges facing the business. You’ve invested in a few, and most work well enough, but gluing them all together into a comprehensive, strategic, and business-enabling solution has been a challenge.

cyberarkWhile your solutions are working, you’re paying a cost whether you know it or not because more than likely, the technical legwork needed to glue those solutions together into a comprehensive & auditable security framework hasn’t been done. Meanwhile, the regulators are knocking at your door, the pace of breaches quicken, and Brian Krebs’ pen is waiting to write about your company.

CyberArk is a good fit there. No, check that. It’s a *great* fit in that scenario. The product addresses threats to your business from both the inside and the outside. It protects Enterprise secrets -the very thing your admins are targeted for- while shining a bright light on your employee’s Privileged Accounts and how they are used.

It’s a product that’s far beyond anything the consumer password management companies are offering…trust me, I’ve looked at them all. It’s a true Enterprise solution. However….

I will say that one area where CyberArk felt a bit less than polished was in how they’ve architected the sharing & use of secrets with non-admin users working in the business. If we return to the healthcare example, think of a person in your business who needs the credentials to login to a state Medicaid site in order to bill the payor of a medical product.

In fairness, this is a complicated problem…while it’s in the business’ interests to control/maintain/audit all secrets, including to third party sites & services that are outside of IT’s domain, the mix of devices/browser here is a difficult puzzle to solve. Yet it’s here that CyberArk’s product left me perplexed. They propose intercepting TLS traffic on your user’s endpoints & injecting credentials into your business user’s browsers, whatever they may be.

This seemed to me -at the ass-end of 2018- to be a poor solution. For starters, we’ll soon see TLS 1.3 across more and more websites. TLS 1.3, as my fellow Delegate Jerry Gamblin pointed out, is not something you can intercept, decrypt, and inject credentials into. Indeed, other vendors in the security space seem to be steering Enterprise customers away from the expectation that we’ll be able to intercept/inspect/fiddle with TLS 1.3 connections. At best, we’ll be able to refuse TLS 1.3 connections in favor of the more Enterprise-friendly TLS 1.2 connections, but even here, the Enterprise’s political power & ability to influence the market & standards bodies is lacking, and Google, for better & worse, rules the roost. Even Microsoft is playing second fiddle here and announced in late 2018 that it would ditch its new Edge browser’s Trident engine in favor of Chromium open source.

Secondly, CyberArk’s solution even here feels archaic. They propose that you put a middlebox in front of your users to accomplish this. This is definitely old-school, calling to mind the many nights/weekends I spent configuring & troubleshooting BlueCoat devices in server rooms across many Southern California businesses. If you’re going to tackle a problem like TLS intercept, you need to think 21st century and go with a cloud interception service, that will follow your users around on the internet. Middleboxes often make your security posture worse, not better.

In my day job, I intercept/inspect TLS connections across several continents and on several thousand endpoints; it’s a tricky science and one that’s filled with compliance & policy questions above my paygrade. Microsoft’s move in the browser arena fills me with questions, and that’s before we consider mobile devices; so too should it fill you with questions if you are looking at CyberArk with an eye towards sharing secrets with non-admin users.

So, caveat emptor on this narrow point friends: a significant selling point of CyberArk’s featured product (injecting secrets into an HTTPS session) may not work a year or two from now. We raised this issue at #XFD1 and CyberArk says they have a plan for it, but eyes open!

Other than that though, I was really impressed. CyberArk gets the challenge facing Enterprise IT in this Wild West era. It understands intuitively complexities of Enterprise secrets, PAM, insider vs outsider threats, and auditing/compliance requirements. The only place it seems to fall short is in sharing credentials from the ‘Vault’ to non-privileged users.

Check it out if:

  • You’ve got a heterogenous stack of best of breed IT hardware & software and you’ve neglected integrating AAA security across that stack
  • You’re in an environment requiring heavy compliance & auditable proof across your stack against both insider & outsider threats
  • You want 2FA/MFA on old network switches, Macs, and Windows Servers
  • You want screen captures of your admin’s work on devices, servers, and services that you consider privileged
  • You’ve got cloud/SaaS management challenges even as you’ve centralized identity in on-prem Active Directory or other system

Ignore it if:

  • You’ve only ever bought Microsoft, only have Windows PCs & servers and Microsoft applications, and you have an MCSE on staff who understands Kerberos, Active Directory, NPS, RADIUS, ADFS, OAUTH2/SAML, and has configured your AD environment to comply with various regulatory statutes and compliance regimes

Other Coverage:

Disclosures
This blog post was written by me, Jeff Wilson, for publication on my blog, wilson.tech. I was not compensated by CyberArk to compose this blog post, and CyberArk did not see it prior to its publication. I learned about the CyberArk products during Security Field Day 1 (#XFD1) an event for IT, Security, and Enterprise influencers that was held in December 2018 in & around Silicon Valley, California. The Gestalt IT group paid for my airfare, accommodations, and meals during the time I was in greater San Jose, CA area. CyberArk and other sponsors paid Gestalt IT to bring Delegate influencers like me to #XFD1. 
I received no monetary compensation otherwise, save for the swag listed below
CyberArk swag I took home:
  • A ballpoint pen
About Me: My name is Jeff Wilson. I am a 20 year IT Professional with a security focus. I hold a GSEC from the SANS Institute, as well as a Bachelor’s Degree in History & a Master’s in Public Administration, both of which are from CalState. I live & work in Southern California. You can reach me on twitter @jeffwilsontech or via email at blog@wilson.tech

Cloud Field Day 3 | Morpheus Data | #CFD3

Morpheus Data was our first sponsor at #CFD3 and, as is my custom before Tech Field Day events, I had done zero prep work on Morpheus. I had never heard of the firm, and as first-at-bat sponsors for #CFD3, they were facing 12 delegates full of energy and with decades of Information Technology experience between them. So how’d they do? I came away impressed. Let me tell you why: they have a heart for operations, and I’m an operations guy.

Morpheus Data – Background

I found Morpheus Data’s story pretty compelling when I read up on it later. The company started off more or less as an internal product inside a cost center of Bertram Capital, a private equity firm in the Bay Area. Now every company has a founding mythology, but Morpheus’s range true to me. Here, I’ll quote from their site:

Bertram Labs is a world-class team of software developers and ops professionals whose sole purpose is to rapidly implement IT solutions to fuel the growth of the Bertram portfolio. In 2010, that team needed a 100% infrastructure agnostic cloud management platform which would integrate with the DevOps tools they were using to develop and deploy applications for a range of customers on an unpredictable mix of heterogeneous infrastructure. Such a tool didn’t exist so Bertram Labs created their own solution…

Just that phrase right there -an unpredictable mix of heterogenous infrastructure- comprises the je nais se qua of my success as an 18 year IT Pro. Using ratified standards sent to us from on high by the greyhairs at the IETF & IEEE ivory towers, a competent IT Pro like myself can string together disparate hardware systems into something rational because most vendors sometimes follow those standards.

But it’s very hard work.  It’s not cheap either. And that act -that integration of a Cisco PoE switch with an Aruba access point or an iSCSI storage array with a bunch of Dell servers- isn’t bringing much value to the business. Perhaps it would be different if IT Shops could just start over with a rational greenfield infrastructure design, but that’s rare in my experience because the needs of IT aren’t necessarily aligned with the needs of the business.

Morpheus Data says they grew out of that exact scenario, which is immediately familiar to me as an ops guy. I find that story pretty encouraging; an internal DevOps team working for a private equity firm was able to productize their in-house scripts & techniques and are now a separate company. Damn near inspiring!

So what are they selling?

It’s Glue, basically. But well-articulated & rational glue

Morpheus’ pitch is that their suite of products can take the pain out of managing & provisioning services from your stack of heterogenous stuff whether it’s on-premises, in one cloud, or several clouds. And by taking the pain out, you can move faster and bring more value to the business.

I’m not going to get into each product because frankly, I think they’re poorly named and not very exciting (Sharepoint-esque in a way: Analytics, Governance, Automation, Evolution, Integrations). But don’t let the naming confuse or dissaude you; it’s an exciting product and the pricing model is simple to understand.clover-b4ff8d514c9356e8860551f79c48ff7c

Instead, let me describe to you what I saw during Morpheus’ Demo at #CFD:

  • Performance data from On-Premise virtualization servers running Hyper-V, VMware, and even Citrix’s XenServer all in one part of the Morpheus web-based portal
  • You can drill-down from each host to look at VM performance data too. Morpheus says they’re able to hook into both Hyper-V performance counters and VMware’s performance counters. That’s pretty awesome for a hetergeonous shop
  • Performance & controls over IaaS & PaaS instances in both Azure & AWS, again in the same screen
  • Menu-driven wizards that let you instantly provision a new virtual machine pre-configured for whatever service you want to run on it. Again -this could be done in the same tool and you can pick where you want it to go
  • Cost data from each public clouds
  • Rich RBAC controls, which is very important to me from a security & integrity standpoint
  • A composable role-based interface. Por ejemplo, you can let your dev team login to Morpheus and not worry about him or her offlining a .vhdx on a Hyper-V server

This chart from their website sums up their offering nicely in comparison with other vendors in this space.

morpheus

Concluding Thoughts

I’ve worked in IT environments where purchasing has been less than most people would consider as rational. Indeed, I’ve worked at places where we had the very best equipment from multiple vendors, but nobody had the time or talent to integrate it all into a smooth & functional machine in service to the business.

Stepping back, the very nature of the integration puzzle has changed. I mentioned above that a competent IT Pro could stitch together infrastructure that used IETF, IEEE, w3c and other standards-based technologies. Indeed that’s been the story of my career.

But in 2018, the world’s moved on from that, for better and worse. The world’s moved on to proprietary Application Programming Interfaces (APIs), and so I’ve moved with it, creating my own Powershell functions and Python scripts to interact with cloud-based APIs. You can do this too, given enough time & study.

But let’s be honest: it’s hard enough to manage & integrate a heteregenous stack of best-of-breed stuff on-premises. Now your boss comes to you and wants you to add some Azure services & Office 365. And then someone on the business side orders up some Lambdas in AWS, surprise! Or perhaps a distant IT group at your company just went and bought Cloudflare or Rackspace. If you’re still trying to solve standards-based puzzles of yesteryear, while learning how to develop scripts & tools for use in a world of proprietary APIs, you’re probably not bringing much value to the business.

And that’s where Morpheus sees itself slotting in nicely…they’ve done the hard work of integrating with both your legacy on-premises standards-based systems and the API-driven cloud ones, and they release new integrations ‘every two or three weeks.’ They even take requests, so if you’ve got a bespoke stack of stuff that doesn’t surface SNMP properly, you can propose Morpheus build an integration for it.

Sidenote: One of the more dev-focused delegates at #CFD3 criticized the prodcut as too ops-friendly (nobody cares to see all that stuff! he said), but I had to push back on him because details are important for ops teams, and Morpheus can surface an interface that’s safe for devs to use. And that’s why I say they’ve got a heart for operations teams.

On pricing: the products which again, have somewhat confusing names, at least offer simplified pricing. To get workload & ‘core features’ running on a VM in your datacenter, you’ll need to spend $25k to start. That seems high to me, but you’re essentially buying a DevOps integrator & engineer who can work 24/7 and doesn’t need health insurance or take vacation, which is pretty cool, and which helps you bring value to the business.

Disclosures
This blog post was written by me, Jeff Wilson, for publication on my blog, wilson.tech. I was not compensated by Morpheus Data to compose this blog post, and Morpheus did not see it prior to its publication. I learned about the Morpheus Data products during Cloud Field Day 3, an event for IT & Enterprise influencers that was held in April 2018 in Santa Clara California. The Gestalt IT group paid for my airfare, accomodations, and meals during the time I was in Santa Clara. Morpheus and other sponsors paid Gestalt IT to bring Delegate influencers like me to #CFD3
Morpheus Data shwag I took home
  • Cool stickers
  • A t-shirt

Defending IT amidst the novel WannaCry worm

It’s been a hell of a few days here in the trenches of Information Technology in 2017. Where to begin?

Between explaining how this all works to concerned friends & family, answering my employer’s questions about our patching posture & status, and reading the news & analysis, I think it’s safe to say that WCry has been in my thoughts for every one of the last 72 hours, including the 24 hours of Mother’s Day and all the hours I spent in restless slumber.

Yes, that’s right. WCry was on my mind even as I celebrated Mother’s day for the three women I’m close to in my life who are mothers. Wow. Just wow.

Having had the chance to catch my breath, I’ve got some informed observations about this global incident from my perspective as an IT Pro. Why is WCry as interesting & novel as it is potent and effective in 2017? And is there any defense of an IT team one might make if their organization got pwned by WCry?

I contemplate both questions below.

WCry successfully chains a social engineering attack with a technical exploit resulting in automated organization pwnage
WCry begins as a social engineering/phishing attack on users in the place they love and hate by equal measure: their Inbox. Using Subject lines that draw the eye, the messages include malicious attachments. This facet of WCry is not new of course…..it’s routine and has been in IT for at least two decades.

How WannaCry works

Once the attachment is clicked, WCry pivots, unleashing an NSA-built cyberweapon upon the enterprise by scanning port 445 across the local /24, cycling through cached RDP accounts and calling special attention to SQL & Exchange services, presumably to price the ransom accordingly.

Then it encrypts. Nearly everything.

All of this from a single email opened by a gullible user.

This behavior -socially engineered attack on human meatbag + scan + pivot to the rest of the network- is also not novel, new or remarkable.  In fact, security Pros call this behavior “moving laterally” through an enterprise and they usually talk about it being done from “jump box” or “beach head” that’s been compromised via social engineering. Typically, security pros will reserve those terms to describe the behavior of a skilled & hostile hacker meatbag intent on pwning a targeted organization.

Where WCry is novel is that it in effect automates the hacker out of the picture, making the whole org pwnage process way more efficient. This is Organization-crippling, self-replicating malware at scale. Think Sony Pictures 2014, applied everywhere automatically minus the North Korean hacker units at the keyboard.

 

The red Wcry “Ooops” message is both informative and visually impressive, which multiplies its influence beyond its victims
As these things go, I couldn’t help but be impressed with Wcry’s incredibly detailed and anxiety-inducing UI announcing a host’s Wcry infection:

This image, or some variant thereof, has appeared on everything from train station arrival/departure boards to manufacturing floor PCs to hospital MRIs to good old-fashioned desktop PCs in Russia’s Interior Ministry. The psychological effects of seeing this image on infected hardware, then seeing it again on popular social media sites, the evening news, and newspapers around the world over the last few days are hard to determine, but I know this: this had an effect on normal consumers and users of technology across the globe. Sitting on my lap Saturday, my four year old saw the image in my personal OneNote pastebin and asked me, “Daddy, is that an alarm? Why does it show a lock? Do you have key?”

What’s interesting is that while computer users saw this or a screensaver version of this image, in reality you could click past it or minimize it in some way. Yet images of this application have proliferated on Twitter, FaceTube and elsewhere. Ransomware used to just announce itself in the root of your file share or your c:\user\username\documents folder: now it poses for screen caps and cell phone pics which multiplies its effectiveness as a PsyOps weapon. By Saturday I was reading multiple articles in my iPad’s Apple News about how regular people could protect themselves from the ‘global cyberattack.’

Its function is not just about encrypting file shares like earlier ransomware campaigns, but about owning Enterprises
If my organization or any organization I was advising got hit by WCry, my gut feeling is that I wouldn’t feel secure about my Forest/Domain integrity until I burned it down and started over. Why? Well, big IT security organizations like Verizon’s Enterprise Security group typically don’t classify ransomware as a ‘data breach’ event. Yet, as we know, Wcry installs a Pulsar backdoor that enables persistent access in the future. This feels like a very effective escalation of what it means to be ransomed in modern IT organizations, so yeah, I wouldn’t feel secure until our forest/domain was burned to the ground.

It is the manifestation of a Snoverism : Today’s nation-state cyberweapon is tomorrow’s script-kiddie attack
I was listening to the father of Powershell, Jeff Snover once and he implanted yet another Snoverism in my brain.  He said, paraphrasing here, that Today’s nation-state attack is tomorrow’s script-kiddie attack. What the what?

Jeff Snover, speaker of wisdom

Let’s unpack: the democratization of technology, the shift to agile, DevOps, and other development disciplines along with infrastructure automation has lead to a lot of great things being developed, released and consumed by users very quickly. In the consumer world this has been great -Alexa is always improving with new skills…Apple can release security patches rapidly, and FaceTube can instantly perform A/B testing on billions of people simultaneously. But not well understood by many is the fact that Enterprises and even individuals can harness these tools and techniques to instantly build and operate data systems globally, to get their product, whatever it may be, to market faster. The classic example of this is Shadow IT, wherein someone in your finance team purchases a few seats on Salesforce to get around the slow & plodding IT team.

I think Snover was observing that bad guys get the same benefits from modern technology techniques & the cloud as consumers and business users do.

And as I write this on Monday, what are we seeing? WCry is posted on GitHub and new variants are being created without the kill-switch/sandbox detection domain. Eternal Blue, the component of Wcry that exploits SMB1, was literally just a few months ago a specialized tool in the NSA’s cyber weapons arsenal. By tomorrow it will be available to any kid who wants it, or, even worse, as a push-button turn-key service anybody can employ against anybody else.

The democratization of technology means that no elite or special knowledge, techniques or tools are required to harness technology to some end. All you need is motive and motivation to do things at scale. This week, we learned that the democratization of technology is a huge double-edged sword.

It was blunted by a clever researcher for about $11
Again on the democratization of technology front, I find it fascinating that MalwareTech was able to blunt this attack by spending $11 of his own money to purchase the domain he found encoded in the output of his decompile. He’s the best example of what a can-do technologist can do, given the right amount of tools and freedom to pursue his craft.

It has laid bare the heavy costs of technical debt for which there is no obvious solution
Technical debt is a term used in software engineering circles and computer science curricula, but I also think it can and should apply to infrastructure thinking. What’s technical debt? Take it away Wikipedia:

Technical Debt is a metaphor referring to the eventual consequences of poor system design, software architecture, or software development within a codebase. The debt can be thought of as work that needs to be done before a particular job can be considered proper or complete. If the debt is not repaid, then it will keep on accumulating interest, making it hard to implement changes later on.

I can’t tell you how many times and at how many organizations I’ve seen this play out. Technical Debt, from an IT Pro’s perspective, can be the refusal to correct a misconfiguration of an important device upon which many services are dependent, or it can be a poorly-designed security regime that takes bad practice and cements it into formal process & habit, or it can be a refusal to give IT the necessary political cover & power to change bad practices or bad design into something durable and agile, or it can be refusing to patch your systems out of fear or a desire to kick the can down the road a bit.
Over time, efforts will be made to pay that technical debt down, but unless a conscious effort is made consistently to keep it low, technical debt eventually -inevitably- becomes just as crippling to an organization as credit card debt becomes to a consumer. Changes to IT systems that in other organizations are routine & easy become hard and difficult; and hard changes in other companies are close to impossible in yours.

This is a really bad place to be for an IT Pro, and now WCry made it even worse by exploiting organizations that have high technical debt, particularly as it relates to patching. Indeed, it’s almost as if the author of this malware understood at a basic fundamental level how much technical debt organizations in the real world carry.

There is no obvious solution to this. We can’t force people to use technology a certain way, or even to think of technology in a certain way. The point of going into business is to make money, not to build durable & secure and flexible technology systems, unless that is your business. Cloud services are the obvious answer, but they can’t do things like run MRI machines or interface with robots on the Nissan assembly line. At least not yet. And nobody wants regulation, but that’s a topic for another post.

It has shown how hard it is to maintain & patch systems that are in-use for more than a typical workday
If we ignore the way WCry rampaged through Russia, China and other places where properly licensing your software is considered optional, something else interesting emerges: the organizations that were hardest hit by Wcry were ones in which technology is likely in use beyond the standard 8 hour workday, which likely makes patching those technology systems all the more difficult.

While reporting on the NHS fiasco has zoomed in on the fact that the UK’s healthcare system had Windows XP widely deployed, I don’t think that tells the whole story, even if it’s true that 100% of NHS systems ran XP, it still doesn’t tell the whole story.  I can easily see how patching in such environments could be difficult based on how much those systems are used.  Hospitals and even out-patient facilities typically operate more than 8 hours a day; finding a slot of time in a given 24 hour period in which you can with the consent of the hospital, offline healthcare devices like MRI machines to update & reboot them is probably more difficult than it is in a company where systems are only required to be up between 7am and 6pm, for instance.

On and on down the list of Wcrypt’s corporate vicitms this pattern continues:

  • Nissan: factory controlled machines were infected with WCry. How easy is it to patch these systems amid what is surely a fast-paced, multi-shift, high-volume operating tempo?
  • German Train system: Literally computers that make the trains run on time have been hit by WCry. Trains and planes operate more than 8 hours a day, making them difficult to patch
  • Telefonica & Portugal Telecom: another infrastructure company that operates beyond a standard 8 hour day that got hit by WCry

I know banks & universities were hit as well, but they’re the exception that points at the rule emerging: Security is hard enough in an 8 hour a day organization. But it’s extra, extra hard when half of a 24 hour day, or even 2/3rds of a 24 hour day is off-limits for patching. Without well-understood processes, buy-in and support from management, discipline and focus on the part of a talented IT team,  such high tempo operating environments will inevitably fall behind the security curve and be preyed upon by WCry and its successors.

It has demonstrated dramatically the perpetual tension between uptime, security and the incentives thereof for IT
This is similar to the patching-is-hard-in-high-tempo organizations claim, but focuses on IT incentives. For the first 2o or 30 years of Information Technology, our collective goal and mission in life was to create, build and maintain business systems that have as much uptime as possible. We call this ‘9s’ as in, “how many ya got?!?”, and it’s about the only useful objective measure by which management continues to sign our check.

Here, I’ll show you how it works:

IT Pro # 1: I got five 9s of uptime this month, that’s less than 26 seconds of unplanned downtime!

IT Pro #2: Still doesn’t touch my record in March of 2015, where I had six 9s (2.59 seconds of downtime) for this service!

Uptime is our raison d’etre, the thing we get paid to deliver the most. We do not get paid, in general, to practice our craft the right way, or the best practice way, per se. We certainly do not get paid to guard against science-fiction tales of security threats involving cyber-weapon worms that encrypt all our data.

We are paid to keep things up and running because, at the end of the day, we’re a cost center in the business. It takes a rare and unique and charismatic manager with support from the business to change that mindset, to get an organization beyond a place where it merely views IT as a cost-center and a place to call when things that are supposed to be up are down.

And that’s part of the reason why Wcry was so effective around the globe.

It has spawned a bunch of ignorant commentary from non-technical people who are outraged at Microsoft

Zeynep Tufecki, an outstanding scholar of good reputation studying the impact of technology on society wrote a piece in the NYT this weekend that had my blood boiling. Effectively, she blames Microsoft and incompetent IT teams for this mess:

First, companies like Microsoft should discard the idea that they can abandon people using older software. The money they made from these customers hasn’t expired; neither has their responsibility to fix defects. Besides, Microsoft is sitting on a cash hoard estimated at more than $100 billion (the result of how little tax modern corporations pay and how profitable it is to sell a dominant operating system under monopolistic dynamics with no liability for defects).

This is absurd on its face. She’s essentially arguing that software manufacturers extend warranties on software forever. She continues:

For example, Chromebooks and Apple’s iOS are structurally much more secure because they were designed from the ground up with security in mind, unlike Microsoft’s operating systems.

Tufecki, whom I really like and enjoy reading, is trolling us. 93% of Google’s handsets don’t run the latest Google OS, which means many people -close to a billion by my count- are, through now fault of their own, carrying around devices that aren’t up to date. Should they be supported forever too? And Apple’s iPhone, as much as I love it, can’t run an Assembly line that manufacturers cars nevermind coordinate an MRI machine.

Rubbish. Disappointed she wrote this.

For all the reasons above, Wcry is not the fault of Microsoft any more than it’s the fault of the element Copper. If anything, the fault for this lies in the way we think about and use technology as businesses and as individuals. Certainly, IT shares some of the blame in these organizations, but there are mitigating factors as I spoke about above.

Mostly, I lay the blame at the NSA for losing these damned things in the first place. If they can’t keep things secure, what hope do most IT shops have?

It has inspired at least one headline writer to say your data is safer with FaceTube than with your hospital
Again, more rubbish and uninformed nonsense from the normals. Sure, my data might be safer from third party hackers if I were to house it inside FaceTube, but then again, adtech companies might just buy that same dataset, anonymized, connect dots from that set to my online behavior dataset, and figure out who I really am. That’s FaceTube’s business, after all!

Nothing Finer than a Well-Considered Powershell Module

Kudos to Intel  for recognizing & implementing a full Powershell module for their network adapters.

This is probably old news to most of you (and indeed, I think this was released in 2013) but I’ve just now managed to explore them.

How do I love them? Let me count the ways.

  1. With IntelNetCmdlets, you no longer have to fart around with netsh cmds to get your NICs primed to push packets properly
  2. With IntelNetCmdlets, your Network Engineering colleague in the cube next to you will no longer laugh as you suffer from Restless Finger Syndrome. RFS is characterized by furious mouse clicking interspersed with curses such as, “Goddamnit, I don’t have time to hunt through all these Device Manager menus just to input the Receive Buffer values I want! And I have four adapters! Somebody kill me. Now!”
  3. With IntelNetCmdlets, engineers who dabble in the virtual arts now have yet another tool in the box that can reduce/eliminate human error prior to the creation of an important virtual switch in a well-considered Hyper-V infrastructure.
  4. With IntelNetCmdlets, even your beater lab environment shines a little brighter because these babies work with my favorite NIC of all time, the  I350 T-4 quad port server adapter, which you can now buy brand new (Probably a Chinese knock-off…but the drivers work!) for about $70 on eBay. Suck on that Broadcom NetExtreme and goofy BroadcomCLI!

Here’s an example of what Intel’s Net cmdlets can do for you.

Let’s say you’re building out a host in your homelab, or you just received some new Whitebox x86 servers for a dev environment at work. Now, naturally this box is going to host virtual machines, and it’s likely those VMs will be on shared storage or will be resources in a new cluster…whatever the case, proper care & raising of your physical NICs at this stage in your infrastructure project not only sets you up for success and makes you a winner, but saves potentially hours or days of troubleshooting after you’ve abstracted all this nonsense away with your hypervisor.

Of course this could all be scripted out as part of a Config Mgr task sequence, but let’s not get too fancy here! I’m no MVP and I just want you to kill your need for Device Manager and the cryptic netsh commands, ok?

Gifcam demo time. Here I’m setting the Jumbo packet value in the Windows registry for the four Intel adapters on my I350-T4 card:

jumbopacket

What I love about this is that Intel’s gone the extra-mile with their Netcmdlets. There’s a full Powershell helpfile, with extras if you tag -verbose or -examples to the end of your get-help query. Any setting you need to toggle, it’s there, from “Green Ethernet” to how many RSS queues you want, to whether VMQ is enabled or disabled.

All you need? A quality Intel card (the Pro1000 cards prior to the I350 family don’t support this officially, but you may be able to trick the Proset drivers into it!), the Proset driver package utility (here) and Powershell. Hell, you can even do this while PS Remoting!

 

What are you going to do with all the time I Just saved you? Cheers