San Francisco – Queen of the Pacific Basin

The ultimate proof of Brechin’s thesis can be had in today’s Washington Post which describes the phenomenal, dazzling wealth in the City by the Bay, named after the humble Francis of Assisi. We see via this piece that the Great Queen of the Pacific Basin, the Colossus, home of “tech”, presses on to the profitable colonization of the globe and enslaves humanity to the production of valuable bits, for which no recompense is returned to the producers and deception reigns supreme.

The Bay Area is home to more billionaires per capita than anywhere on Earth, one out of every 11,600 residents, according to Vox. The entire region, as far as two hours away, has been affected by spiraling real estate prices. Venture capitalist John Doerr has claimed that the area’s economic growth is “the greatest legal accumulation of wealth in history.”

Read more

But today’s Bay Area is much much more powerful than it was in the period Brechin describes.

Figure 4. “Man’s Great Storehouse of Wealth.” A graphic in the Hearst newspapers celebrates mining as a violent assault upon the “Beautiful Planet Devoted to His Use.” San Francisco Examiner, February 8, 1907. -Brechin, Imperial San Francisco, 2006, UC Berkeley Press

Continue reading “San Francisco – Queen of the Pacific Basin”

California Technology Exports

Check out this sentence. I’ll reveal who wrote it later:

…the American West had been the most fertile field for technical innovation…California engineers exported their technology to the rest of the world and improved on that which they imported from everywhere else.

Interesting sentence, right? The author is making the point that California, particularly the Bay Area in this case, is a hub of technical innovation and engineering prowess.

And indeed it is. I mean just look all around us. Silicon Valley companies dominate the world. Three of the top five technology companies (Google, Facebook, Apple) are headquartered there, and the other two, Microsoft & Amazon, have significant presence in Silicon Valley.

Consider those five companies and what they’ve done. Just as the author alleges, those five companies have found a formula for success; they’ve “imported from everywhere else” elemental technology primitives, things like standardized and open protocols built by academics and expert committees in the IETF, IEEE and other standards bodies. These companies have taken those elemental primitives and packaged them up into new exciting innovations and won dominance in the marketplace with them. How much dominance?

Look at this chart I made in Excel. $3.5+ trillion of market dominance, that’s how much dominance. And notice how few they actually employ compared to other titans of the marketplace. They’re massively efficient. That’s the whole point. That’s why capital is so excited about the Big 5.

marketcap
Numbers are out of date reflecting 2017 LTM Revenue & employment numbers but you get the idea

All around the world, people have tried but largely failed to replicate the supposed success of this vibrant hive of technical & engineering prowess. I hear it all the time on podcasts, I read it on Twitter, I read it in blogs. Everyone wants to be Silicon Valley, to be the exciting hub of innovation. Indeed, they want to be the next Silicon Valley, as if this is a repeatable formula there for the taking, as if you could just divine it out of the ether and bam, the next Silicon Valley. 

You see the big 5 marketed endlessly by the apostles of the Disruption Gospel, by the trade press, by us, even when we just think we’re talking about a new device or service. Oh yeah, I love this new feature on my Android. Oh Instagram is introducing end-to-end encryption & direct messaging. People love the products they’re using from these big five companies, and some study them so much they’ve launched ancillary careers just by studying how they work.  I’ve mentioned it before how I admire Ben Thompson, of stratechery.com for the one-man punditry business he’s built atop what he calls Aggregation Theory.

And the founders! We construct mythologies about them too. We build them up into icons. They collectively have more money than God or the tycoons of old.

Now circle your mind back to the quoted sentence. That’s it. Now let’s zoom out:

By 1893, the renowned Canadian mining operator James Douglas could claim that the American West had been the most fertile field for technical innovation in the development of hardware, techniques, and chemistry. California engineers exported their technology to the rest of the world and improved on that which they imported from everywhere else.

The quoted passage is from Dr Gray Brechin’s masterpiece polemic, Imperial San Francisco:Urban Power, Earthly Ruinpublished by University of California Press in 1999, revised in 2006.

Brechin, is, in the words of people I follow on Twitter, my spirit animal. He’s a Geographic Historian who lectures at Berkeley and other universities in the Mountain West. His book -which invokes huge themes about mining, agriculture, cities vs rural areas, and what he terms the Anglo-Aryan race- is all about the conquest of the frontier, and how that conquest was directed by a cartel of mining interests in San Francisco just after the start of the Gold Rush. If you’re interested in Manifest Destiny, you can’t miss this book.

Throughout his polemic, Brechin details the ruthlessness of the early titans of gold & silver mining in and around San Francisco. How they pushed out or simply killed natives. How President Polk, on discovery of gold in California, sparked a war with Mexico and ultimately won control of the west for America. How the early miners scooped up and collected the easy gold first, then pitched a false vision of California to the rest of America and got suckers to move out west for cheap & easy gold. How the miners & miner interests leveled entire forests in the Sierra Nevada, changed the course of rivers, dynamited and blasted their way deep into the scarred earth. And how, once the great con was over, they set their eyes westward again, to spreading the Anglo-Aryan race across the Pacific Basin from the mouth of the Golden Gate.

It’s really a yarn, quite the page turner I tell you. Definitely a great purchase, especially if you’re interested in place and history. Brechin even links the mining & mineral themes almost up to the present day, with the founding of Lawrence Livermore Labs in the east Bay, and its work on developing nuclear weapons.

We see all the time in technology commentary people invoking the same themes Brechin masterfully describes. They talk of atoms versus bits, as in the mining of precious metal atoms vs the mining of non-physical bits, or elements of technology. We ourselves call the titans of bit-mining today founders, and we all listen to the founders as they pitch a vision that, like the mining cartels and newspaper barons before them, results in more wealth accruing to them, and, like the rubes we are, only marginal value for the rest of us*.

It is hardly surprising that the bronze men at the prow of the Pioneer Monument were gold panners working the Sierra placers. California artists almost always depicted the Western miners as free men working under friendly Western skies—not underground,not for others, and not in squalor of their own creation. Such hardy individuals quickly came to symbolize Western opportunity itself, for they were the first to tap untouched bonanzas amid then-unspoiled scenery, and they remain the most enduring agents in the legend of entrepreneurial independence and of he-men living close to nature’s ample bosom.

ibid, Chapter 1, A Promised Land Plundered

And just as the gold miners of the 19th century externalized costs onto society, the environment, indigenous peoples, the Chinese,so too do the mining titans of the 21st Century externalize their costs onto our society. 

These founders, and the people working to sell the vision have, like the mining cartels before them, become digital prophets and invoke almost with religious intensity the themes of the frontier, the very words & phrases of Manifest Destiny.Simon Wardley, for instance, has built another business atop bits and bit mining. He calls them Wardley Maps, and they offer strategic advice and interesting mapping techniques to software engineers & technology companies. Wardley consistently uses the words pioneers, settlers, town planners and ‘uncharted’ as if there’s still more frontier left to exploit.

Untitled pictureThe founders in charge of today’s mining cartels have been using these words and phrases for more than a decade. I just don’t think we realized they actually meant what they were saying.  I think we all got confused by the razzle dazzle of what we saw on our screens, and so we listened to and trusted the razzle dazzle prophets and founders. In short order, we’ve all adopted the language of this new frontier. We’ve all taken Manifest Destiny a step further, even if we’d object to the old Manifest Destiny in principal if not in our history. Because we don’t see the metaphors the founders use for what they truly are: actual frontier-speak. 

The founders’ conquests are occurring in and around San Francisco, where the last frontier closed a little over a century ago. It’s a place that, on the surface, looks much different than the one Brechin details in his polemic. Yes, there is chronic homelessness and skyrocketing rents on the surface, but no one could claim San Francisco or the Bay Area is uncivilized, that it is not a world class city, that most people feel safe there.

But San Francisco -and the Bay Area- always looked beautiful. It’s a beautiful and lovely place. As beautiful as it was in 1898 to be sure, probably more so. But that’s just the surface. You’ve got to dig deeper, you’ve got to peer across whatever industry vertical you work in in 2019 to see the real costs. To see the con and misdirection. Until you do that, you’ll miss the externalized costs and exploitation of the 21st century mining cartels. You need to look at the razzle dazzle on your screen and realize the words you’re seeing are deceptive, that the metaphors have been used to misdirect you, to create a ‘smoky hall of mirrors’ effect, as I called it in an earlier essay. And then you’ve got to read the news and study it and think about it: Rohingya violence, violence in India, the amplification of bad information, anti-vaxxer ads, measles cases soaring, the flat earth, white supremacy on the march, and so much more. All of it organized, spread, and amplified at lightning speed with tooling created by the founders, their cartels, and the engineering prowess of the Bay Area.

tahoeAs Brechin would point out, the costs of the first mining cartels were hidden from the eyes of the wealthy urbanites in San Francisco as they extracted value out of people and the land far away.  They never saw the destruction of old growth Sierra Nevada forests because they didn’t want to see it. They never saw the Chinese Coolies -practically slave labor- herded into railcars and dispatched post-haste once the mining was done and the railroads were built. They never saw the mud and floods as millions of metric tons of toxins and earth flowed down the Central Valley and into the Bay itself. They never saw any of the costs because those costs were intentionally remote.

But in our age, we do see the costs. The exploitation. We see the costs all the time and everyday on our screens, if we just flip the script and study a little bit. You see the costs and you even think about the costs in the privacy of your own home, with yesterday’s Momo freakout. You see the costs but you don’t conceive of them as costs on you or your loved ones. You think of them as social media problems or platform abuse. 

Zoom out a bit, and the vista becomes clear. You see that the founders imported the elemental primitives of 20th Century standards bodies -things like TCP/IP, SMTP, and DNS, the WWW, and packet-switched networking- and got busy constructing and exporting Manifest Destiny 2.0 with those elements. And they’ve been telling us what they’ve been doing the whole time, we just didn’t realize it.

*I have noted in a previous essay how wonderful these technologies have been for women, People of Color and LGBTQ folks. I celebrate their agenda and the fact that they are seizing real political power long denied to them in the old, physical world. The value & benefit to them is immense, and I acknowledge that, and I want to ally with them in my politics. But this essay explores the costs side of the equation.

On the Advocacy & Entitlement of tech workers

Google backs out of Pentagon cloud contract after workers protest20,000 Googlers walk-out to protest sexual harassment and workplace discrimination….Microsoft workers protest use of Halolens by US ArmyGooglers protests AI board..Google closes AI board..Microsoft workers stand-up for Chinese tech workers….Googlers claim retaliation for walkout….

On and on over the last 18 months we’ve seen headlines and stories like this, stories about political advocacy at what are technically -and legally- private workplaces, but which, in reality, function differently. We see these stories on our screens, and we read about the workers and their workplaces, but what are all these stories really about? Are we seeing the birth of a proto-labor movement, or is this something else? Why did I feel support & solidarity with Googlers walking out following sexual harassment at their workplace, while secretly resenting their ability to organize & protest?

Their Workplace and Our Workplace

These are workplaces where all the familiar trappings of American workplaces are present. Por ejemplo, if a Google workplace is in California, I’m 100% certain there’s a “Your rights & responsibilities” placard in the facility that’s meant to inform workers of their rights. There’s likely OSHA placards too. Information about worker’s comp. Minimum wage notices. Exit signage. Fire & building regulations. There’s probably compliance hotlines for employees to dime on bad or unethical behavior they see at their employer. All the legal trappings and rights that labor won for us politically in the 20th century, all those things are at Google, at Microsoft, at Amazon, just like they are at your workplace & my workplace, no matter how big or small it is.

Untitled
A Googler is entitled to complain publicly about workplace politics and decisions and to even hire counsel to assist in their complaint

And yet, step back, and these tech workers enjoy much more liberty -indeed, are entitled to more rights- at their workplaces than we are at our’s. So much liberty, in fact, they can be themselves on our screens. During work hours even. Rarely do they have to say, “Thoughts & Opinions expressed here are mine nad mine alone and do not represent those of my employer, ” like I have to say. They can even show off the gifts they’ve received on social media -no need to disclose how or why they are receiving compensation from other companies- they can just show it off.

They are evidently entitled to so many rights within their workplaces, they can even criticize their superiors in public, by name. Wild!

Meanwhile, the rest of us don’t have that power. We’d be fired *instantly* if we did a 10th of what the tech workers do.

The Tech Workers’ Entitlement to Rights We Don’t Have

This then, is their entitlement, their privilege. They are entitled to lobby and organize and effect political change at their workplace -which intersects with our workplaces & homes by virtue of the internet, our screens and their products-  during work hours, while we cannot do the same at our’s. What explains that and what are their goals?

Well, their entitlement to rights we don’t have springs from their ability to code a world -a frontier- that achieves returns in an effecient manner for the shareholders. They are, in effect, hired guns building out a frontier that ultimately will deliver a reliable annuity to their sponsors, the bank*. Let’s take a look at this chart I made in Excel to understand how this works.

marketcap

Particularly pay attention to the revenue per employee number & market cap/employee. Those numbers are the Rosetta Stone for divining the political power of these tech workers, and indeed, the economic model that SiliconValley itself uses in its conquest of the world, and of the 20th Century’s verticals.

Notice that, for instance, every Google employee generates almost $1.4 million worth of revenue for the company. A Costco employee, on the other hand, generates only about half that much. Notice too that the Googler’s share of market cap (this chart was done on 2017 numbers but generally is accurate) is almost $9 million, 18x that of the Costco employee’s share of that corporation’s market cap.

Apple’s revenue/employee figure is off the charts too. Nearly $2m of revenue is generated by each Apple employee**. Microsoft & Amazon’s are more modest -particularly Amazon’s due to the larger number of people they employ- but even they are able to protest and organize politically at their workplaces, though their employees seem to make fewer waves as compared to Google’s.

Facebook employees, in contrast, are the great outlier. What little we hear from inside Facebook comes from ex-employees. The firm, like Apple, seems to have a stronger management culture than Google, but a less politically aware one. But notice that Facebook -whose revenue per employee numbers are very high- outsources the dirty work of managing its frontier to third party firms. We recently heard from one company’s employees about the work they do to moderate content on the Facebook frontier. It was not a happy story.

Now look at Walmart: Walmart employs almost 3 million people yet its revenue per employee is the lowest of the bunch, as is its market cap/employee figure. How often do we hear from Walmart employees?

The Political Objectives of some Big Tech workers

Every political movement needs to unite disparate and sovereign peoples under a banner of change and a well-understood set of objectives. Typically, we calll this an “agenda” or even a manifesto. As best I can tell, the tech workers’ agenda is this:

  • To have a say in how AI Systems are built and work, such that these products don’t discriminate or target marginalized groups inside the company and outside the company
  • To be free to not work on products that will be used by the US military, public sector governments, or even cities and municipalities
  • To blunt or evade the power and sovereignty of foreign nations like China by delivering products that guarantee anonymity, encryption, and privacy on the internet in opposition to China’s or another nation’s laws
  • To freedom at their workplaces from harassment & discrimination
  • To not be censored at work, and to not have their external communities be censored within the products they build

Notice, this is very much a political agenda.  These -apart from harassment & discrimination- are not things most people working in most workplaces expect or demand to have a say in. I certainly don’t at my work place. We use our government for that. We go to the polls for that. We pester our representatives in government to write laws for this kind of stuff.

But on the frontier, there is no government and there is no law.

Notice too that these employees’ demands are not material. That is to say, they aren’t about compensation, a shorter work week, more time off, or a greater share of revenue. They are not about the relationship of the employee with the employer, largely. They are more about product development, about the next stage of the conquest of the frontier.

Notice too that these employees are free to leave the employ of their workplaces, just like you and I are. But they largely….don’t.

And notice that largely, these employees are looking out for their own cohorts’ interests in the development of those products. Not your’s necessarily, nor mine necessarily.

And unlike my workplace, or your workplace, the work these employees do at their workplace intersects and impacts us at our homes and work. Daily. Globally. Their work impacts you and me, and our loved ones, and people as far away as Myanmar, South Africa, and New Zealand.

Mostly, I like their agenda. But it’s inherently a non-democratic agenda because I have no say in how the products are being developed. It’s an agenda that includes some laudable aims & goals -especially as it pertains to empowering marginalized groups- but it’s still an agenda that’s predicated upon their employers & sponsors conquering what little remains of Hannah Arendt’s ‘public’ and all the institutions thereof. It’s an agenda not necessarily at odds with Zuboff’s surveillance capitalism, so long as their cohorts are protected -favored even- on the frontier.

And I’m not sure I’m okay with that.

*I call this arrangement Capitaltech, and you can see how it works here, in this chart which I made in Visio, based largely off Everett Rogers’ Theory of Diffusion of Innovation with some bits added on from Wardley & Brechin.

**Notice Apple still primarily designs, builds and sells tangible products to customers. The business is therefore different & relationship between buyer & seller is transparent and easy to understand

Defending IT amidst the novel WannaCry worm

It’s been a hell of a few days here in the trenches of Information Technology in 2017. Where to begin?

Between explaining how this all works to concerned friends & family, answering my employer’s questions about our patching posture & status, and reading the news & analysis, I think it’s safe to say that WCry has been in my thoughts for every one of the last 72 hours, including the 24 hours of Mother’s Day and all the hours I spent in restless slumber.

Yes, that’s right. WCry was on my mind even as I celebrated Mother’s day for the three women I’m close to in my life who are mothers. Wow. Just wow.

Having had the chance to catch my breath, I’ve got some informed observations about this global incident from my perspective as an IT Pro. Why is WCry as interesting & novel as it is potent and effective in 2017? And is there any defense of an IT team one might make if their organization got pwned by WCry?

I contemplate both questions below.

WCry successfully chains a social engineering attack with a technical exploit resulting in automated organization pwnage
WCry begins as a social engineering/phishing attack on users in the place they love and hate by equal measure: their Inbox. Using Subject lines that draw the eye, the messages include malicious attachments. This facet of WCry is not new of course…..it’s routine and has been in IT for at least two decades.

How WannaCry works

Once the attachment is clicked, WCry pivots, unleashing an NSA-built cyberweapon upon the enterprise by scanning port 445 across the local /24, cycling through cached RDP accounts and calling special attention to SQL & Exchange services, presumably to price the ransom accordingly.

Then it encrypts. Nearly everything.

All of this from a single email opened by a gullible user.

This behavior -socially engineered attack on human meatbag + scan + pivot to the rest of the network- is also not novel, new or remarkable.  In fact, security Pros call this behavior “moving laterally” through an enterprise and they usually talk about it being done from “jump box” or “beach head” that’s been compromised via social engineering. Typically, security pros will reserve those terms to describe the behavior of a skilled & hostile hacker meatbag intent on pwning a targeted organization.

Where WCry is novel is that it in effect automates the hacker out of the picture, making the whole org pwnage process way more efficient. This is Organization-crippling, self-replicating malware at scale. Think Sony Pictures 2014, applied everywhere automatically minus the North Korean hacker units at the keyboard.

 

The red Wcry “Ooops” message is both informative and visually impressive, which multiplies its influence beyond its victims
As these things go, I couldn’t help but be impressed with Wcry’s incredibly detailed and anxiety-inducing UI announcing a host’s Wcry infection:

This image, or some variant thereof, has appeared on everything from train station arrival/departure boards to manufacturing floor PCs to hospital MRIs to good old-fashioned desktop PCs in Russia’s Interior Ministry. The psychological effects of seeing this image on infected hardware, then seeing it again on popular social media sites, the evening news, and newspapers around the world over the last few days are hard to determine, but I know this: this had an effect on normal consumers and users of technology across the globe. Sitting on my lap Saturday, my four year old saw the image in my personal OneNote pastebin and asked me, “Daddy, is that an alarm? Why does it show a lock? Do you have key?”

What’s interesting is that while computer users saw this or a screensaver version of this image, in reality you could click past it or minimize it in some way. Yet images of this application have proliferated on Twitter, FaceTube and elsewhere. Ransomware used to just announce itself in the root of your file share or your c:\user\username\documents folder: now it poses for screen caps and cell phone pics which multiplies its effectiveness as a PsyOps weapon. By Saturday I was reading multiple articles in my iPad’s Apple News about how regular people could protect themselves from the ‘global cyberattack.’

Its function is not just about encrypting file shares like earlier ransomware campaigns, but about owning Enterprises
If my organization or any organization I was advising got hit by WCry, my gut feeling is that I wouldn’t feel secure about my Forest/Domain integrity until I burned it down and started over. Why? Well, big IT security organizations like Verizon’s Enterprise Security group typically don’t classify ransomware as a ‘data breach’ event. Yet, as we know, Wcry installs a Pulsar backdoor that enables persistent access in the future. This feels like a very effective escalation of what it means to be ransomed in modern IT organizations, so yeah, I wouldn’t feel secure until our forest/domain was burned to the ground.

It is the manifestation of a Snoverism : Today’s nation-state cyberweapon is tomorrow’s script-kiddie attack
I was listening to the father of Powershell, Jeff Snover once and he implanted yet another Snoverism in my brain.  He said, paraphrasing here, that Today’s nation-state attack is tomorrow’s script-kiddie attack. What the what?

Jeff Snover, speaker of wisdom

Let’s unpack: the democratization of technology, the shift to agile, DevOps, and other development disciplines along with infrastructure automation has lead to a lot of great things being developed, released and consumed by users very quickly. In the consumer world this has been great -Alexa is always improving with new skills…Apple can release security patches rapidly, and FaceTube can instantly perform A/B testing on billions of people simultaneously. But not well understood by many is the fact that Enterprises and even individuals can harness these tools and techniques to instantly build and operate data systems globally, to get their product, whatever it may be, to market faster. The classic example of this is Shadow IT, wherein someone in your finance team purchases a few seats on Salesforce to get around the slow & plodding IT team.

I think Snover was observing that bad guys get the same benefits from modern technology techniques & the cloud as consumers and business users do.

And as I write this on Monday, what are we seeing? WCry is posted on GitHub and new variants are being created without the kill-switch/sandbox detection domain. Eternal Blue, the component of Wcry that exploits SMB1, was literally just a few months ago a specialized tool in the NSA’s cyber weapons arsenal. By tomorrow it will be available to any kid who wants it, or, even worse, as a push-button turn-key service anybody can employ against anybody else.

The democratization of technology means that no elite or special knowledge, techniques or tools are required to harness technology to some end. All you need is motive and motivation to do things at scale. This week, we learned that the democratization of technology is a huge double-edged sword.

It was blunted by a clever researcher for about $11
Again on the democratization of technology front, I find it fascinating that MalwareTech was able to blunt this attack by spending $11 of his own money to purchase the domain he found encoded in the output of his decompile. He’s the best example of what a can-do technologist can do, given the right amount of tools and freedom to pursue his craft.

It has laid bare the heavy costs of technical debt for which there is no obvious solution
Technical debt is a term used in software engineering circles and computer science curricula, but I also think it can and should apply to infrastructure thinking. What’s technical debt? Take it away Wikipedia:

Technical Debt is a metaphor referring to the eventual consequences of poor system design, software architecture, or software development within a codebase. The debt can be thought of as work that needs to be done before a particular job can be considered proper or complete. If the debt is not repaid, then it will keep on accumulating interest, making it hard to implement changes later on.

I can’t tell you how many times and at how many organizations I’ve seen this play out. Technical Debt, from an IT Pro’s perspective, can be the refusal to correct a misconfiguration of an important device upon which many services are dependent, or it can be a poorly-designed security regime that takes bad practice and cements it into formal process & habit, or it can be a refusal to give IT the necessary political cover & power to change bad practices or bad design into something durable and agile, or it can be refusing to patch your systems out of fear or a desire to kick the can down the road a bit.
Over time, efforts will be made to pay that technical debt down, but unless a conscious effort is made consistently to keep it low, technical debt eventually -inevitably- becomes just as crippling to an organization as credit card debt becomes to a consumer. Changes to IT systems that in other organizations are routine & easy become hard and difficult; and hard changes in other companies are close to impossible in yours.

This is a really bad place to be for an IT Pro, and now WCry made it even worse by exploiting organizations that have high technical debt, particularly as it relates to patching. Indeed, it’s almost as if the author of this malware understood at a basic fundamental level how much technical debt organizations in the real world carry.

There is no obvious solution to this. We can’t force people to use technology a certain way, or even to think of technology in a certain way. The point of going into business is to make money, not to build durable & secure and flexible technology systems, unless that is your business. Cloud services are the obvious answer, but they can’t do things like run MRI machines or interface with robots on the Nissan assembly line. At least not yet. And nobody wants regulation, but that’s a topic for another post.

It has shown how hard it is to maintain & patch systems that are in-use for more than a typical workday
If we ignore the way WCry rampaged through Russia, China and other places where properly licensing your software is considered optional, something else interesting emerges: the organizations that were hardest hit by Wcry were ones in which technology is likely in use beyond the standard 8 hour workday, which likely makes patching those technology systems all the more difficult.

While reporting on the NHS fiasco has zoomed in on the fact that the UK’s healthcare system had Windows XP widely deployed, I don’t think that tells the whole story, even if it’s true that 100% of NHS systems ran XP, it still doesn’t tell the whole story.  I can easily see how patching in such environments could be difficult based on how much those systems are used.  Hospitals and even out-patient facilities typically operate more than 8 hours a day; finding a slot of time in a given 24 hour period in which you can with the consent of the hospital, offline healthcare devices like MRI machines to update & reboot them is probably more difficult than it is in a company where systems are only required to be up between 7am and 6pm, for instance.

On and on down the list of Wcrypt’s corporate vicitms this pattern continues:

  • Nissan: factory controlled machines were infected with WCry. How easy is it to patch these systems amid what is surely a fast-paced, multi-shift, high-volume operating tempo?
  • German Train system: Literally computers that make the trains run on time have been hit by WCry. Trains and planes operate more than 8 hours a day, making them difficult to patch
  • Telefonica & Portugal Telecom: another infrastructure company that operates beyond a standard 8 hour day that got hit by WCry

I know banks & universities were hit as well, but they’re the exception that points at the rule emerging: Security is hard enough in an 8 hour a day organization. But it’s extra, extra hard when half of a 24 hour day, or even 2/3rds of a 24 hour day is off-limits for patching. Without well-understood processes, buy-in and support from management, discipline and focus on the part of a talented IT team,  such high tempo operating environments will inevitably fall behind the security curve and be preyed upon by WCry and its successors.

It has demonstrated dramatically the perpetual tension between uptime, security and the incentives thereof for IT
This is similar to the patching-is-hard-in-high-tempo organizations claim, but focuses on IT incentives. For the first 2o or 30 years of Information Technology, our collective goal and mission in life was to create, build and maintain business systems that have as much uptime as possible. We call this ‘9s’ as in, “how many ya got?!?”, and it’s about the only useful objective measure by which management continues to sign our check.

Here, I’ll show you how it works:

IT Pro # 1: I got five 9s of uptime this month, that’s less than 26 seconds of unplanned downtime!

IT Pro #2: Still doesn’t touch my record in March of 2015, where I had six 9s (2.59 seconds of downtime) for this service!

Uptime is our raison d’etre, the thing we get paid to deliver the most. We do not get paid, in general, to practice our craft the right way, or the best practice way, per se. We certainly do not get paid to guard against science-fiction tales of security threats involving cyber-weapon worms that encrypt all our data.

We are paid to keep things up and running because, at the end of the day, we’re a cost center in the business. It takes a rare and unique and charismatic manager with support from the business to change that mindset, to get an organization beyond a place where it merely views IT as a cost-center and a place to call when things that are supposed to be up are down.

And that’s part of the reason why Wcry was so effective around the globe.

It has spawned a bunch of ignorant commentary from non-technical people who are outraged at Microsoft

Zeynep Tufecki, an outstanding scholar of good reputation studying the impact of technology on society wrote a piece in the NYT this weekend that had my blood boiling. Effectively, she blames Microsoft and incompetent IT teams for this mess:

First, companies like Microsoft should discard the idea that they can abandon people using older software. The money they made from these customers hasn’t expired; neither has their responsibility to fix defects. Besides, Microsoft is sitting on a cash hoard estimated at more than $100 billion (the result of how little tax modern corporations pay and how profitable it is to sell a dominant operating system under monopolistic dynamics with no liability for defects).

This is absurd on its face. She’s essentially arguing that software manufacturers extend warranties on software forever. She continues:

For example, Chromebooks and Apple’s iOS are structurally much more secure because they were designed from the ground up with security in mind, unlike Microsoft’s operating systems.

Tufecki, whom I really like and enjoy reading, is trolling us. 93% of Google’s handsets don’t run the latest Google OS, which means many people -close to a billion by my count- are, through now fault of their own, carrying around devices that aren’t up to date. Should they be supported forever too? And Apple’s iPhone, as much as I love it, can’t run an Assembly line that manufacturers cars nevermind coordinate an MRI machine.

Rubbish. Disappointed she wrote this.

For all the reasons above, Wcry is not the fault of Microsoft any more than it’s the fault of the element Copper. If anything, the fault for this lies in the way we think about and use technology as businesses and as individuals. Certainly, IT shares some of the blame in these organizations, but there are mitigating factors as I spoke about above.

Mostly, I lay the blame at the NSA for losing these damned things in the first place. If they can’t keep things secure, what hope do most IT shops have?

It has inspired at least one headline writer to say your data is safer with FaceTube than with your hospital
Again, more rubbish and uninformed nonsense from the normals. Sure, my data might be safer from third party hackers if I were to house it inside FaceTube, but then again, adtech companies might just buy that same dataset, anonymized, connect dots from that set to my online behavior dataset, and figure out who I really am. That’s FaceTube’s business, after all!

Find Office problems before they find you with Telemetry server

I’ve not always had a bromance with Microsoft’s Office suite. I cut my word processing teeth on WordPerfect 5.1, did most of my undergrad papers in BeOS’ one productivity suite ((GoBe Productive, still the best Office suite name)) , and touch-typed my way to graduating cum laude in grad school with countless Turabian-style Google Docs papers.

Office?

That was for corporate suits, man. Rich corporate suits.

But all that’s ancient history. Or maybe I’ve become a suit. Either way, I’m loving Office today.

In 2015, Office has transformed into the ultimate agnostic git ‘r done productivity package. It’s free to use in many cases, but if you want to ‘own’ it, you can subscribe to it, just like HBO ((For the IT Pro, this is a huge advantage, as a cheap E-class sub gives you access to your own Exchange instance, your own Sharepoint server, and your own Office tenant. It’s awesome!)) . It’s also available on just about any device or computing system you can think of, works just as well inside a browser as Google Docs does, and has an enormous install base.

telemetry
From the Office Telemetry PDF guide, linked below

Office has become so impressive and so ubiquitous that it’s truly a platform unto itself, consumed a la carte or as part of a well-balanced Microsoft meal. I’m bullish on Windows but if Office’s former partner ever sunsets, I’m convinced my kid and his kid will still grow up in an Office world.

All of that makes Office really important for IT, so important that you as an IT Guy should consider standing-up some easy instrumentation around it.

Enter Office Telemetry, a super-simple package that flows your Office data to a SQL collector, mashes it up, and gives you important insight into how your users are using Office. It also surfaces the problems in Office -or Office documents- before your users do, and it’s free.

Oh, did I mention it’s called Office Telemetry? This thing makes you feel like an astronaut when you’re using it!

Here’s how you deploy it. Total time: about an hour.

  1. Download the Office 2013 ADMX/ADML files for Group Policy and deploy them to your Domain Controllers.
  2. Spin-up a 2008 R2 or 2012 VM, or find a modestly-equipped physical box that at least has Windows Management Framework 3.0/Powershell 3.0 on it. If it has a SQL 2012 instance on it that you can use, even better. If not, don’t stress and proceed to the next step.
  3. Set-aside a folder on a separate volume (ideally) for the telemetry data. If you’re going to flow data from hundreds of Office users, plan for a minimum of 5-25 megabytes per user, at a minimum.
    • If your users are on the WAN, plan accordingly. Telemetry data is pretty lightweight (50k chunks for older Office clients, 64k chunks for Office 2013)
  4. gptelemetryInstall Office ProPlus 2013 or 365 on the VM. You do not need to use an Office 365 license for it to run.
  5. Download the Deploy Office Telemetry powershell script package from TechNet or via Script Browser in Powershell ISE.
  6. Because it’s a script, you’ll need to temporarily change your server’s execution policy, self-sign it, or configure Group Policy as appropriate to run it. TechNet has instructions.
  7. Run the script; it will download SQL 2012 express and install it for you if you don’t have SQL. It will also set proper SMB read/modify permissions on that folder you set up earlier.
  8. As if that wasn’t enough, the script will give you a single registry keyfile you can use to deploy to your user’s machines.
  9. But I prefer the Group Policy/SCCM route. Remember the ADMX files you deployed? Flip the switches as appropriate under User Configuration>Administrative Templates>Microsoft Office 2013> Telemetry Dashboard.
  10. Sit back, and watch the data flow in, and pat yourself on the back because you’re being a proactive IT Pro!

As I’ve deployed this solution, I’ve found broken documents, expensive add-ons that delay Office, and multiple other issues that were easy to resolve but difficult to surface. It’s totally worth your time to install it.

Office Telemetry PDF

Sign of the Times or just the best PKI book ever?

Like a lot of IT Pros, I’ve been studying up on security topics lately, both as a reaction to the increasing amount of breach news (Who got breached this week, Alex?) and because I felt weak in this area.

So, I went shopping for some books. My goals were simply to get a baseline understanding of crypto systems and best-practice guidance on setting up Microsoft Public Key Infrastructures, which I’ve done in the past but without much confidence in the end result.

Well, it turns out there’s not a whole lot of literature on Microsoft PKI systems. It seems the best of the genre is Windows Server 2008 PKI & Certificate Security, a Microsoft Press book published in 2008 and authored by Brian Komar:

pkiwin

This 3.2lb, 800 page book has a 4.9 out of 5 star rating on Amazon, with reviewers calling it the best Microsoft PKI guide out there.

Great! I thought, as I prepared to shell out about $80 and One Click my way to PKI knowledge.

That’s when I noticed that the book is out of print. There are digital versions available from O’Reilly, but it appears most don’t know that.

For the physical book itself, the least expensive used one on Amazon is $749.99. You read that right. $750!

If you want a new copy, there’s one available on Amazon, and it’s $1000.

I immediately jumped over to Camelcamelcamel.com to check the history of this book, thinking there must have been a run on Mr. Komar’s tome as Target, Home Depot, JP Morgan, and Sony Pictures fell.

Result:

pkiprice

 

The price of this book has spiked recently, but Peak PKI was a full three years ago.

I looked up security breaches/events of early 2012. Now correlation != causation, but it’s interesting nonetheless. Hopefully this means there’s a lot of solid Microsoft PKI systems being built out there!

Rather than shell out $750 for the physical book, I decided to get Ivan Ristic’s fantastic Bulletproof SSL/TLS, which I highly recommend. It’s got a chapter on securing Windows infrastructure, but is mostly focused on crypto theory & practical OpenSSL. I’ll buy Komar’s as a digital version next or wait for his forthcoming 2012 R2 revision.

Microsoft’s commitment to open initiatives & the riddle of whitebox networking

On Tuesday Microsoft surprised me by announcing an open switching/networking plan in partnership with Mellanox and as part of the Open Compute initiative.

Wait, what?

Microsoft’s building a switch?

Not quite, but before we get into that, some background on Microsoft’s participation in what I call OpenMania: the cloud & enterprise technology vendor tendency to prefix any standards-ish cooperative work effort with the word Open.

Microsoft’s participating in several OpenMania efforts, but I only really care about these two because they highlight something neat about Microsoft and apply or will soon apply to me, the Converged IT Guy.

Open Compute, or OCP, is the Facebook-led initiative to build agnostic hardware platforms on x86 for the datacenter. I like to think of OCP as a ground-up re-imagining of hardware systems by guys who do software systems.

As part of their participation in OCP, Microsoft is devoting engineering resources and talent into building out specifications, blueprints and full hardware designs for things like this, a 12U converged chassis comprised of storage and compute resources.

ocs
Are those brown Zunes in the blades?

 

Then there’s Open Management Infrastructure (OMI), an initiative of the The Open Group (TOG). Microsoft joined OMI almost three years ago to align & position Windows to share common management frameworks across disparate hardware & software systems.

That’s a lot of words with little meaning, so let me break it down for the Windows guys and gals reading this. The promise of Microsoft’s OMI participation is this: you can configure other people’s hardware and software via the same frameworks your Windows Server runs on (CIM, the next-gen WMI) using the same techniques and tooling you manage other things with: Powershell.

All your management constructs are belong to CIM
All your management constructs are belong to CIM

I’ve been keenly interested in Microsoft & their OMI push because it’s an awesome vision, and it’s real, or real-close at any rate: SMI-S, for instance, is gaining traction as a management play on other people’s hardware/software storage systems ((cf NIMBLE STORAGE NOW INTEGRATES WITH SCVMM)) , and is already baked-into Windows server as a feature you can install and use to manage Windows Storage Spaces, which itself is a first-class citizen of CIMville.

All your CIM classes -running as part of Windows or not- manipulated & managed via Powershell, the same ISE you and I use to deploy Hyper-V hosts, spin-up VMs, manage our tenants in Office 365, fiddle around in Azure, and make each day at work a little better and a little more automated than the last.

That’s the promised land right there, ladies and gentlemen.

Except for networking, the last stubborn holdout in my fevered powershell dream.

Jeff Snover, the architect of the vision, teases me with Powershell Leaf Spine Tweets like this:

//platform.twitter.com/widgets.js

but  I have yet to replace Putty with Powershell, I still have to do show int status rather than show-interface -status “connected” on my switch because I don’t have an Arista or N7K, and few other switches vendors seem to be getting the OMI religion.

All of which makes Microsoft’s Tuesday announcement that it is extending its commitment to OCP’s whitebox switching development really odd yet worthy of more consideration:

The Switch Abstraction Interface (SAI) team at Microsoft is excited to announce that we will showcase our first implementations of the specification at the Open Compute Project Summit, which kicks off today at the San Jose Convention Center. SAI is a specification by the OCP that provides a consistent programming interface for common networking functions implemented by network switch ASIC’s. In addition, SAI allows network switch vendors to continue to build innovative features through extensions.

The SAI v0.92 introduces numerous proposals including:

Access Control Lists (ACL)
Equal Cost Multi Path (ECMP)
Forwarding Data Base (FDB, MAC address table)
Host Interface
Neighbor database, Next hop and next hop groups
Port management
Quality of Service (QoS)
Route, router, and router interfaces

At first glance, I wouldn’t blame you if you thought that this thing, this SAI, means OMI is dead in networking, that managing route/switch via Powershell is gone.

But looking deeper, this development speaks to Microsoft’s unique position in the market (all markets, really!)

  1. SAI is probably more about low-level interaction with Broadcom’s Trident II ((At least that’s my read on the Github repo material)) and Microsoft’s participation in this is more about Azure and less about managing networking stuff w/Powershell
  2. But this is also perhaps Microsoft acknowledging that Linux-powered whitebox switching is really enjoying some momentum, and Microsoft needs to have something in this space

So, let’s review: Microsoft has embraced Open Compute & Open Management. It breaks down like this:

  • Microsoft + OCP =  Contributions of hardware blueprints but also low-level software code for things like ASIC interaction
  • Microsoft + OMI = A long-term strategic push to manage x86 hardware & software systems that may run Windows, but likely run something Linuxy yet

In a perfect world, OCP and OMI would just join forces and be followed by all the web-scale players, the enterprise technology vendors, the storage guys & packet pushers. All would gather together under a banner singing kumbaya and praising agnostic open hardware managed via a common, well-defined framework named CIM that you can plug into any front-end GUI or CLI construct you like.

Alas, it’s not a perfect world and OCP & OMI are different things. In the real world, you still need a proprietary API to manage a storage system, or a costly license to utilize another switchport. And worst of all, in this world, Powershell is not my interface to everything, it is not yet the answer to all IT questions.

Yet Microsoft, by virtue of its position in so many different markets, is very close now to creating its own perfect world. If they find some traction with SAI, I’m certain it won’t be long before you can manage an open Microsoft-designed switch that’s a first-class OMI citizen and gets along famously with Powershell! ((Or buy one, as you can buy the Azure-in-a-box which is simply the OCP blueprint via Dell/Microsoft Cloud Platform System program))