Category Archives: Uncategorized

The Myth of the “Five Nines”

May 10, 2017Uncategorizedleon2

“Five-Nines” refers to something that is available 99.999% of the time. It’s become a catch phrase in various parts of the IT industry.

It’s also complete bullshit.

Sean Hull did a great job explaining why five-nine’s is over-rated in this post. But my point is NOT that this level of reliability is expensive. It’s that it’s nearly impossible to functionally achieve.

I’m also saying that the demands for (or the claims of) “Reliability in the five-nines” are highly over-blown.

Let’s do the math.

In a single minute, 5-9s means you would be unavailable just .0006 of a second.
In an hour, you could have .036 seconds of downtime
In a day, your system would get .86 seconds of breathing room
In a week, you could take a 6.04 second break before being less than 5-9’s
In an entire month, you’d only get 24.192 seconds of downtime
In any given fiscal quarter, you could expect just over a minute – 72.576 seconds, to be precise – of an outage
Half a year? You get over two minutes – 145.152 seconds – where that system was not available
And in a whole year, your 5-9’s system would experience just under 5 minutes (290.304 seconds) of outage. Total. Over the entire year.

You seriously expect any device, server or service to be available to all users for all but 5 minutes in an entire year?

This has implications for us as monitoring professionals. After all, we’re the ones tasked with watching the systems, raising the flag when things become unavailable. When someone is less than 99.999% available, we’rethe ones the system owners come to, asking us to “paint the roses green”. We’re the ones that will have to re-check our numbers, re-calculate our thresholds, and re-explain for the thousandth time that “availability” always carries with it observational bias.

Yes, Mr. CEO, the server was up. It was up and running in a datacenter where a gardener with backhoe had severed the WAN circuit; it was up and running and everyone in the country could see it except for you, because wifi was turned off on your laptop; it was up and running but it showed “down” in monitoring because someone changed the firewall rules so that my monitoring server could no longer reach it…

There’s an even more pressing fact-on-the-ground: Polling cycles are neither constant nor instant. Realistic polling intervals sit at around 1-2 minutes for “ping” type checks, and 5 minutes for data collection. If I’m only checking the status of a server every minute, and my monitoring server is dealing with more than that one machine, the reality is that I won’t be cutting a ticket for a “down” device for 3-5 minutes. That blows your 5-9’s out of the water right there.

But all of that is beside the point. First you need to let me know if your corporate server team is down with 5-9’s availability guarentees. Are they promising that the monthly patch and reboot process will take less than half a minute, end to end?

I’m thinking ‘no’.

Do-Over

May 3, 2017Uncategorizedleon2

The report you’ve been updating for 2 hours, but somehow never hit “save”. The code you tweaked all day but hadn’t gotten far enough to upload to the repository. The server build you had to bail on because the hard drive was faulty (but you only discovered halfway through all the installs). The software patch that SEEMED to be working OK until you pushed it to 100 machines and it failed on 30 of them.

What do you do when you have to start over?

Like most of us you probably curse a little (or a lot), take a walk around the office (or the block), maybe vent to a coworker (or two, or seven).

But after that…

Do you continue to look for ways to salvage the situation? Go searching for backups? Dig online for ways to restore the auto-save copy? Reboot the box to see if “maybe it was just a temporary glitch”?

In short, do you get caught up in your own private dollar auction, stubbornly trying to outbid reality – not so you can win, but so that you lose less?

Or do you accept that this is actually a chance to do it better the second time?

PostScript: Steven Tyler wrote the lyrics to “Walk This Way” the night before Aerosmith was set to record it, as a last-minute addition “Toys in the Attic”. But then he forgot them in the cab on the way to the recording studio. When he realized they were missing, it was too late to find the cab. He had to re-write them more or less from scratch. The original lyrics were never found. Tyler is certain the version we all know is better.

Day 3 – Search

September 11, 2016Uncategorizedblogelul, elulleon2

As IT pro’s, we find ourselves searching for many things. We search for solutions. We search for truth (both regular and capital-T truth). Most of those things we either have a good chance of locating, as long as we’re persistent and intelligent about it.

But one of the searches that many (if not all) IT pro’s undertake is the search for the right fit in their job.

Forums, job boards, and advice columns – not to mention innumerable after-work-over-beer discussions – are filled with tales of horrific bosses, harrowing workplaces, and hideous jobs.

If there were easy answers, they’d be out there already. After 30 years, the only wisdom I can give is this: it’s the same as any other problem. You have to be persistent. You have to be smart. You have to be willing to abandon your pre-conceived notions and start over – again and again if necessary. You have to accept that the solution which worked for another person in another place may not be your solution.

And sometimes the search has to be given up for now, with the trust that you’ll take it up again another day when you are fresh and ready to try again.

Respect Your Elders

January 12, 2016Uncategorizedleon2

This post originally appeared on SolarWinds GeekSpeak

“Oh Geez,” exclaimed the guy who sits 2 desks from me, “that thing is ancient! Why would they give him that?”

Taking the bait, I popped my head over the wall and asked “what is?”

He showed me a text message, sent to him from a buddy—an engineer (EE, actually) who worked for an oil company. My co-worker’s iPhone 6 displayed an image of a laptop we could only describe as “vintage”:

(A Toshiba Tecra 510CDT, which was cutting edge…back in 1997.)

“Wow.” I said. “Those were amazing. I worked on a ton of those. They were serious workhorses—you could practically drop one from a 4 story building and it would still work. I wanted one like nobody’s business, but I could never afford it.”

“OK, back in the day I’m sure they were great,” said my 20-something coworker dismissively. “But what the hell is he going to do with it NOW? Can it even run an OS anymore?”

I realized he was coming from a particular frame of reference that is common to all of us in I.T. Newer is better. Period. With few exceptions (COUGH-Windows M.E.-COUGH), the latest version of something—be it hardware or software—is always a step up from what came before.

While true, it leads to a frame of mind that is patently un-true: a belief that what is old is also irrelevant. Especially for I.T. Professionals, it’s a dangerous line of thought that almost always leads to un-necessary mistakes and avoidable failures.

In fact, ask any I.T. pro who’s been at it for a decade, and you’ll hear story after story:

When programmers used COBOL, back when dinosaurs roamed the earth, one of the fundamental techniques that were drilled into their heads was, “check your inputs.” Thinking about the latest version of exploits, be they an SSLv3 thing like ‘Poodle’, or a SQL injection, or any of a plethora of web based security problems, the fundamental flaw is the server NOT checking its inputs, for sanity.
How about the OSI model? Yes, we all know its required knowledge for many certification exams (and at least one IT joke). But more importantly, it was (and still is) directly relevant to basic network troubleshooting.
Nobody needs to know CORBA database structure anymore, right? Except that a major monitoring tool was originally developed on CORBA and that foundation has stuck. Which is why, if you try to create a folder-inside-a-folder more than 3 times, the entire system corrupts. CORBA (one of the original object-oriented databases) could only handle 3 levels of object containership.
Powershell can be learned without understanding the Unix/Linux command line concepts. But, it’s sure EASIER to learn if you already know how to pipe ls into grep into awk into awk so that you get a list of just the files you want, sorted by date. That technique (among other Unix/Linux concepts) was one of the original goals of Powershell.
Older rev’s of industrial motion-control systems used specific pin-outs on the serial port. The new USB-to-Serial cables don’t mimic those pin-outs correctly, and trying to upload a program with the new cables will render the entire system useless.

And in fact, that’s why my co-worker’s buddy was handed one of those venerable Tecra laptops. It had a standard serial port and it was preloaded with the vendor’s DOS-based ladder-logic programming utility. Nobody expected it to run Windows 10, but it fulfilled a role that modern hardware simply couldn’t have done.

It’s an interesting story, but you have to ask: aside from some interesting anecdotes and a few bizarre use-cases, does this have any relevance to our work day-today?

You bet.

We live in a world where servers, storage, and now the network is rushing toward a quantum singularity of virtualization.

And the “old-timers” in the mainframe team are laughing their butts off as they watch us run in circles, inventing new words to describe techniques they learned at the beginning of their career; making mistakes they solved decades ago; and (worst of all) dismissing everything they know as utterly irrelevant.

Think I’m exaggerating? SAN and NAS look suspiciously like DASD, just on faster hardware. Services like Azure and AWS, for all their glitz and automation, aren’t as far from rented time on a mainframe as we’d like to imagine. And when my company replaces my laptop with a fancy “appliance” that connects to Citrix VDI session, it reminds me of nothing as much as the VAX terminals I supported back in the day.

My point isn’t that I’m a techno-Ecclesiastes shouting “there is nothing new under the sun!” Or some I.T. hipster who was into the cloud before it was cool. My point is that it behooves us to remember that everything we do, and every technology we use, had its origins in something much older than 20 minutes ago.

If we take the time to understand that foundational technology we have the chance to avoid past mis-steps, leverage undocumented efficiencies built into the core of the tools, and build on ideas elegant enough to have withstood the test of time.

#BlogElul Day 5: Accept

August 20, 2015Uncategorizedleon2

Larry Wall (creator of the Perl programming language) famously said, “Most of you are familiar with the virtues of a programmer. There are three, of course: laziness, impatience, and hubris.”

In one brilliantly succinct phrase, Mr. Wall took three traits commonly understood to be a character flaw and re-framed them as virtues.

As I sat and thought about how acceptance is generally seen as a positive trait in life, I realized that in IT it could be just the opposite.

Accepting the status quo, that the system “is what it is”, that things are (or aren’t) changing (or staying the same) and there is nothing that we can do to affect that… all of these are anti-patterns which do us no good.

As I sat and pondered it in the wee hours of the morning, I heard the voice of Master Yoda whisper in my ear:

NOT accepting leads to curiosity
Curiosity leads to hacking
hacking leads to discover
discovery leads to innovation
innovation leads to growth

When we refused to accept, we grow.

Bringing this back around to personal growth, I think there is a time and place when refusing to accept – our perceived limitations, our place (whether that’s in the org chart, or in society at large), our past failures, etc. When we refuse to permit those external forces to define or limit us – that is when we find the path toward personal growth.

Walking Away

June 18, 2014philosophy, rant, Uncategorizedleon2

I deleted my Facebook account today. I didn’t just close that tab in my browser; I didn’t go on a 5 day “technology cleanse”; I didn’t suspend my account.

(You may now clutch your pearls and gasp dramatically. I’ll wait.)

Now that your shock and horror have passed, I wanted to explain why, and why this is not an act worth emulating, and why it is absolutely something I think you should consider emulating.

First the boring reasons:

My original reason for getting onto Facebook was to keep track of my kids. They aren’t on Facebook any more (possibly because I *am*.)
Meanwhile, I found myself spending an inordinate time hitting “refresh” waiting for something interesting to show up.
When something interesting DOES show up that I want to comment on, it fails the 3-point checklist I use to keep me from embarrassing myself (more than usual):
- Is it kind?
- Is it true?
- Is it necessary for me to respond?

AND THEN there’s the whole privacy piece, which has been recently (and in my opinion, best) summarized here. Reading my texts? NSA impersonation? data leaks? Fraud? It’s a security hit-parade.

I’ll have to admit, the whole ‘turning on my microphone when I’m doing a status update” thing is just a step too far for me. I deleted the FB app off all my mobile devices immediately.

And then this announcement that FB is selling my information to advertisers. Taken on its own, this would be a yawner. But for me it was one in a growing list of concerns.

I am completely clear that FB isn’t the only offender. The other social media platforms have similar problems. In fact, it’s still worth having the discussion on whether “social media” should come with any illusions of privacy at all. But FB currently seems to have the most egregious flaws. And (for me at least) very few benefits.

That’s not going to be true for everyone. When I made the announcement that I would be shutting down my account, many people told me they couldn’t survive (read “stay in touch”, “keep up to date”, or even “while away the boring hours before/after/during work”) without it.

And that’s fine. My decision to delete my Facebook account is not something you should feel compelled to emulate.

On the other hand, FB is acting less like a social media platform and more like a corporate entity looking to monetize any way they can. I don’t begrudge them that, and I appreciate the fact that they made an announcement about selling my information. This time. Because people are looking.

It’s clear that Facebook is really not interested in your privacy in any meaningful way, and as a user you simply can’t expect it.

So, while this isn’t an action you should feel compelled to emulate, you may want to consider it never the less.

If you haven’t visited in a while, or if you are following a link looking for one of my old documents, you may be asking yourself “where did everything go?”

In a word, I got focused.

AdatoSystems has always been a “me, myself and I” operation, and the website has therefore always focused on whatever it is that I was focused on. Since I split my time fairly evenly between the completely-not-overlapping areas of systems monitoring and automation on the one hand, and website design and monitoring on the other, the website has always suffered from a bit of an identity crisis.

While it didn’t bother me too much, it was increasingly confusing to my customers. The old logo – highly monitoring-centric – was just a bit overwhelming to my website customers. And my blog comment was almost entirely related to web design, which left out people who came here looking for monitoring ideas.

To resolve this, I created two sub-sites, each dedicated to it’s specific area of focus. There is monitoring.adatosystems.com (focused on monitoring, of course). And the site devoted to web design is named, aptly enough, webdesign.adatosystems.com.

This original site, www.adatosystems.com, remains, and will be the place I post some of my more generic (or esoteric) ideas.

Everything will link back to the social media you are probably using to following me anyway – my Twitter feed, Facebook page and LinkedIn profile.

So look for more monitoring articles to appear on the Monitoring and Webdesign subsites in the coming days and weeks. I’ll be moving some of the old stuff from the main site that belongs here, as well as generating brand new content for you to enjoy.

You can now find this post over on my web desgin site. Thanks!

What’s the heaviest load you can carry? How much work can you handle before you crack?

I need to shift gears for a minute and talk about something personal. If you don’t feel like reading about me gushing about my family, it’s time to click “Next”.

I really love my kids. All (4) of them. They do some amazing, funny, incredible, funny, interesting, and funny things. And they’re funny, too.

But recently one of them has done a few things that are amazing enough that I had to comment on it. I’m well aware that she hates being publicly acknowledged (she takes after her Mother, rather than me) so she’s going to hate this. And you know what? Tough.

Let’s start with her committment to education. Back in the summer before eighth grade, she decided that public school wasn’t for her. That’s not unique. Lots of 13 year olds say “school sucks”. But her reasons were a little different: It went too slow, the kids were (generally) too unfocused, and she felt she could get more done on her own. So, for her eight-grade year she basically home-schooled herself, with light supervision from my wife and I. And it worked. She rocked through the year, usually in less hours per day or week than school would have taken. She was happier, she had learned more, and everyone was happy.

While she stopped homeschooling and moved to an online school the following year, she still enjoyed a level of freedom and control over her schoolwork that few kids get to experience.

The net result? Here at the end of her 11th grade year she has exactly one credit left in order to complete High School. And that’s after taking just one credit this year. She had all but completed high school in 2 years.

She held back those two credits because my state has a “Post Secondary Education Option”, which means it will pay the tuition if high school kids go to college before graduation. So this year, at the tender age of 16, she also enrolled in her first set of freshman courses at a local college and took more-or-less a full load of classes. By the time she graduations from high school, she will have 2 years of college also under her belt.

Honestly though, lots of kids do that around here – dozens, if not hundreds. If that was all, I’d be proud but I wouldn’t be writing this post.

During this past year my daughter also got a job, at a local bakery. She liked to bake at home, and thought she could leverage that interest into work that didn’t make her want to gouge her eyes out with a happy meal toy. It was a good job, and my daughter learned a lot and had a good time in the process.

Then the owner of the bakery went in for a routine medical exam and the doctor found a lump near his kidneys. Suddenly he was looking at surgery and several weeks of recovery, and nobody at the bakery to cover during that time.

So, for the next two weeks, my daughter went in to work at 4:00am to learn all the recipes. And for 2 months after that, she was the baker. She didn’t run the store – there were other adults that handled the books and billing and such. And the rest of the back-room staff were still there to do their jobs. But every morning it was my daughter who came in and lit the ovens, maintained the inventory, mixed the ingredients, rolled out the cakes and breads and cookies, tested and approved the results before it moved to the front to be sold.

For two and a half months, she ran herself from 4am until 9pm, working the bakery, catching classes at college, going back to the bakery, then coming home to study and write reports and attend her one high school class. All so a man she had only recently met would have a business to come back to after he recovered from having a 2.25lb cancerous mass removed from his back.

During that time, if you asked her about it, you’d get her trademark shrug, a “whatever”, and then she would tell you how the cake decorator threw flour at her this morning in retaliation for the prank she pulled on him the day before.

Then came the crash. The bakery stayed open late into the night one weekend, and everyone was on hand to deal with the anticipated flood of customers. What they didn’t anticipate was a car coming through the front of the store. Nobody was hurt, and in the end the store didn’t even lose a single cupcake. But having a 2002 Mercury come through a plate glass window can be unsettling, to say the least. Some of the staff was so shaken up they had to go home. But, according to the adults who were there, my daughter was unflappable. She moved between tasks – pulling bread out of the oven before it overcooked; moving product away from the broken glass; finding boxes and buckets for the cleanup; ringing up sales for customers who were undaunted by the damage and still wanted their two loaves of rye, sliced if possible.

For a grizzled old jaded adult, there are things to be learned in all of this.

Working harder today does not always mean you earned the punishment of having to work harder again tomorrow. Sometimes it means you get to do what you want tomorrow.
If your values say “yes”, it should always trump your fear saying “no”
Don’t underestimate yourself, don’t overestimate the challenge, and don’t overthink the situation.

Yesterday, she reached another milestone. At 17 she took her last final, and simultaneously completed her junior year of high school and her freshman year of college. I don’t think she’s going to pull straight-A’s this year. I believe there will be a “B” or two in the mix.

In this case, I’m not inclined to sweat the small stuff.

My friend Doug over at asknice.com sent me this link:
http://www.gapingvoid.com/Moveable_Type/archives/000932.html

I went out and bought the book. It’s a good read. Not exactly life changing, but potentially habit changing or perception affirming, depending on how you do/look at things to begin with.

Here’s one of the key ideas that I thought was relevant to “IT people”:

Companies that squelch creativity can no longer compete with companies that champion creativity.

Nor can you bully a subordinate into becoming a genius.

Since the modern, scientifically-conceived corporation was invented in the early half of the Twentieth Century, creativity has been sacrificed in favor of forwarding the interests of the “Team Player”.

Fair enough. There was more money in doing it that way; that’s why they did it.

There’s only one problem. Team Players are not very good at creating value on their own. They are not autonomous; they need a team in order to exist.

So now corporations are awash with non-autonomous thinkers.

“I don’t know. What do you think?”
“I don’t know. What do you think?”
“I don’t know. What do you think?”
“I don’t know. What do you think?”
“I don’t know. What do you think?”
“I don’t know. What do you think?”

And so on.

Creating an economically viable entity where lack of original thought is handsomely rewarded creates a rich, fertile environment for parasites to breed. And that’s exactly what’s been happening. So now we have millions upon millions of human tapeworms thriving in the Western World, making love to their Powerpoint presentations, feasting on the creativity of others.

What happens to an ecology, when the parasite level reaches critical mass?

The ecology dies.

If you’re creative, if you can think independently, if you can articulate passion, if you can override the fear of being wrong, then your company needs you now more than it ever did. And now your company can no longer afford to pretend that isn’t the case.

So dust off your horn and start tooting it. Exactly.

However if you’re not particularly creative, then you’re in real trouble. And there’s no buzzword or “new paradigm” that can help you. They may not have mentioned this in business school, but… people like watching dinosaurs die.