O can you TEL me how to get…

Someone (I don’t remember who. If it was you, feel free to take credit in the comments) once told me,

“If you want to break a habit, don’t tell anyone. If you want to build a habit or achieve a goal, tell everyone.”

So here we are.

After more than  25 years of using monitoring tools – using everything from Tivoli, Openview, Nagios, and janky Perl scripts (which is kind of another way of saying “Tivoli”); to SolarWinds, New Relic, Kentik, and Grafana – I’ve decided to forego the toes-dipping and just dive headfirst into the world of OpenTelemetry (which I’m abbreviating to just “OTel” for the rest of this post).

I’ve bought a couple of really great ) books* and I’m working my way through Getting Started with OpenTelemetry from the Linux Foundation. All of this with the goal of adding the shiny new OTCA certification to the alphabet soup of certs I’ve gathered over 3 decades in tech.

An obvious (if somewhat simplistic) question may be “why?”. The answers are (to me at least) equally obvious:

• OTel is not only new (or at least new-ish. 2019 wasn’t THAT long ago) and cool, it’s proven not only that it has staying power, but also utility.
• Charity Majors and Honeycomb said it best: “The Three Pillars of Observability are Metrics, Logs, Traces, and DRAMA”, and OTel has survived A LOT of that drama.
• Slinging SNMP is great and all, but it’s not the whole story.

Bombastic hyperbole aside, there’s another very real reason I want to dig into OTel: vendor lock-in is exhausting.

Obviously, it’s exhausting for customers, who have to do the heavy lifting of implementing a new observability tool and all that implies. It’s exhausting in those first few months, separating engineering fact from sales fiction, and stumbling through the process of applying in ways that both replace or enhance the previous solution, and also serve the business needs. And even after that, there’s the exhausting toll of constantly questioning the choice – whether it was really worth the functionality you got versus the missing features you needed, wondering if it’s too late to change, or if that feeling is really just the sunk cost fallacy at play.

But, believe it or not, it’s also exhausting for the vendors. Nobody will say it out loud (at least not where C-level executives can hear), but no solution can do it all. Not only is there not a “one-size-fits-all”, there’s not even a “one-size-fits-MOST”! So vendors have to dance around indelicate truth. Some hand-wave away any gaps (“Most customers come to realize they don’t really need that.”), some kick it down the road (“Let’s get the main installation out of the way, then we can revisit this.”), and the worst ones flat-out gaslight (“That wasn’t part of our original conversation.”)

Of course, good observability vendors – and there are a few – avoid those tactics and are honest about what they do and don’t cover. But even that can be exhausting, because even as you feel good about being up-front and honest, you also have to cope with the existential dread of a prospect walking away when your solution doesn’t measure up to their (often impossible) laundry list of desired features.

Into this fraught environment, in the misty distant bygone era of 2019, arrived OTel. It came with a completely different paradigm: Take whatever data you have – be it metrics, logs, or traces; shoot it through a home-grown collector; and send it anywhere you need – whether an equally home-grown database, or one vendor, or three. And do it all for free**.

Even as I noted that the technology itself was still in its early stages, I recognized an ethos I can respect, a mission I can get behind, and a story I’d enjoy telling.

Since 2019, both OTel and I have been growing – in OTel’s case it has been both a matter of adoption and the natural maturation that any open source project goes through. In my case, it was building familiarity and skills beyond my sysadmin and network engineer roots – going from macro maven and script kiddie to… if not a full fledged developer, then at least a coder of modest skill. Because OTel is first and foremost a tool for developers to understand how their application is performing; and second how their app infrastructure (meaning containers, microservices, and the cloud) is holding up. “Real” infrastructure – the things that live in layers 1-4 of the OSI model are a distant third in terms of OTel priority.

Therefore I had to raise my awareness through those layers in equal measure, finally wrapping my head around spans and traces and the instrumentation that enable them.

Just to be clear, I’m not abandoning my humble beginnings. You’re going to read about my attempts (quixotic though they might be) to get good old SNMP, WMI, and other “traditional” hardware metrics into the OTel pipeline. You might even see me try to instrument a janky Perl script or two. Because an important part of growing your skills is finding ways to integrate them with what you already know.

That’s where I’m at now, and why I’m starting this series. It’s a way of both documenting my journey and leaving breadcrumbs for others to follow. It’s not that I think I’m blazing a trail that is particularly new or novel. It’s that for people with my background – who haven’t been programming since they were nine; or who remember when IPv6 was both new and critically necessary (and why); or whose experience with monitoring involves 3 janky Perl scripts in a trenchcoat – for all those folks, the current world of modern observability can seem opaque, if not inscrutable.

I hope you’ll join me on this journey.

  * shout out to Ted Young and Austin Parker for “Learning OpenTelemetry”
    and Steve Flanders for “Mastering OpenTelemetry and Observability”

  ** Well, except the last part. The folks at Grafana, Honeycomb, Data Dog, and the rest all gotta eat.