The Open-Source Escape Hatch: Replacing DataDog Without Losing Your Sanity (or Your Budget)

Transcript

[music] >> Morning everyone. I am Rohit MS, working as cloud developer at Cloud Modeling Group. We operate in oil and energy industry. We are based out of Calgary, Canada, but I have come from Bengaluru. So, what exactly is this about? Today, I'm going to talk about something which comes at the very end stage of software development life cycle. We collect the requirements, we design, we code, we test, we deploy, and then we maintain and observe.

Observability and performance. This is where this entire thing is concentrated about. Let's look at what exactly are we going to see in the next 10 to 12 minutes. Datadog. Datadog is one such super easy, user-friendly tool which is used for observability and performance. When I say observability and performance, everyone codes, everyone deploys, but after that, what's going to happen? How exactly is my code performing? How exactly is my infrastructure performing?

Every second we are expected to collect many things. So, when I say many things, what are those things we are going to know? And then, what exactly is the problem? Datadog is very, very user-friendly, then what exactly is the problem? That's what is going to come up next. If Datadog is really posing some kind of problems, especially with the bills, then how am I going to replace them? That's comes next. And then comes with the replacement tech. Now, we have replaced Datadog, then our replacement, how exactly it works. Is it for each and everyone? Can anybody and everybody use this? We are going to discuss that.

And then, we are going to conclude who exactly is the contender to use this replacement tech. Now, what you're seeing now is directly taken from DataDog's website and before going into each and every part, let me tell you brief Let me brief you about what exactly the DataDog is. When we deploy any kind of an infrastructure, every second we're going to retrieve three different things. I did mention things. We used to get logs, metrics, and traces. These are the things which are very very important to see how exactly my solution is currently

working in my infrastructure, right? So, now, what does DataDog do? DataDog is such a user-friendly program where you just use have to use its one small program into your code and then it's it's going to collect each and everything for you. It's going to collect metrics, logs, and traces. We're going to see how exactly these three differs by themselves. Now, what the problem with the DataDog is it's very very costly. It's just like buying a flat in Bandra.

Everyone wants it, but it's too costly. DataDog is like that. Everyone wants it, but it's too costly. Then, what's the alternative? The alternative is a set of suits which is going to do the same thing, almost the same thing as what the DataDog is doing right now. If we just collect the current metrics from the companies, every host, almost every host on which the DataDog client or DataDog plugin is expected to run, it's going to cost us like $15 to $31 a month.

And once it's integrated, irrespective of how big is your platform is, it's going to work very very seamlessly for you. And off also, we are going to deploy that agent and then you have to forget. You need not maintain it. It's going to do each and everything for you. Just deploy. Just use that agent. It's going to collect your logs, matrices, and traces. Now, coming to the big problem, it's going to be very, very costly for us. If it is costly, then do we have anyone who's going to do that for free?

Let's see who exactly is that. For example, this is how the graph goes. Assume our observability and performance bill crosses the infrastructure bill itself. If we just look at the graph where the gray line shows the cost for infrastructure. And when our infrastructure is very, very huge, if you are paying millions of dollars to AWS or Google or Microsoft just to do the infrastructure deployment, and if you are paying more than that for the DataDog to the DataDog to get these

matrices, these logs, these traces, then the cost is just going to pick up like anything. Then, what's the solution? This is the replacement tech. Everything done by DataDog can be done very seamlessly, but with more effort. DataDog says, "Install my agent. I'm going to do each and everything for you." But, these replacement technologies, these freely available software suits, are to be managed independently in order to do or in order to get what DataDog is doing for us

right now. For example, when we come to Kubernetes, if you want to look at each and every pod, if you want to collect matrices. What is a metric? Metric is a value. Let's say if I want to uh alert or send an alarm when CPU utilization peaks 90%. That 90 is a value. That is a metric. That exact thing is done by Prometheus. Prometheus is a very, very user-friendly, freely available package which can run for you.

There is no need for Datadog just to do this. No need to spend hundreds and millions of dollars just to get this. What it requires, what it demands is little human effort. And then comes Loki. What is Loki? Loki is a log aggregation tool. What do you mean by aggregation? When logs are coming from different different places, we are going to aggregate them at one fine place and then we are going to push it further to whoever it needs. Now, who is going to collect the log for us? Loki is a log aggregation tool.

Someone has to ship the logs. That thing is done by the Promtail. Once again, it's free. What does a Promtail does? It's an agent which sits anywhere you want. Forget Kubernetes. If you are just using your Mac, one single laptop, you are running an application, you want logs from it, use Promtail. You want You are just using Docker, you're not using your Kubernetes, you don't have the use case for Kubernetes, use Promtail. You are using Kubernetes, still you want logs, use Promtail.

Anywhere, everywhere, you can use Promtail without spending a single penny. And then come Aegar. What is What exactly is the problem solved Aegar? Aegar deals with traces. What do we mean by a trace? Now, assume I have an error. I have a warning. I need to know where exactly is that. So, we have a class, we have a method, we have then we have a line. Now, we need the stack trace, class A, method B, line 20. So, this is a trace. This is a tree where it exactly tells you this is where your code is going wrong.

So, this tree, this path is given by the eager. And then comes the most beautiful application you can ever see on the internet Grafana. What does Grafana do? Grafana is just there to show you all the things collected by these four different tools. Grafana shows you the details, whether it's a log, whether it is metric, whether whether it is uh trace, whatever it is. It has a place for you.

Now, coming to Prometheus, how easy it is Prometheus? Prometheus runs with something called as exporters. Exporters are readily available packages in Prometheus which is expected to run with your pods. Okay? If Prometheus is not used, then DataDog runs its own agent. Now, we are talking about replacing the DataDog, then we should have something which does the same thing as DataDog. That is your Prometheus.

And this Prometheus is only related to metrics. This is the thing which collects metrics for you. How many times your application has run? How many times it has failed? How many times has it reached 90% of the peak CPU utilization? You tell a metric, it does for you. And it follows a language called as PromQL, Prometheus query language. Very, very user-friendly. Nowadays, no one codes. We just need to know how to talk to an agent, right? So, but we need to know what do we have to talk? What do we have to question? What do we have to query?

If we have that thing in mind and if you are using the Prometheus, then we have the deal. Prometheus can collect metrics from anywhere you want. Whether it is inside the Kubernetes, outside the Kubernetes, just your own PC if you are running a web application, Prometheus does the job. There is no need to pay a single penny to the DataDog. I repeat it again and again. So, we have a proven solution for uh this and we have been using from very, very long. Now, what does Loki does? Loki does the aggregation. It does not do the aggregation just on each and every kind of a log. It does the aggregation based

upon indices, index. Instead of uh indexing each and every log, it just indexes the log streams, continuous streams of logs instead of indexing each and every line. So, how does this work? So, this comes from Promtail. So, I did mention that Loki is an aggregation tool, but someone has to supply push logs to it. This is done by the Promtail. And then we have a distributor. There can be multiple uh Loki instances also if our if our infrastructure is very, very big.

And then comes the ingestor. Now, to different, different uh databases or different, different places where we want to store, whether it is an Excel sheet, whether it is an MS Word or the very big terms like MongoDB, Cassandra, whatever it is. We are just interested in storing the data in such a way that is it is easily retrievable. That's the only thing which differentiates the Microsoft Excel with different, different kinds of databases. It's all about retrieval. It's not just storing, storing storing.

So, this is the right path for the Loki. Now, the Promtail. Just like DataDog, this is a very, very small agent, DataDog being a bigger one, it does the log shipping. You install this agent anywhere you want Kubernetes, Docker, or whether it is a single application running on your laptop. What it can do is it can collect the log. You will need to just say from where you have to collect the log. And it sits there, runs for you, gets the log, and it tails for you. What do you mean by tailing? Live logs are generated, it keeps on extracting, scraping, scooping the logs for you. And it ships to It ships to Loki.

So now, how exactly these things happen? So the Promtail works with something called as discovering stage, labeling stage, pipelining stage, and batch and push. So discover is like finding where exactly is the logs needs to be shipped from. And then, if you want relabeling So for example, if you're collecting logs from different different places, you can relabel. So this is from 48, this is from PCB, this is from some other infra. Like this. And then you can build a pipeline.

Pipeline means what? Like you have a starting point, you have a ending point. So you need to create a chain of commands of how exactly the logs has to pass through. And then comes batching and pushing. It can group for you, and then it can push to the exact Loki server wherever you want. Jaeger. One thing which every developer, coder would like to see whenever something goes wrong. Where exactly has it gone wrong?

Which is that method which is causing that problem for us? So Jaeger deals with traces. Irrespective of how big the tree is, irrespective of how big your code is. Traces with the trace ID, just like just like how the DataDog does its job, Jaeger creates such a kind of a tree for you. And for every such kind of a trace, it has it's going to associate that with a trace ID and make sure that every kind of an error or warning or whatever you're going to set can be traced to its core.

So, the metrics, logs, and the traces, the three pillars of the observability, which is given by the Data Dog, is or can be implemented using different different suits. And among that, Jaeger is the one. So, coming to span. So, span like if you if there is no problem, but still if you want to search for a timed operation. So, at 10:52, I want to check from 10:52 to 11:00 a.m. what exactly has happened, how exactly the data has flown through among which user-defined functions or readily available libraries within the classes, it does the job for you.

So, this is the beauty of Jaeger. And most importantly, my dear friends, it is free. Everything is chargeable in Data Dog. And if you use Jaeger and if you are interested in traces, which you should be being a developer or being a student of computer science, then Jaeger is something which can help you to come out of the Data Dog. And the next one is Grafana. Now, we have collected logs, matrices, and the traces. Now, how to display them? Data Dog has got its own very very super friendly user UI.

Now, Grafana also does the same thing, but the only thing what it needs is little human effort. Everything is readily available in Data Dog, but in Grafana, you need to set it up. Set up the sources. Set up the views. It has got its own readily available views. You can choose among them. Make sure you have selected the right source, right panels, and it can does the job for you.

Another important thing why people go behind Datadog is how it's going to send you alerts. It's linking with different messaging system, whether it is Slack, whether it is Teams or whatever it is. Very, very similarly, Grafana is also capable of doing that. The entire thing what Datadog does broken down into different chunks and assigned to different different suits what we have seen. So, this Grafana can be connected to your Prometheus, Loki, Jaeger, and you can create dashboards, you can explore or like even if

a readily available pane is not suiting for you, you can just create a new pane within just like no time. Within a minute, readily available panels are there, you can use them, select the source, and then you can have your own view. >> [snorts] >> So, how does a production architecture looks like? So, if we really want to come out of Datadog, then how exactly are we supposed to use that in the production? So, at the very lower level, so I this is just an example taken from the Kubernetes cluster because that's what's running everywhere right now.

So, for the Prometheus, we have got metrics exporters. And on the pods, we have got the Promtail running. And then, we have got application pods. What is OpenTelemetry, which does the same thing like it's capable of getting you the logs, traces, and matrices. Now, within this, after this Kubernetes cluster, this scraping is done by the Prometheus for the time series database. Time series, a very metric timestamp up to milliseconds. And then, the Loki gets the logs from the Promtail. And then, the application parts, which is running your code, right? That's the smallest unit

available in our Kubernetes cluster. Every uh metric is or every metric which is responsible for the traces. Traces is just like taking uh the top view of what's happening. If you decode, once again, you may have to deal with values. If you're dealing with values, you're dealing with metrics. And then, that comes with the Jaeger. That is, once again, pushed to Grafana because at the end, you want to see what's happening, right? So, to see, to visualize what we have, we have Grafana. And coming to alert manager, alert manager for alerting, we have got two different options. Even Prometheus has got its own alert manager, and even Grafana is able to alert you, whether it

is to send you uh messages over messaging apps or whether it is to email you. Now, how do we come out of Datadog after knowing little bit about or even knowing about the terminology of how exactly to replace the Datadog? Primarily, what we have to do is, we have to run the inventory. So, we have to export Datadog dashboards, and we have to rank them how exactly they have been useful. There may be many people coming into your company creating their own dashboards and leaving them obsolete.

Now, we have to eliminate them first. And then, we are not just going to discard Datadog by just one single button click, but we are going to run that parallelly with our current setup just to compare. And then, we are going to rebuild the Grafana dashboards as per the dashboards which we have been using in the Datadog. And then, we are going to take the call, and then we are going to switch the alert routing. So, now this comes up to your taking the logs and matrices and showing up on the data on the newly built Grafana. And then comes the alert managers because that's how we are going to be notified. So, that comes at the very end.

So, if you look at the pricing, so just taken as an example, so DataDog, the bar for the DataDog stays very high. And if it is self-hosted, when we say self-hosted, once again, I'm just talking about the cost for the infrastructure, but not it doesn't cover the cost for the Loki Promtail Jaeger because that's freely available. So, what can be easily achieved? And after this replacement, what still remains as a gap? When it comes to very, very easily achieved metrics and the data from the Kubernetes dashboards,

logs tracing is very, very easily achieved and with zero dollar cost. That is very, very important. And very, very far away or something which is not little realistic comes with cross signal correlation. So, tracing the links with the metrics, that's little bit difficult with the replacement tech. And out of the box APM magic, like there is nothing goes automatic in Grafana. So, for everything we need to have a think head behind.

And then comes with scaling and retention. Once you Once you have your DataDog, so it handles each and everything for you. But here, so we have to deal with upscaling and downscaling. And then comes the database dashboard migration. So, if I if we say if I have dashboard A and I want it readily converted to Grafana dashboard, it's not readily available. So, we need someone to work for that. And then there is no vendor SLA, like there is no on-call monitoring for Grafana. Like so, it's completely on us. So, if you want that also to be done by someone else, then DataDog is the thing which you should follow. But if you are ready to

put some effort and take the dirty thing to your hands, then Grafana and this replacement tech is going to help like heaven. So, taking a call. So, concluding. So, who should use it? And when it's not the right time. If you want something to be built on the very next day, Data Dog is the best. You just install, it does everything for you. You have time, you have resource to build it on your own, Grafana is the thing.

So, what does Data Dog gives you? It gives you convenience. You pay, I'll take care of you. I'll take care of your entire infrastructure. But, in Grafana, Loki, Promtail, so we have to deploy, we have to take care. We are the owners of it. So, when are we going to use this replacement tech? When the bill goes faster than your infrastructure. Your billing, you're paying $1,000 for your infrastructure, and if you're paying $1,200 to just to observe and monitor it, there is no use, right? And then, if you have a small team, and how about like data residing about yourselves? When you collect the data the logs matrices, it goes to Data Dog

servers. But, if you want the data retention, use Grafana. And when it comes to your SaaS, SaaS means the exact Data Dog what we are using. When the team is small, and you don't have the enough bandwidth to do everything on your own, Data Dog is your cup of tea. And then, you want as I said like you want readily available thing for tomorrow, Data Dog is the thing. If you are ready to put effort, replacement tech what I have shown you, which can be easily done, is the thing.

Thank you so much for allowing me to come here and present in front of you. I thank the entire team. >> [music]