Observability Standards for Nodes - James Prestwich - Signet

Transcript

Hello. Hello. Ah, there's my slides. Uh, okay. So, I've been really enjoying learning about performance improvements, about EHRPC and reliability, and I'm here to ask you to kind of like stop working on features for a second and focus on something that is really important outside of crypto that hasn't really made it into the crypto ecosystem yet, which is observability standards. So, uh, I'm James. I've been working on Ethereum as long as Ethereum's been around. Uh, some of you have met me, some of you haven't. I've taken a few years off core dev stuff. I've been spicy in calls in the past. Uh, right

now I work on roll-up tech and deep integrations with the ref node. Uh, so observability is not really well understood. Uh, everybody has logging like, uh, I took these out of like our Kubernetes cluster. This is stuff we're logging out. Some of it's Ref stuff, some of it's our stuff. You print something to console, you shove it into a collector, you ship it off to the cloud, and you never look at it again. Right? Logs are unstructured data about program execution. Uh, they put a specific point in the program life cycle. They serialize some data. they print it out to console. Uh sometimes

these are useful, but it's like the uh elevated version of print line debugging. You know, you're putting something out of the console and expecting a human to look at it. Um tracing, which is most of what I want to talk about, is structured data about program execution. So whereas a log or an event is a specific point in time, a trace covers some logical section of the program, not some section of code, some section of the purpose of the program. So a trace might cover a single RPC request. This is a unit of work for the server is a request. The trace should cover everything that happens between receiving the request and sending the

response. And we should do this hierarchically. So we have one outside span for the request itself and then inner spans for any significant work done on that request. Right? So a trace takes the idea of logging which is event driven like a single point in time and turns it into a hierarchical tree of regions of time. And each span in your tree should have information about that span, about what happened, about what the outcome was, about how long it took, how long the CPU is running or not running, etc., etc.

Uh, so tracing is now well standardized um outside our ecosystem. Um, and then metrics, we have pretty good metrics in G, in refs. It's structured application data. It's not telling you what the program's doing. It's telling you about the health of the application. It's telling you how many requests were processed, how many requests errored. So, as opposed to events, which are like this specific thing happened at this moment in time, and traces, which are an image of the logical, you know, purposeful execution of the program. Metrics are telling you something about the health or the current state of the program.

And you know, everyone tends to have logging. Everyone tends to have metrics. Traces are a little spotty. Uh there's some in Lighthouse, there's some in ref. Uh Netherm has them turned off at the moment in the compiler. Um I haven't looked at the CL's in depth, but uh you know, I know Lighthouse does an okay job but observability is not just like one of these things. It's a combination of all of them. and it's putting them in a place where we can interact with them and learn from them. Logs in the console not very useful. Metrics in a dashboard a little bit better. But metrics are better when we can, you know, have alerts. Like that's why we make them. We

don't make a metric so I know how many JSON RPC requests we processed or how many blocks we processed. I make a metric so I can say when the JSON RPC requests stop, shoot me an email, uh, ping my phone, whatever. Uh so there are standards for this called open telemetry that are used by every major enterprise these days. Open telemetry is a uh conceptual pipeline for how we produce observability data events spans traces metrics and how we collect and export them to outside tooling. Open telemetry includes the structure of those things as well as the serialization format for

exporting them and uh standards for how you process them, what you do with them. Open telemetry is supported by many many outside services. Uh we like to use ground cover right now. Um, the goal of open telemetry is to take all that information your program is generating, not just on your laptop, not just on your one server, but across all of your servers and across the connections between your servers. Uh, open telemetry provides standards for how two different pieces of software communicating with each other pass span data back and forth. So hypothetically, if you had some sort of blockchain node that was composed of two different binaries that

communicated over some API, you could pass your trace over that API alongside the actual data that you're communicating. And that way when you ship your trace out, it includes information not just from your ref node, but also from your teu node or your lighthouse or your prism or whatever. Um there are standards for this that we are not yet implementing and the important part of this is open telemetry protocol. This is how the traces are serialized or the metrics are serialized and exported to a collector.

Uh one of the great things about this is because this is implemented by a bunch of big enterprises all the libraries exist in all the languages we use and are pretty darn reliable. So if you want to add OTLP to your existing tracing setup in Rust, takes about 15 minutes. Uh it's not hard to do this. It's just that we as a community have not prioritized it yet. So when you have this uh there are open telemetry semantic conventions for what information your trace should contain.

Um this is actually real trace data from software I wrote. Um this is from AJ which is a JSON RPC server in Rust. Uh I used AJ specifically because it implements the open telemetry semantic conventions for JSON RPC. They've already defined what information you should externalize from your JSON RPC server. Right? They've defined what the event should be. They've defined what the span should be, when they should start and end, and what information should be attached to them. So, you can see here I've circled uh RPC message type, RPC message uncompressed sized, the name of the event, RPC message.

These are just really simple standards for what a JSON RPC server should do in order to be observable. And these are broad industry standards used in JSON RPC servers that aren't in Ethereum nodes, which is kind of a weird concept. I didn't think anyone else used this, but you know, whatever. So, semantic conventions are specific ways of using observability to standardize across different applications with the same components. All JSON RPC servers should implement the JSON RPC semantic conventions. They should have one span per request. they should emit a received and a sent event

at the beginning and end of request processing. Uh, and if we do that, we gain access to tool a tooling suite that allows us to inspect JSON RPC processing uh, in a way that's standard with the rest of the industry and in a way that is well supported by tools like ground cover, Jerger, etc. These are shortcuts for direct integration with the developer and observability tooling that we want to use. So by not implementing these in our JSON RPC server which might have been written by parody 10 years ago and be kind of a mess like we are missing out on observability for our JSON RPC processes. We don't have good tooling for observing this.

So, I've mentioned the current state a little bit. Uh, you can probably tell I have some opinions about it, but it's in a word inconsistent. Different clients have all done different things. There is inconsistent adoption of uh metrics. The metric standardization effort turned into a bit of a mess. If I recall correctly, everyone logs out slightly different info at slightly different points in the timeline. and spanning like actual structured traces are relatively uncommon and are not at all standardized.

Uh a lot of people tend to use spans as essentially extra info on the logs rather than as a structured tree of execution info. So there's some really lowhanging fruit for us right now. The first is like let's actually use OTLP. Let's take any trace information we're already generating and make it easy to export with standardized tooling. These libraries already exist and again in Rust at least it takes about 15 minutes to implement if you know what you're doing.

Uh learning what you're doing is the hard part. Like learning is always the hard part. Uh this is by the way an actual uh RPC request um from one of our production services. Uh it breaks down a hit on an Axom endpoint in Rust and you can see that most of the time is spent in this you know AWS server uh AWS library kms.sign sign. And so because the AWS library in Rust already spans everything correctly and well, we get just by using their library free observability of everything that AWS lib is doing under the hood. Uh

and this we ship out to ground cover. We can go and inspect. We can see when AWS is failing and when that causes unreliability in our code. It is trivial to tell when the components break because quality components use traces, create spans in a standard format that we can export. And the other lowhanging fruit I think is I already talked about like let's comply with semantic conventions. These conventions exist so that tooling can be reused across many many different organizations.

It is easy to comply with JSON RPC semantic conventions. And this one is near and dear to my heart. Uh there is a standard uh for propagating trace over HTTP requests using the trace parent header. Uh in this way your HTTP client can communicate to the server what the trace context is and can propagate that span you know to the server. You can include both the client side execution and the server side execution in the same trace.

This is why it's called distributed tracing. These traces can pass through many different pieces of software, client side, server side, different web services and we can collect and export them into one place and observe how work flows through the system. So for this, the thing I think is the most lowhanging fruit is standardizing the connection between the CL and the EL in the engine API. The CL and the EL should have a standard trace format with semantic conventions and we should ensure that every request and every response over the engine API

propagates tracing information. The engine API is speced so that all clients must perform all actions in exactly the same way. And if we propagate spans, propagate trace information, we can ensure with automated tooling that all clients are doing that and we can flag and create alerts for any deviation from that process. Right now we do a lot of kind of fud fuzz testing like fuzzing around with it. uh we try to ensure that clients are doing it the same way but with traces we can ensure they are doing it the same way and we can alert when they're not.

So this is where we get into like opportunity for bigger wins. People are deploying Ethereum nodes in production. I know that's surprising. uh enterprises are starting to run Ethereum nodes and observability like this is table stakes for running at a big enterprise. If we don't add this to the client, other people will do it for us in ways we don't control and don't have insight into. The benefits will be privatized. So the big wins are like semantic conventions for Ethereum.

uh what should the engine API externalize? This isn't actually a proposal. This is saying what kinds of things we should be thinking about when the engine API is operating. We should be externalizing trace information about the payload version, about the size of the payload, about the status, about the clients involved etc. etc. uh and we should be doing this in all clients as part of the engine API specification.

Right? The benefit of a semantic convention, the benefit of tracing conventions is that we can observe the same behavior in different software using the same stack. Uh and so by standardizing trace context in the engine API, we can observe that all combinations of CL and E work the same way and externalize the same information about their you know inner processes. And again in case anyone's wondering or worried tracing is a logical section of program execution. It is about the request and the response. It is not about the specific architecture or the

specific code that runs in the middle. We don't need the code structure to be the same because the engine API is structured by design. We can treat access to it as a black we can treat the code as a black box and just focus on the semantic meaning of the engine API and externalize information about how clients handle that. Uh the other thing I think would be kind of interesting is like instrumenting the block validation loop. Uh this is not good for performance. Uh tracing is cheap like it's a few atomics and a little bit of memory allocation if you want strings but uh tracing is pretty cheap but it's not you know hot path

consensus cheap. Uh but for nodes that are not in the hot path consensus for no you know the vast majority of nodes that are not building a block it's great to know why block validation failed or what the output of the 15th transaction in the block was and to be able to export that information to standardized collection. So you might want to make a trace that triggers alerts when you receive invalid blocks. uh you might want to go through your ground cover and search for the entire history of past traces that have resulted in invalid blocks and why they did that. And if clients have standards for how they report that information,

then we can do that very easily. And uh yeah, so this is like what uh this is a tool called hotel desktop viewer for open telemetry. You use a collector, it ships it off to some, you know, thing, anything. Uh, could be Jerger, could be ground cover. This is a local collector. This you just run on your command line and you can see traces locally. You can explore them. Uh, this is, you know, what block validation could be able to look like is you just see each TX and you can click in and see what happened with that TX and why.

So, that's kind of all of my talk. I know I'm asking for work that isn't performance, that isn't feature work, and isn't a network upgrade. Uh, but this is important work for deploying Ethereum in production for real financial systems. Enterprises do not want software that they cannot observe and they will not deploy the software until they can't observe it. >> Excellent. Thank you, James.

[Applause] There's there's actually quite a lot of questions that are there, but I think we're only going to be able to sneak in maybe one or two just given the time here, but the top one asks the very simply where's the EIP or the ERC? >> Uh, I didn't know that every uh client PR needed an EIP, so I can go write those. Well, let me let me ask a follow along to it and uh on the similar state, it's the last question here. It says, "If it only takes 15 minutes to add traces to R E and using open telemetry, have you considered just simply submitting a PR for it?"

>> So, uh let me actually talk about that for the remainder of the time. Uh it takes 15 minutes to add open telemetry exporting if you have an existing tracing setup. So in Russ there is a tracing facade library that is uh extremely common throughout the ecosystem. Everybody uses it and there is a simple integration with OTLP where you just point it to your endpoint and go. It's low configuration. You set three end vars uh and then it's about 15 lines of code to integrate. Um, however, when I wrote that PR for ref to add OTLP exporting, uh, many months ago, uh, we discovered that the OTLP exporter

must be instantiated within a Tokyo runtime while Reth instantiated their metrics collections before the Tokyo runtime. And so, Reth needed to do a bunch of rejiggering of their node boot up in order to make this work. Uh, so you know, I'm not as good at C or at Go as I am at Rust. So I haven't yet written those PRs for other clients, but you know, maybe I'll get there. >> Excellent. Another round of applause for James here. [Applause]