Dark Forest, Public Cloud: Building for speed and scale - Flashbots x Google Cloud Web3.

Transcript

[Music] [Music] Okay, GM. Uh, hey everybody. I'm Devon Mitchum from Google Web 3. I'm Kristoff from Flashbots and this is joined work also with a bunch of Flashbots mates that you can find there. Yeah, thanks for joining us on the session on this hot day here. Uh yeah, we have a this is a a big team effort, a lot of names on here, a lot of folks to look into and follow. Uh we're going to dive into um the dark forest and public cloud and we have three parts in this

session. Uh part one uh actually we're going to jam all this in 20 minutes. So after 20 minutes you're going to come out expert in uh physical networking and everything you need to know about uh uh Ethereum networking. Uh so the first part will cover the physical flow of bits uh across the internet. Uh part two we'll talk about how those bits go from various internet locations into the uh Ethereum supply chain. And then third we have a couple callouts for uh areas for further research and some recommendations. And then for those that are into um data science or open source um uh components, we have a a data notebook that we created that you can use to uh look into the numbers behind us and uh and hack and remix some of the data that we uh came up with.

Uh so let's talk about the physical layers. So um this is how the internet uh is basically built. Here's this how the tubes are built. Uh you have the local last mile on the left. These are typically aerial fiber cables, copper lines, um high voltage power lines where the top line is typically aerial fiber and then internet exchanges that are in most u most cities. And then on the right you have uh uh effectively how these bits then travel from the various cities uh into the ocean uh across continents. And uh if you zoom out uh here's a map on the left of the submarine cable systems uh that span the globe right and you can see these are all physical cables dropped into the ocean that follow certain paths uh navigable paths and on the right these are the virtual

networks built on top of the physical networks and in today's talk we're going to talk about how latency on the right and geography on the left are actually not really one and the same and there's some important nuances you should understand as you're building apps as you're designing mechanisms or you're building and solving for users. Uh so when it comes to geography right so um again internet path might not be the best in uh physical path and um the one thing to understand is that ISPs are actually very large local area networks okay and you have to understand the relationships that ISPs might have between one another. Um, some ISPs might be smaller, might be regional ISPs.

Other ISPs might be transcontinental, right? Where they have deep networks in a major tier one, tier two, tier three cities. And so, it's important to understand the business relationships between ISPs because that often determines the path. Uh, if you're interested in this topic even further, you can look up um as uh autonomous systems, BGP in terms of routing. Um but again what's most important here is uh the the paths your packet take really depends on uh this connectivity which might not be uh fully uh understood. Uh now what does it mean for web 3? So on the right um it's often that you see uh this this case called tromboning where you might have packets that you you think might go one direction but in fact they go another direction and we have

some data here to share uh in terms of what that looks like. Um, one other thing to note, um, for folks that might trade on centralized venues and also trade on decentralized venues, it's also important to note that, uh, location matters. And, uh, Kristoff will get into some of the details. Um, quick shout out. So, so, one example in terms of how um, at least we at Google Web 3 have some experience in terms of uh, both the web 2 uh, form of internet routing and web 3 is we actually operate a faucet. Um, you can go to this page, you can scan it, you can get a drip for Ethereum test net, um, and and others. Um, but generally the way this works is when you scan this and you get a drip from Google into your wallet, a couple things happen on the

left. I'm not going to read all these, but uh, in short, uh, your request goes over some uh, radio towers. It goes into some back hole metro fiber. Uh, it'll bounce around your ISP. Depending on your ISP's relationships, it'll bounce to other ISPs. uh it goes onto the ocean or it goes over the mountain ranges and eventually lands on Google Cloud. And then from Google Cloud, we dump that into a test net from one of the uh nodes we might have running on Ethereum test net land that in a test net minestnet validation then it gets gossiped out and then all the way back uh in terms of whatever node your wallet's reading from to see that that actual drip came in. So there's a lot happening, right? Um things in white are the physical layer.

Things in uh uh gold here are the uh Ethereum test net layer. Now um here's a study uh all the raw data is available on the top right. If you want to scan this um this is some data that we took out from um uh this thing called the perf kit that Google developed where we have uh VMs in every single Google data center constantly running pings between different uh locations, right? And we built this map from Tokyo to understand okay from Tokyo how long does it take to get to different regions and if you know there's an interesting trend here um when it comes to connectivity Tokyo to Virginia is much faster than Tokyo to Europe. It's interesting and this

matters right in terms of uh maybe your design and it's important to know right um when when it comes to that decision if you even dig into the data deeper it actually turns out from Tokyo it's actually faster to get to the US west coast and US east coast than it is to get to Europe Frankfurt and Amsterdam. Uh now why is that? You have to follow the tubes. Okay, it all depends on again those maps that I showed you before and uh the routing. So putting it all together on the left um these are the tubes in terms of why it matters in terms of uh your connectivity and then how it lands into um the RPC and MIMPool. Uh it it's it's all somewhat related here.

Um we have further data here on the talk if anyone's interested from a data science perspective. We have a lot of this data on um Google's BigQuery public data sets. We have a notebook available that's up there. Um we also have some feedback and some uh um ideas to share below. But with that I'll hand over. Awesome. Um right another plug. So, mele dumpster which is um quite comprehensive data set of public transactions in Ethereum is you can now also query on bitquery instead of uh what else you do dune or whatever you can also work on it with um Google products and I think it's a nice sort of avenue for for research to think

about this connection between um transaction data and latency data. And I give you some ideas. Most of that is actually due to research from other flashboards people. Shout out to data always to to bark and so on. Um and that should give you some idea how latency matters in many different ways and how geography matters in many way different ways in Ethereum. So this is a chart that probably many of you are familiar with. Um it's the rise of private order flow, right? So um it is true that if you don't have access to certain kind of orderflow which is sort of exclusive to to to block builders then you cannot build that uh profitable blocks right

that is a vector of centralization. So vector of centralization that is quite visible if you compare that to people who build their blocks locally right basically on public data. So um if you have access to this kind of private order flow, you can pack your uh blocks with much more much more valuable transactions and you can generate more um more profit from that. Okay. And there's a clear um trend in the in the data and all of that has to do to a certain degree also with connection connectivity.

Another touch point where sort of latency matters is how fast your transactions are included in in in the blockchain and that depends on sort of which nodes see it and which nodes distribute it. For example, blocks route um has has this distribution networks of of um different so they have a relay but also they have proxies um where they quite quickly uh distribute um transactions that they see over the network. So if you distribute a transaction over blocks route um it will get on average and also in the long tail um much faster in into sort of the P2P network than if you I

cost it for my note in Zurich at my home right and it would be even worse if my home wouldn't be in Zurich but in I don't know where Iris okay so there are differences in the notes notes are different right we treat them as the same from from the point of view of the protocol but not all nodes are the same. Here are some notes. Um this is data from a robust incentive group. Um sort of gives you an idea of where Ethereum validator nodes are geographically located. This is question mark. I don't know whether that is sort of broadly distributed over the world.

Certainly there is like a a majority of the these nodes in North America and Europe definitely and there are nodes lacking for sure in in places like Africa and in South America and and India right if you compare sort of population density with node distribution and that has to do with infrastructure that has to do with latency if you look at the layer where blocks are built I don't know whether you know that but basically ally relays are in two places right they are in east coast America and they in Europe th those places are the places where uh blocks are actually produced right in some sense because builders colloccate

with it tend to collocate with these place uh with these relays right um so the network is in some sense everybody sends a transaction to these two red dots right why these two red dots and that has to do with the entire infra So those two red dots are connected by very fast connection through through this um transatlantic fiber corridor where you basically have very fast connection between uh Europe and North America and you wouldn't have that if I don't know you run your relay in I don't know Nigeria or if you run your relay in Buhen Cyrus takes longer. Okay. So there's there's this correlation between

um infrastructure and where our infrastructure for Ethereum that we use is located sort of it's depend on history right history is influencing uh infrastructure infrastructure is public infrastructure that we're using public or private infrastructure that we're using and that shapes Ethereum even though we maybe wanted to build something completely different okay another way in which latency enters that's again beautiful data by data always shout um is in this bidding data on the MEV boost relay. So if you compare over time how frequently bits in the MEV boost relay are updated, ME boost um ME boost is the market in which blocks are um um

proposed to to Ethereum. So um we bit on who produces the highest value uh block and that is an as you compare these curves over time bits are updated more and more frequently. What does it mean? It must mean that people have the infrastructure to update those bits more frequently. Right? So they must sit close to the auctioneer the relay and um there is is a vector of centralization. It is a vector of centralization. I not only claim that it's visible in the data, right? Until January of this year, basically the market was a dual poly almost by two big builders being colllocated with these relays. Okay, so those are facts that are sort of shaped

in some sense by infrastructure even though we wanted to build maybe something different. So that's is data on um on on on bidding in in me boost and on um um distribution of relays. Another fun thing that you can do that's um a project that our internet flash um Sean is currently pursuing, you can take this latency data that Google cloud provided us and you can run simulations in how certain kind of games play out in Ethereum block production infrastructure. For example, timing games. I know don't know whether you've heard about them. uh people trying to

delay propagation of blocks to build more valuable blocks. You can wonder how that extra value is shaping sort of the location of um nodes in the network. And they built this nice simulation tool. I don't think it matches reality yet. But you have this sort of nice way of feeding in the data latency data and you see the convergence of nodes um clustering right because there is a benefit of being close to certain kind of nodes. There's a benefit of being close to the relay um and there is a benefit of being close to each other.

Okay, that's one way of how you can use this data set. um other questions that you can ask I just give you a bunch I it's not comprehensive we have um plug another plug we have a research database and flashbots with some of problems that we're interested in and we look for people collaborating with us on these questions um so some of the high level questions that I I'm interested in or would find interesting to hear about research that other people are doing is this entire you know influence of location or connectivity on what different people do. So what does it mean for example for users right? So you

cannot be a high frequency searcher if you're located I don't know in can right uh doesn't work for you you have to be colllocated with uh with with a relay right but it also matters for sort of other kind of use cases right so your user experience is shaped by by where you are located in the world by your connectivity your sort of experience user experience as a validator I don't know whether we call them a user valid data experience looks different in different regions of the world. How easy it is to run in valid data. Um how much profit you make from running a valid valid data right up time and these kind of things. You miss maybe um at a stations and these kind of

things. Um we could wonder okay what happens if we change the system there's discourse around reducing slot time. I mean there has benefit from that right reducing slot times but it affects people asymmetrically right it's doesn't affect everybody in the same way and you can think about these questions Ethereum is not an island it interacts with L2s right which also servers and you're wondering okay where are the so located usually also Europe or North America or you have perex centralized exchanges in usually in Europe or um more of them in Asia Binance in Tokyo and so on right um I can wonder How does this infra structure interact with Ethereum and why does a network structure looks like it?

You can do event studies. So another idea that I had okay you could wonder okay maybe you've heard that some or you've discovered in the data that some rollup has moved the server somewhere else and then you you know you can figure out what does latency do to what people are doing. Um you can do also more like um research of the practical research of the kind. Okay. Uh how hard or how easy is it to run critical infrastructure within Ethereum in different places in the world? We have very little knowledge about that. And finally the bigger question is okay how can we make location matter less? All of those they're probably more ideas, right? But

all of this you can answer with a mix of data and probably good ideas. And I wanted to to encourage you to look into that. Final slide and maybe most important slide. Maybe the question zero that you should ask before you ask all of these questions is actually what do we want? Um so I've redrawn the previous validator relay map in in some way. I added I just just for fun a relay in Buenosirus, right? We will all be there in I think November. So we should have a relay there. I removed 40% of the nodes in North America and Europe. I don't know whether you have noticed there were too many. So you can remove 40% without you know changing this picture. I added

a bunch of in Africa and so on. I don't know whether that's a map that we want to have. But we should ask the question okay what kind of map do we want to have? And sort of it's shaped by infrastructure as we have learned. But um history is is not everything but there are also you know uh memes and ideas that we want to follow right. We we might we want to have an idea of how this map should look like. We should have a vision of how this map should look like and if we can build it with current infrastructure with Google and whatever with uh internet service providers good if we can hack that but if we cannot do that then we should find a different way because we don't want to replicate the same kind of map that we

have right that's it any final words yeah so if you found this topic uh interesting uh definitely reach out again we have a notion page on the top right you can hack our uh data on the bottom right uh we're around. Uh if you want some Google uh slushies in the back, uh uh happy to grab a slushie as well. So, thanks for your time and uh thanks for joining us.