ZKPodcast: Blockchain 101: Transaction lifecycle & the mempool

Transcript

[Music] welcome to zero knowledge a podcast where we explore the latest in blockchain technology and the decentralized web the show is hosted by me anna and me frederick [Music] in this episode we sit down with tommash from parity technologies to talk about what a mempool is and the life cycle of a transaction on the ethereum network [Music] hey guys hello hello hey today we're

gonna do uh another 101 episode which i'm pretty excited about this was originally a listener request and um it's a good topic it's about the quote-unquote mempool or transaction queue or transaction pool like it has a lot of names and there's a lot of confusion around this but basically we're going to talk about the the lifetime of a transaction like where from leaving your computer to being in a block what happens to it and we have a special guest with us which is probably one of the best guests we could possibly have for this topic because he wrote the transaction queue implementation in parity uh welcome tomorrow

hi there fred uh thank you for the introduction i actually wrote the transaction queue twice already once when i was starting pin party but then i figured that it's it's a good um it's a good time to do a rewrite after after some time and i and i saw how shitty the previous code was when did you do that rewrite uh so the rewrite happened um like two months ago i think oh wow it has been released with uh parity 111 yeah and i'm i'm pretty happy about it but uh now no couple of things that could be improved further it was a big big improvement i did get a couple comments from miners saying that it was performing a lot better and uh

they were getting better block prop times and stuff like that yeah but but it was like couple of changes contributed to to this it wasn't only on the transaction fully right but uh yeah it was pretty significant release actually so let's start maybe by defining some of these things uh what is a transaction queue and why is it sometimes called a mempool um yeah so so pretty much all the three um uh things you mentioned so transaction queue transaction pool and memple they are they they signify the same thing um so the mempool is is sort of like bitcoin

nomenclature um in parity we call it transaction queue mostly and i think in gaf it's uh it's called transaction pool but they all mean the same thing so if we are talking about mampo uh in context of ethereum it means that the thing that stores the transactions uh on your um on your note locally and uh propagates them to to the peers and also if you are mining it it's used to determine what what are the uh best candidates for the next block that you are actually making i guess the name mempool would come from sort of you have all these transactions floating around on the network and you

can only capture and store in memory a certain amount of them before like your memory pool is exceeded and you can't really accept any more transactions and you kind of have to prioritize this queue and not accept any more into the queue so i guess that's where the terminology comes from originally i do like that word mempool it sounds like it should be the name of a band i'm just throwing it out there in the world i like it it's a future podcast uh name or something the memphis yeah that sounds like that too cool so uh yeah we were talking about these transactions and the how they get into this transaction queue but maybe we

take a step back and and look at like the the network what is the the ethereum network exactly and like how do these computers connect together yeah um ethereum is ethereum network is um [Music] just a bunch of computers connected together um obviously we cannot connect to all of the computers running ethereum clients because like uh it there's too many of them and and no uh hardware will be able to to handle so many connections so we we always only see like um only couple of uh nodes that are nearby us and we only connect to a

couple of them so we create this kind of mesh of uh of connected computers so um if you want to send something in the network it actually has to propagate through the network because you only send it to to the peers that you know about and you hope that they will propagate it further to to their peers until finally it gets to to the recipient and there are a couple of ways how we can um connect to this network so because it's completely decentralized we don't really have a single server that you can you know go to and and everyone can connect to you actually have a couple of special peers which are called uh boot notes and

those piers uh those notes they have many peers that and they are able to kind of you know like um show you that the other part of the network so you only know this disappears as the as a first point of contact and then they tell you hey there is a bunch of other guys in that network so maybe you will try to connect to them you might get like better uh connection a lot lower latency to those other guys so we are discovering that network uh by first connecting to boot notes then connecting to other nodes and they also can give you their peers so you're discovering the entire network through this way and the second way to discover

the network is to run the discovery protocol uh which is just kind of like broadcasting hey i am an ethereum node uh is there anyone else here uh running an ethereum node and uh the other guys who can reply and and you can get connected when you mention these boot nodes is it maybe i didn't fully understand that but is that like a fixed are these like special notes like individual special notes and it's always the same it's not like it would change for different individuals boot notes are just nodes that are hard coded in the client and it's there it doesn't have to be fixed so that boot node list can be managed and can be changed by anyone you can actually override that the boot

notes when you are connecting to the network as well it's just the notes that we kind of see that are are maybe run by some companies like we parity we also run couple of boot nodes so people can connect to them but it's it doesn't mean they are required to connect to the network if that makes any sense so on a sort of basic level the network is a distributed hash table or a dht or runs on a dht and this is like similar to how bittorrent runs and how you can discover appears in bittorrent but in this sort of dht structure if you imagine the dht the distributed hash table as being

like a long list of all the peers in the network then to discover this dht and to like get the information from it you need an entry point into the network you can't so you need someone to first tell you like where do i find information here and either you can do what omash said it would just like broadcasting widely and like pinging all local ips and then trying to ping you know all public ips or search some ip space for another ethereum node that can be your entry point or you pick one of these boot nodes so geth has a set of boot nodes parity has a has another set of boot nodes we have some overlapping servers in each and

these are just servers that have been shown to be reliable and well connected in the network and so they serve as that entry point to give you the first sort of entryway into discovering the rest of the network they don't have any sort of special trusts uh it's not that you're downloading the blockchain off of these and trust you know what they're telling you or anything so so it's pretty much just to like speed up this process right because you know completely decentralized way would be just to find a peer a random peer that is connected to the network but it might take quite a long time maybe to continue so you started talking a little bit about like the beginning of

a transaction there or you've talked about like how a network first gets connected but i think it would be really cool to follow the journey of a transaction maybe you can like walk us through where it goes and how it gets to a minor in the end okay yeah so um so with the transaction so the first step to actually have your transaction included is to generate the transaction and that happens locally on your computer because every transaction in ethereum network has to be signed with your private key so obviously like a good security practice is to not never share your private key with anyone so on a on your trusted computer you are signing the transaction and then you are

submitting that transaction um perfect in a perfect situation it would be submitting to your local note that is already connected to the network and what happens is that that that signed transaction is um tries to get to the pool um so it has to go through uh first the verification pipeline so we check if the transaction is correct uh if um if there is enough balance on your account to actually send that transaction if you are paying enough if you if you declared enough gas for the transaction so if the gas limit is set

correctly the gas limit also cannot exceed the block gas limit um there is also a parameter that controls like how much you want to pay for transaction execution but when you are submitting that transaction to your local node it doesn't matter that much uh in in the in how fast it will get propagated through the network from your local node because we prioritize local transactions always so your transaction gets like a priority boost because it's it's just yours and you want to get it out first um yeah so so when it gets to the pool then we start to propagate that transactions to to other peers so what happens is that imagine that you are

connected to hundreds of peers we don't spam the network uh we don't send it to all the hundred peers at once rather we select uh some random peers from from all the all the ones that you are connected to and we propagate first to those random peers and we repeat that process every a second i think or a second and a half and then we select more random peers and send to them again we also store who did we send the transaction to and also who did we receive the transaction from so that we don't really send the same transaction twice to other peers it's it's considered like a bad behavior so

the the other end might actually penalize us for doing that so they can disconnect us if they don't like it so yeah we start at your local note we send to random peers we do do it um every second or every two seconds uh we select more random peers we send it to to our like local neighborhoods and then we hope that those peers will propagate that transaction further until in the same fashion until we get to a node that is actually running um that is that is a minor so it's it's running uh um some kind of mining pool and that a particular uh node will do

exactly the same thing so it goes through the verification pipeline and then tries to figure out what is the priority of your transaction and the priority is actually determined by the by how much they can earn from including that transaction so what is that the fee um that is paid for that transaction and they create a block if your transaction is like at the beginning of the queue okay so i have a question there you sort of you're starting your you sort of mentioned a transaction a node transaction gets priority because or sorry a local transaction gets priority in your own in your clients yeah so that's the first thing that goes out then it goes to a random number of people but a random

number of nodes how how many uh so the the precise formula is like x over square root of x if i remember correctly so it's it's just like a a percentage of what's there yeah yeah it's a percentage but it's a it's a function that like grows it's not really uh because percentage is is leaner so if you have like ten thousand peers it would be a thousand peers if we are speaking about 10 for instance but we are trying to keep this like very limited so so this function is like growing slowly so even if you have a network of and then you are connected to thousands or tens of thousands of peers you will only send to

up to i don't know like hundreds okay okay what what's the reasoning behind doing this rather than just like instantly sending to everyone uh it's just like actually i'm i'm not quite sure i think it's just like politeness and uh preventing network contention because like those other peers they are connected between each other as well um so there is a good chance that uh they gonna like gossip that transaction between themselves uh in the future and then because we are monitoring the transaction that the other peers are sending to us so we we we kind of send a bunch of transactions we we save your bandwidth so we you don't spam the the network with your

transaction because then there is a high chance that it it will get propagated anyway is it sort of to avoid duplication of work or like you just you only want like how many once you send it out to that first group of random nodes like how many do you need to pick it up to keep it going like are you looking for a certain percentage to keep pushing it or do they all push it that's i guess what i'm kind of confused on there so so every every peer in the network have their subjective view of what the member is so you don't really have like this you know like global consensus uh on on the maple itself uh so every node is able to choose like any transaction they want

to to propagate faster or to propagate slower the this kind of like um game theoretic uh agreed upon uh way of doing that is to to prioritize transactions that that are um paying the most fee to the miners and this is what other peers are also doing because they have like a good chance of including a blog but um yeah this is this is just like um the expected behavior of the peers because there is no way to prove that you actually like receive that transaction and then propagate it further so you don't you cannot really you know like penalize other peers for not following the protocol this is like completely

completely subjective i think this is all sort of within the definition of what a gossip protocol is and gossip is a very broad term of like it's not a specific defined protocol and there's been a lot of analysis done on gossip style protocols where you you know look at how fast do things spread and exactly those questions of like how many people do you need to pick it up to actually propagate to the whole net network and stuff like that i don't think there i've not seen any conclusive answers on like this is the exact number that you should have and expect them it's sort of like it it's a very um generous protocol in that you don't really need all that many

people to pick it up and propagate it to for the gossip to actually spread throughout the whole network um and yeah if you just send to 25 of people and most of those pick it up and send it to 25 of their people then you know it it has this sort of exponential increase anyway so highly likely that it'll uh propagate throughout the whole network just uh sort of to go back to that thing about prioritizing the local transactions so like if uh if a node picks up just a different transaction from a mempool and then it's also sending its own transactions it would prioritize its local transactions over that yes so so even if you even if you are

paying a smaller fee like your local node will send your transactions first okay but it doesn't mean that they propagate like faster in the network because the the other end that is receiving that transaction they don't really have that information that they are you know they are not of a higher priority for them how many like so a node is pushing out its own transactions and then propagating other transactions how many transactions maybe per second does a node deal with or maybe i don't know what what your time frame is but i'm just curious so so i think we didn't really do like benchmarks how how fast we can uh propagate transactions or how fast we

can receive transactions but i think there was a point where uh at a point in time when in ethereum main network there was like 150 000 transactions uh floating around in the network uh because like the blocks were full the the fees were increasing and there were there was a lot of transactions and uh yeah at that point in time i i was spending some time doing optimization of the transaction uh queue because we uh we saw increased cpu usage of the nodes um but yeah it was handling that at that amount correctly so so

um but i don't have like uh numbers in terms of throughput how fast we can propagate do you have even like a sense of it though i'm just trying to imagine if it's like 12 or like hundreds or like thousands like any anything so so i think we are um i would have to check the numbers but uh like if if you are creating an ethereum block right uh current gas limits allow you to i think store 200 transactions so um even if we could send more it doesn't really make sense to to propagate more transactions than for the next like two or three blocks so actually in parity currently we

um the limit of how many transactions we propagate to the network is uh um current block capacity um like multiplied by by two or by three so we try to propagate transactions for the next three blocks um but if you if you want to be selfish you can actually like you know send your transactions as fast as your like you know hardware um support sets that the only issue might be that the other peers might not be able to like keep up with uh all the transactions that you are sending to them and they may decide that they want to they don't want to be connected to you because you are spamming them with the transactions so i don't really have the numbers

but we are sending like um 600 transactions every second currently uh in parity uh ethereum on the on the mainnet because this is like how much we actually have to send but we it it would be really easy to send way more uh it would just like increase your cpu usage and time with something yeah so i mean there there's a difference here between what a node can do and what a node needs to do so i just looked up on on ether scan and there's currently sixty thousand pending transactions in the mempool and uh it's quite a lot but it's not like outrageously long like so much was saying there's

been 150 000 plus sometimes and i know we've run some benchmarks on um like not only propagating transactions but like actually including them in blocks on like poa networks in closed environments between two nodes or whatever and the the bottleneck is never the you know the propagation time like you we could probably propagate 50 000 transactions in a matter of a couple of seconds so the the slower thing is like validating these transactions um running them like including them in blocks and all this other stuff yeah actually validation is kind of two parts so one part can be

done completely in memory and it's just like cpu bound and this was never a limit um as well but the the like the heaviest part is actually running the transaction so writing something to the to the state 3 or reading something from the state tree because it's usually stored on the disk and it requires couple of disk random discretes or disk rights and it's it's really slow it's it's currently the limiting factor for uh for ethereum network i think the so the reason that i'm asking the question i first asked which was like the reason i'm just trying to imagine what the node is doing kind of at what frequency it's doing that so the node itself like it's sending local transactions it's propagating

other transactions that it's picking up and i'm just wondering is this like is it pinging at a beat of like this is it like fat like that's that's that's sort of what i'm trying to get to i'm just trying to envision how that thing itself is working yeah so so it's it's like interval based so every second we are doing quite a lot of work we are taking the the like the the best transactions that we have in a pool uh we are taking up to let's say uh a thousand transactions from the poll and we are trying to send it to this random peers but first we need to figure out or or if we send some transactions in the past to that peer as well or we maybe receive some transaction from that peer so we are doing this like

difference what has to be sent what what that peer already knows about and it happens every second and also like in between that time we're actually handling all the requests that other peers my sending might send you to us so they either like propagate transactions to us so we need to verify them include them in the pool or they might request some data from us like um block headers or block bodies of the past blocks as well so there is like quite a lot of things happening in the note every every second there so your question is is i think it's a good one because it also sort of highlights we get quite a lot of questions or

complaints about parity getting slower over time and from a normal user's point of view it's like using more disk it's using way more cpu than it used to do and it's just like it's way slower like why every release it seems to be getting worse and that's not because parity is getting worse it's because the network is getting more congested and the blockchain is getting bigger and a common misconception of like the cpu increase is actually there's now so much transaction spam there's all of these bots like trading on decentralized exchanges trying to over bid each other on gas prices and things like that so what

happens in a bidding war in between two bots for instance is they send a transaction with you know one g way as their gas price and then uh another bot sends one with one point one g way as their transaction price uh gas price the exact same transaction and so the bot number one says hey you're not going to beat me to this because this trade is still profitable if i get it through so they send another transaction at 1.2 like superseding their previous transaction so they keep doing this bidding war on like in the transaction pool and this only happens in the transaction pool it it's never shown on the blockchain that this happened they they only like the blockchain only sees

the one that was actually included at the end so phil diane uh multiple guests on the show he has uh he keeps tweeting out every once in a while like the highest gas price paid and like the last tweet from him was 17 000 g way and that's just that's a profitable trade for about to make so it's gone up like probably thousands and thousands of transactions in a trade war to get to that point in that example then are there some transactions that just die they never do anything they just sort of sit there fill it up and like do they get removed at some point yeah i think this is a good question as well because it

ties into um what happens when a transaction gets stuck and you want to like redo it at a higher gas price or something to make sure that it gets included so so in ethereum um transactions doesn't really have an expiry date so they can they can be in a member for forever pretty much if uh if like if other transactions are actually higher priority so they pay more fee and they are included first and um also um in ethereum we have something that is called nons that serves uh two purposes first purpose is to to have a replay protection for your account um so when

you are doing a transfer transaction uh you you actually want to do like only one transfer to a specific person right you don't really want anyone else to take that transaction put it in a blockchain again and do let's try first for the second time so so this is what we have nonce for it actually every every sender has this this this particular number that has to be included in the next transaction that is originating from from that particular account and also nonce uh allows you to order the transactions so first you have to process transactions with uh lower nonces and then and only if transaction web say norms five is processed you can only

then process the transaction with non six so if you send a transaction um with a very low gas price and then you try to send more transaction from the same account and even if you try to increase the gas price so you're trying to increase the priority of the next transaction transactions this one first one might actually be like blocking all of this this other transactions because it has to be included first it's what the what the protocol says but it's not included because it's like low priority and it's blocking this this higher priority transactions so what you can do is actually you you have to resubmit that same transaction so use the same noms

for the second time but specify higher gas price and this is what fred already mentioned this is how this like trading course happens so they are sending a next transaction but they are then sending an update for that particular transaction trying to over bid them themselves so like increase the gas price and then that's uh that actually like pushes out the old transaction from the pool uh we also have a ddos protection there because you could you could just increase the gas price by one way and it would it should replace the transaction but to prevent the nodes from doing too too much stuff and especially miners from actually um replacing those transactions

from the block we specified that the minimal gas price increase has to be at least like 12 percent this nonce question so when we actually we did an episode about blocks and block headers and there was a no non nonce was used there as well is this related or is that a completely different use of a nonce it's a it's a different one but so nonce is a general term it just means yeah nonsense number or but yeah in the transaction it's actually um yeah it's it's a transaction numbering scheme so it is a strictly increasing thing whereas in in the block header the nonce is what

produces the proof-of-work result that you were looking for so when a transaction a new transaction with the same nonce gets posted what happened like does the node just remove the old one from the transaction queue and forget about it or is is there some procedure happening there uh yes uh the the new transaction if it would so if it occupies let's say the same slot so the same like sender plus nonce uh slot uh we actually replace the old transaction we forget about the old transaction and only include the new one um yeah but but those transactions will will never really die i mean

for them to die you'd have to kind of force every node in the network that has ever seen that transaction to forget about that transaction and to stop propagating that but it's it's something that won't really happen for for sure i mean it's impossible so so they never died but they are just like never included and never gets to to any block so they they kind of stuck forever where do they do they live somewhere do you see them can you see them i mean it depends on the on the peers that are storing it so um because the pool is is limited so if you have enough capacity to store all those transactions that won't really be included like will never be included

then sure like from time to time you might even propagate that transaction to some to some other peers uh but yeah and we have a couple of nodes in the network for instance eaters come they are they are they are having like three large pools and they are storing all transactions that they have seen um and re-propagating all the transactions that they have seen in the in the past so they are kind of like making sure it won't die but yeah but most nodes mempools will be smaller and so it will just drop off like it just won't be included yes but it might like you know it might drop off at some point and then reappear at some other point just to be like pushed out by by some other transactions in the

future again so it might be kind of like flipping by the way is that correct does a node have its own mempool or is there a mempool that a node taps into like is it it's individual to the node okay so so every every node in the network has its own mapple and it's it's like this subjective view of uh all the transactions that they have seen ordered by the priority and and the nodes so there is no global number that everyone is using it's just like i saw those transactions they seem valid to me and they seem useful for others for like miners they they can um by including those transactions they can earn fee so i'm gonna propagate that to my to my other peers i had a question on

uh the propagation part we talked a little bit about this sort of gossip structure and propagating to a portion of them but there's there seems to be a flaw here in that you can produce this sort of infinite loop if i tell a about a transaction or let's see node a tells node b about a transaction node b tells node c about a transaction and then node c tells a then obviously a shouldn't be telling b again because you're you've entered this infinite loop how do you prevent that happening so um every every node actually stores um [Music] tries to remember what transactions has

been already sent to to to its sphere and also it stores uh what transaction it has received from from that other peer so we try to never send that transaction twice if we know that that appear already received that transaction from us and also if we know that a particular transaction has been sent by this no so we got that transaction from that note we we don't really like propagate it back because it doesn't make much sense although um randomly we also choose uh peers that we kind of forget that we propagated the transactions to them to just keep the transaction flow transactions flowing because

we we cannot really if we send the transaction to appear we don't have any guarantees that they received it and included that in their mempool because their mampo could be full and we are thinking oh yeah they should have that transaction because we sent it but they didn't have time to process that transaction or their manpower was full so they don't really propagate it further so from time to time we select random peers where we kind of re-propagate the transactions to them again so is this sort of metadata that's attached to a transaction and then when the transaction leaves the the queue this all gets removed as well or is this stored somewhere else um yeah

like technically i think it's stored on the on the node table so it's it's part of the like node metadata not really transactionally manually yeah so there you were talking about like gossip but i guess i i have a quick question about gossip like would the convert would it when you talk about sort of communication going from note a to node b would it also go backwards is there like something coming back in gossip or is it a one-way propagation yeah it's it's one way we are just like broadcasting uh what we know what we have learned like this this hot rumors to to all the peers that we have and uh it's it could go back if we go through like as fred said earlier as we if we go from

node a to b then b to c and then c to back to a because node c doesn't really know that this message originated from from a but it doesn't really change anything i mean it's not something that is that is bad for the protocol necessarily uh because like that the important thing is that that process ends ends at some point right so either that um that the message that we are spreading becomes like obsolete and we should just like all forget about it or we are keeping um we keep to propagate that's um the entire time because we just have resources to do so also a little question about the word is gossip like a

traditional computer science networking word or is that something that's more coming from this space i guess it's become that but it comes from analyzing actual gossip and like the the name comes from like how studying how gossip spreads in like offices and social networks wow but was it the question here is is it used like has it been used in other just networking things yeah it's been a networking term for a long time okay okay i wanted to touch a little bit on the security properties of this network so we obviously have this dht discovery network where we gossip

out transactions and this looks very similar to a lot of other types of networks i mentioned bittorrent and um typically these are kind of these are quite resilient networks uh it's very hard to sort of take them down but also like something that's very common in networking is having netsplits for instance where you kind of split this network into two halves and they kind of start acting on their own and they don't see what the other half is doing what is done on on the networking level to avoid net splits is anything done um so there's there's nothing done to to prevent uh the network splits uh on the network level um because that's that's actually like that

the whole point um with um with decentralized networks that uh we only see like a subset of the network we are never connected to the to all the peers we are never like connecting uh we we never like form this like fully connected graph so we only see this this small part and we hope that the network is connected at some and and even if we disconnect for a short period of time if the network split happens uh when we reconnect back uh we we will still go get into a consensus like what what is that the world what would the world should look like and and what's the state of the network after after that split

yeah i mean so a net split would essentially mean that you're creating a temporary fork until that net split is resolved and then you might have to reorg you know in varyingly long chains depending on how long the net split was but something that i'm like curious about it and if anyone has done any study of is we have a lot of the mining power in china and china has the great firewall what if they suddenly managed to split off like all western miners from all chinese miners you could potentially have a very long fork to deal with yeah but here you're talking

about the miners splitting but what if like just nodes like could nodes just go in different directions like there nothing gets written to the blockchain it's just the mempools change or like maybe the priorities in them are different like do you actually have to reorg anything from that no so what would happen if um a bunch of just normal clients net splits like say you have a small group split off for some reason all the transactions they're trying to send and trying to propagate out would be stuck in just their little group and it would be stuck in their little group and never really propagate to a minor until they can reconnect to the whole network and then they propagate all of this at once and like it would

spread to a minor at that point so the worst case scenario of a net split for like a normal person is that their transaction gets delayed which i mean could be bad but it's less bad than a long fork and the network part and the mempool part is not really um part of the consensus so we don't really need to you know have a shirt and and common view that we all agreed upon uh it's just like subjective to every note it's just as i said earlier like all the things that i have seen and think that might be useful for others but i can make my own decisions on how do i prioritize that if if i propagate the transactions further and

so on for instance in ethereum uh we have light nodes and light nodes pretty much don't have the state so they are not really able to verify the transactions entirely so what we do we they don't really propagate the transactions at all they receive the transactions data they just ignore them and they never send other transactions than your local ones to to other peers because they are not able to verify if they are if they are correct or not or maybe they they are able to verify that but it's like it would take too much resources for them to do so because they would have to pull that the verification um

the data that they need for the verification from from other peers anyway which makes sense for transactions that you really care about like your own but it doesn't make sense for all the transactions flowing through the network exactly given the design of this network why can't i just send a billion transactions and take this whole thing down um you could i mean you could send those transactions you probably couldn't take the whole thing down because first you are only connected to a limited number of peers so you're just like spamming those peers then these peers uh like the transaction is received by them and it goes to this verification pipeline and we make sure that this verification pipeline first

does all the checks that are like really cheap to do so for instance checking the structure of the transaction um the consistency the uh the minimal the base gas if it's if it's um if it's enough to um to cover the the the base gas for the for the transaction then the signature and this is actually like pretty um pretty expensive part and then we checked the balance and this in the states but anyway all those checks are are really cheap compared to actually running those transactions so if those transactions are um not valid in um in one of those checks we will reject

them really really fast so you would have to have like really huge computing power to generate thousands generate and sign 1000 transactions [Music] that would not be rejected in this like early checks in the in the verification pipeline in this verification pipeline do you also look for identical transactions like like if it was just like the exact same thing over and over and over and over again would that also be something that's found in that process that's that's actually one of the first check that we are doing so we are checking if the transaction is already in the mempo we are using hash of the transaction for this and like hashing is is quite cheap and then we are checking if that transaction is already in the in

the blockchain which is like a little bit more expensive but but still cheap and recently with the changes that we did with parity 111 we actually also have a cash for transactions that we have seen in the past but we know that are going to be rejected from the pool so we are going through this verification pipeline then we figure out okay this is an invalid transaction so we reject it but we also start the hash so that when we see it again because it's like through this gossip protocol it's we receive that somehow again we actually uh reject it like much faster it's all those checks plus the fact that even if you were to try to mount a successful attack in this

way you're only attacking your pairs so even if you manage to ddos and take down your peers so what you've taken down 25 of the thousands on the network you kind of have to move on and try to take down the next you know even if you connect to a thousand and take down a thousand then yeah you're kind of messing with people it would be annoying for those that you're sending [ __ ] to uh but you wouldn't take down the network because you would have to like move on and take down the next set and by that time like you're not attacking the first set anymore and so you are kind of yeah you've let them be free but what

about those like is there any mechanism to like sort of shun a bad actor a bad node i don't think we have anything implemented currently but uh we could implement some kind of reputation system um it could either be like you know off chain and subjective so if you see a node that is like sending you too many transactions and it's actually causing you a lot of um it generates like a lot of resource usage on your side you can disconnect that node and and change the reputation of that node or we could also build something on chain um we would just have to provide some proofs of this kind of like misbehavior

so that other nodes can can also know that but the problem is that uh like it's subjective to something that is called civil attacks so the attacker if they see that they are kind of like blacklisted or someone figure out that they are malicious they can just generate a new identity and start lover again so you just explained sort of a scenario where somebody would attack the network but i'm not really clear like from what you said is it possible to actually take it down or like is it possible to actually affect it but um you can't really attack the entire network but you can do a targeted attack right if you if you actually like

somehow figure out that that particular peers are mining you can you can target your attack to to the miners and they are kind of like this this weak part of the entire network right but uh hopefully the miners are um you know protecting themselves correctly so there are a couple of ways to to avoid that like um mining on two nodes in parallel or preventing too many peers from connecting to your to your mining node i mean so it would obviously theoretically be possible to attack the network and actually take it down like it just requires enough resources let's

say there's ten thousand nodes on this network you can lurk in the discovery process for months try to discover all of these 10 000 peers and just like keep running the discovery algorithm for months and months if eventually you'll have a significant portion of the network discovered and then you launch a massive ddos attack with like tens or hundreds of thousands of computers with unlimited bandwidth on all of those existing pairs then yes you can certainly affect it i mean it would be a super costly attack and you know it's very unclear what you would gain from doing it other than to [ __ ] with the

network uh and would that just that would just knock it offline for like us a little bit it wouldn't be yeah i mean it would probably knock it offline until everyone can sort of take their notes down and like re-key their identities and get it back up again uh or something like that or and even like miners might not actually be part of the discovery process so the question is you know you could probably prevent transactions from being propagated but you probably can't prevent blocks being produced now this is a super extreme case but like say there are glitches or something like are there times where transactions are sent and they're just like actually lost before they hit anything

like are there cases of that oh yeah sure um because like either you're propagating that to appear that just goes offline right after it receives that transaction or um your transaction if he is too low to get that transaction propagated over because like we all when you are choosing a fee we also have something that is called gas price fee oracle and we are using um last uh transactions that are in the last 100 blocks and we are trying to average the gas price for that so if you choose um too small

if the fee that you choose is too small your transaction might look okay in your like you know subjective view of the mantle in your mantle but then if it's propagated to some some other pier uh they might drop it because there is like way more transactions with higher fees so it it kind of never goes you know out of your uh your notes so you're you're you're trying to propagate it but it's it's never acceptance what would happen on the side of like is there a ping back are you told that it wasn't propagated like how do you track that so if your transaction if you think that you're paying like high enough fee and your transaction is not accepted in the next

few blocks then you probably start to think yeah maybe maybe i did something wrong you can either like try to increase the fee or you can try to look in one of those you know like widely known kind of centralized um services that allow you to view their mantle so for instance on etherscan or i i believe my crypto as well you can go there and see if the transaction actually got from your note to to their note so it's visible in their memphis it doesn't guarantee that it's going to be mined but at least you know that it's you know it went through your through the network and it propagated to some other peers but in this case you'd have to search for that transaction

you'd have to check it yourself there's nothing that comes back and says like oh your transaction's lost uh yeah yeah that there's there's no way to to figure that out so yeah you have to you have to check it for yourself so we've been explaining all this from the perspective of like running your own node but obviously there's other ways to get transactions onto the network um like submitting it through you just mentioned my crypto what's the difference there how does that work yeah so um when you are using some uh other services to to to send your transaction you are actually submitting the the signed transaction to someone else's note so you're kind of you know like

going bypassing the the ethereum network propagation uh protocol and you are just going directly to your browser and like submitting to to someone else note and this is also a way to you know to propagate that transaction faster so you don't really need to only rely on material networking for that it's it's sufficient but if you want to do it like faster you can also look for some other ways to submit your transaction to different parts of the of the world or different parts of this this network one could expect that um big services like my crypto etc are reasonably well connected to the miners as well so that your you have maybe a little more security that it gets to the

miner but is there any downside in like centralizing this transaction you know propagation yeah i mean obviously there is there is some trust involved right if they see that you are trying to submit the transaction that like over bids them in some kind of auction for a crypto kitty or whatever then obviously they can sense or your transaction and never propagate it if you trust them then sure uh if you if you if you have multiple services you can also you know send to all of them and there is probably a high chance that at least some of them won't censor you is there any performance or like scaling

bottleneck involved with trying to send all all like say everyone sent all their transactions through my crypto um obviously it's it's like de-dosing their service because it's it's like one centralized server somewhere there in the world so if everyone starts to send transactions using their service it will probably go down and if if this is the only like way or the only mean you are using to send transactions you will be like you know cut off the of the network and won't be able to send more transactions so it's just like you know an additional option that you should use but you should also you should always consider running your own notes so that your transactions are are propagated

even if it's just a light node you have all the security and everything that you that you could want sure for for sending or propagating your transactions like notice is more than enough maybe just given that this is a blockchain 101 episode maybe it would be helpful for our listeners to have like a just a quick kind of recap of what we've covered uh in this episode so so we were talking about how you can create and propagate transactions over the network and what are the how how it happens so just to recap when you are creating a transaction you are signing it locally you are using a correct nonce which determines the the

order of the contract transaction you can also choose a fee for that transaction which is a gas price pretty much and this uh also determines the priority of your transactions compared to to other transactions that are in the member then you are submitting the signed transaction to your local node that local node is actually taking care of gossiping that transaction around and propagating it to other peers those other peers they have their own subjective view of the mempool so what they seen on the on the network they verify your transaction they insert that transaction to the poll they figure out what's the priority of that transaction and then propagate it further until it

gets to the to the miners and miners are this kind of like terminal nodes that actually take the transactions from the map for from their memphis and they they create a block uh from that transaction so they actually execute those transactions and um propagate the block back so that's how it happens hopefully people learned something in this episode and were able to follow along and if you have any questions or thoughts or feedback please send them to our twitter or email at zeroknowledgefm on twitter or hello at zeroknowledge.fm and we're also gonna probably link to tomac's twitter do you have a twitter

yeah i do cool so we'll link to yours in our show notes as well um maybe quickly we can say thank you again to christopher uh from 1kx for putting these questions forward because uh it definitely helped shape this episode and we really want to encourage our listeners to to do that to send us if there's topics that you're really into please send it to us send us questions that we can ask on on the air with some people that we're interviewing and thank you to thomas very much for being on this show and and being patient with our questions and helping to explain this stuff thank you so much for having me oh one last thing before we go uh i just want to say that uh i may be starting a new band called anna and the memples uh

if anyone wants to be part of it as this episode has gone through i just like realized the name is is too good to pass out um yeah so that's gonna happen uh cool okay so to our listeners uh thanks again for coming on this journey and thanks for listening thanks for listening [Music]