Lessons Learned from the Onyx Testnet

Transcript

[Music] guys terence from prismatic labs super excited to be here thanks icon for having me today my talk is about lesson learned from onix testnet as you know madasha launched a couple days ago and we sunset own it last week in preparation for madasha so this talk will discuss the lessons learned from onits to madasha and some of the nice things we have implemented so let's get started first i want to introduce my team we are a team of nine

follow us on twitter check us out on github there's nine of us and uh shout out to reddit who just joined our team last week and shout out to tons of github contributors that we have had over the last few months you guys have been amazing and we wouldn't have been here without you and if you're interested to contribute um join our discord we're happy to onboard you so let's chat a bit about the history and the present on the on this test next so we started out a ruby so back in ruby it was just a local chain that simulates proof of state between chain behavior it

was our earliest take on what the b channel we would we it was our earliest take on what the beast know we would look like and there's sapphire which was launched last year it uses 3.2 each instead of 32 eat and we only had less than a thousand genesis validators to begin with but we started implementing apis and we want more people to build up bitcoin chains and which such effort enabled a bit fly and ether skin to build awesome beacon um change explorers on top and there's topaz which was launched earlier this year it was aimed to be uh maine compatible

and he used 32 it per deposit it has slashing he has nice optimizations and then there's only switches launched early june it has been running for about two months and it was aimed to be multi-client compatible it implements the latest pad and we went through a run of audit with it so as you can see prismatic labs went from ruby to sapphire to topaz to onyx which is multiclan compatible and in the parallel multiclient testnet universe there was there was a schlesi there's wheatie there's autumnal these efforts were largely led by um

alfie and each multi-client test net has its own improvements and whether it's different genesis validator size whether it's the amount of validated team that participated and whether it's overall robustness and um just last week we have sunset on it and we we have moved to modesha so there's just one test net to work with and that is very nice quick look on the only stacks before we sunset there was 30 000 validators prismatic labs only ranked four thousand validators so that's about 10 of the network and those 200 000 slots 10 000

finalized epochs and about 250 to 400 peers in the network and in the next few slides i go over some of the lessons learned what we worked on and some of the exciting things that happened so um from onix to modesha there was tons of improvements there was about two months of the time there was seven releases about 500 issues and 500 prs they were closed and merged and were able to onboard uh 20 plus new contributors so that's very nice so yeah so let me go through some of the license learned and so the for those that have been

running a validator right you may have noticed that our existing key management tool isn't the best user experience so we have been working on clean that up and we've been working on revamp of our account management and since the introduction of eip2335 so what eip2235 is is the standardized keystore directory and file format across all clients to follow a similar strategy so image swapping keys between clients seamlessly and we have been refactoring the key management tooling so we're also making sure that new wallet tooling is much easier to use and uh we have the following new key

management type that's direct which utilize cip2235 there's derived which use utilize hd wallet and there's remote which utilizes report signing server via grpc they also integrate nicely with the e2 launch pad so yeah check our documentations as part of our effort to um revamp um how the key management or hub key management working prism we also want to uh sharpen our toolings as well so in particular many users want to run about the client in a cloud and they want to use a remote sign-in server elsewhere

that's able to securely store the key and send data as needed so in the prism validator client we have a grpc client building that that connects security to a remote signer as desired by the user so for reference and we have put our reference implementation on github it's licensed under apache 2 and also you can write your own on remote designer server in any popular language that you want which supports your pc as long as your server implements the required api schema for a remote signer defined in prison so one of the major improvement that's

required to speed up change synchronization has to do with how we verify block signatures right so the naive way of processing block is to perform sequential processing check every state transition check every signature sequentially but every time we verify one signature we incur our significant performance cost but so instead we actually improve upon that so we verify signatures in parallel we batch verify aggregated signatures we batch verify block proposal random signatures as well so we'll need to verify once

after a certain checkpoint so this helped drastically improve the speed of initial thinking and and such improvement it's very notewild if you look at alfred's latest uh report that you can see um in that link over there and with the full si with the full batch signature verification prism is almost as fast as lighthouse and we're confident that we'll continue improving this benchmark as us as we approach mainnet so lesson three in allnex we have experienced volatile

peer-to-peer messaging so user were reporting high error rates in process new blocks and that seemed to affect the finality of the nail watch so we identified the root cause seemed to be that prism client was getting out of sync and producing blocks that's older than last finalized plot so based on what we fixed and the issues was we fixed the issue and the um and we have reflected that on to the upstream spec as well and another thing is rate limiting right so rate limiting and peer scrolling refractory or has been started so our aim is to make sure that prison beaches know is more robust and

capable of performing its duty in presence of slow broken peers so the current ray limited system that we have is already capable of being pierced let's send us requests too aggressively and we're actually extending it to be more sophisticated in a way that how you and bad peers are detected and what actions are taken to prevent such behavior in addition to being we're also um diversifying a reputation system as well which will score appears based on a wide range of different performance matrix and we also throttle pier that violates uh care for these selected thresholds so point stamp is a security

the blockchain security firm has recently completed its audit of our prism client and it went over the course of two months so constant exam our food co-base for critical security vulnerability gaps in testing and he found a few bots were running on its test net as well and then he also gave us great feedback on end user considerations how to minimize security risk so you can read the full audit report here and quan sam also did an excellent job not only it's explaining great depth on some of the vulnerability

vulnerabilities he also included a comprehensive appendix would suggest the best practice to reduce a further attached service and so upon complete complete upon completion of the audit we created the comprehensive tracking issue to resolve all the items found by uh coin stamp we are um 70 or 95 there on addressing all the relative issues and we'll get to 100 very soon so lesson number five that uh we have base monitoring is very important we have made significant improvements on monetary

we've added a better matrix better rpc debug endpoint and better logins and better instrumentations we've added matrix such as runtime so we're able to see like when the cloud drift gets too wild gets too wide between a big node and validated client so test coverage improvement yes so one of the feedback we've gotten when running the onyx testnet is that we do like unit tests in certain areas so we have made improvements for the last for the last seven releases and that was the before and after and you can see the

blockchain package went from 36 percent to 57 and that's quite nice and do please do expect that we'll continue improving this as we get towards mainnet and in lesson seven in christmany labs we love postmodens whenever there is when whenever there is a drastic issues we are always on top to write the postmortem report this is just a good practice to have and report documents like recovery steps it documents the root cause it documents all the timeline as well so it's nice to have something to refer to and if you're interested

feel free to take a look so we started um thinking carefully about the biggest pain points for contributing to prison project and we have decided to run a small community survey asking everyone about their experience running a note interacting with prison and reading our documentations and it ended up yelling really productive results so the biggest feedback comes from the pain of learning and using our build system bezel and uh for instance contributors or that want to just use go that was

really hard so we actually make our project go getable after that and the other feedback that was resolved around account management which we addressed in backing lesson number one and we also got feedback such as like people want to run any sort of either one node such as such as nethermine which is not fixed in our latest branch so overall um community of community feedback drives really good initiative and the people's experience with art with prismatic labs seems to be really positive we're really happy with it and uh but by integrating testing understanding its complexity i

think we can do a better job and code health is going to be a big priority for us going through mainnet so if you have some feedback please do let us know so bionics now this is the last um screenshot that i took for onix before we sunset and as you can see it was already um since prismatic labs only run about 10 of the network and it was you you already had problem with finality right so it was about uh 500 epochs since finality that's about two days worth of no finality and even with that um

the note was still quite stable it was still like 2.5 gigabyte of rams and then cpu is fairly under constrained and there's no grpc error so we're quite happy to see that so hello my dasha so this is a screenshot i took before recording this and uh as you can see my dashboard has been growing great it's about 8 000 slots 250 finalized epochs and chris and we only run about three percent of the network but yeah i'm just really happy this is going really great so you may ask what's next for us right what's next for us is that we wanna be

as ready for many as possible so that includes define satisfied beta requirements we'll define the beta requirements we will share the beta requirements with the community and will satisfy that and then we want to do some further audits we want ux improvements as well we want people have the easiest time using our client and as i mentioned earlier we want to improve the test coverage and we want to go through more rigorous intervention testing so if you're interested in any of these items feel free to contact us we're happy to work with you and that's it from me thank you for having me and feel free to join our discord and ask us

any questions thanks guys