Hijacked at the registry: 26 hours without our domain

Transcript

[music] [music] Good afternoon Mumbai. Uh good afternoon DevCon. Um glad to be here. Okay. Um on April the 14th this year, u someone took our domain car. They did it without compromising our AWS infrastructure, our servers or our smart contracts. Um, it took us about 26 hours to get back our domain. Uh, and our users lost money. Today we're going to talk we're going to talk about how it happened, what we did and what the team did to recover the domain and how you can also protect your own domains.

Quickly about us. Uh, I'm Manoj and on the stage we have Nuno with me. Uh, together we are part of the DevOps team at Kowo. uh we look after the infrastructure uh powering cow swap. >> If you [clears throat] don't know about cow protocol, it's an intentbased metadex aggregator uh where users sign intents and solvers compete to fill them uh using onchain liquidity market makers and other dexes. Uh the users use cows because it allows them to get me protection less price impact traders also receive surplus and we have orders like limit orders and drops. uh the solvers also compete them to fill them within the traders itself. Uh that's what cow stands for which is coincident of want once. If you haven't checked us out go check us out at cow.fi and this

is also the domain that we're going to talk about today. So on April the 14th around 1421 UTC we got a couple of emergency alerts in our channels. Uh the on call was paged and the whole team sort of jumped into the alerts. Uh we also had some production deployments going around at that time. So the initial thought was that something went wrong with the deployment itself. Uh nobody's first thought was that we had lost our domain. Um so we started looking into the alert and the last line of the alert was sort of the hint of what was happening. Um our graphana instance was not able to reach our Prometheus or it was not able to resolve the DNS for our Prometheus and we sort of realized something was going around wrong with our DNS or our domain itself.

>> [clears throat] >> A bit of a funny thing. Uh we have a new joiner join our company that day and just after a few hours of their welcome message, they posted a message about uh the certificate being bad and this message sort of became our war room for that day. You can see it has like 388 replies. Um so yeah, not a very good first day. [clears throat] But about 1 hour into all of these alerts, we sort of realized our domain had been hijacked. This is what the hijack page looked like for our users. It's very similar to what you would see on co swap. The same theme, the same domain, but the code was a bit different over there. Um, so as soon as you open the website, uh, the website would pop up for a wallet connection and once you

collect the wallet, it will ask you for an approval for spending unlimited amount of tokens and sending it to a attacker's address. So the whole incident in numbers uh it took about uh 6 hours for us to get the attacker under control or the attack under control. Um the estimated user loss came out to be around $1.2 million. And just to emphasize none of our infrastructure was hacked. Our AWS accounts, our servers, our smart contracts were not compromised. This happened at the registration level. Uh CO also did a discretionary grant on a case- by case basis to reimburse the users. uh Nuno will go into a bit on how this happened.

So, how could any of this actually happen and what led to it even being possible? To answer the question, we kind of need to take a step back because in reality, web 3 security starts at web two. And even though we look at this whole mess on web 2 and think, oh, we're so glad we aren't over there and that we are over here where it's absolutely great and nothing can go wrong. The reality is while our smart contracts may live in web 3, while everything we often audit and care about lives in web 3, we actually need to worry about web two as well because our front end is there, our domains are there, our entire infrastructure is very often just there. So securing your infra is just not enough. It doesn't matter if you secure your stack, if you audit your contracts, uh if you secure entire stack and get

everything uh properly secure, uh if you lock your devices, MFA everywhere, you train your employees, you have everyone uh uh you have everyone expecting OBSAC related attacks and prepare for it. At some point, something can go terribly wrong when your register for example decides to transfer your domain to an attacker without you knowing. So, how [clears throat] does all of this work and how is actually a domain managed? At the top level you have trafficcom they are the ones who managefy. So when you have cowfi they are the ones responsible for just the fi part. Below them we have Gandhi who are our register. They are the ones responsible for managing our domain for like the administrative part. Oh who are we? Who owns it? Where do they where are

the name servers for this domain actually set up? And then it's us. We are over there. We manage the end DNS records. So we we decide where swap.codify cut five points but everyone above us is also responsible for everything else. We are AWS customers so we handle everything through Route 53. We don't actually deal with Gandhi. They are a company that we uh that AWS outsources the domain register portion to but unfortunately they were what we believe to be the the breaking point and where the attack actually happened.

So just to very briefly go into DNS just to like make the link because we talk about records, registars. Okay, but how does any of that actually work? So 60 seconds into DNS, you have a client that's like your laptop, your phone, whatever. Uh you have a resolver. So that's where you just perform DNS queries. You ask it, oh where's socket cloud fi? And it goes, okay, I'm going to ask the root server where's fi oh fi is over there. So I'm going to ask, okay, then where's kletfi? Oh, klet fi is over there. Okay, then where's socket cloud fi? Oh, it's over there. Then you tell it back to the client. The client just reaches out to the server hopefully via HTTPS and you have a connection established. Great. But how does that link to registars registry and whatnot?

Well, the root servers are managed by AA. You don't really have to worry about them. Hopefully they never got compromised. If they do, we are all absolutely screwed. [clears throat] Then you have the registry. They are responsible for the TLD. So the top level domain. It can be fi, it can be com or whatever it is. Then you are at the bottom. You have you are the registrant you are responsible for the authoritative name server so the one that decides where swap is in codifi that's you but I mentioned the registar and where are they actually positioned well they are in between the registry doesn't actually want to deal with you you are one customer among potentially millions they don't want to deal with you they will have registars and they are the ones who make like bridge the gap but the registry also doesn't want

to have to deal with the tld servers themselves like they manage the overall small thing but individual levels it's the registister and so basically you are reliant not only on yourself but on the registar the registry and INA not doing anything awful so what did it actually take for our domain to get taken there was no hack there was no malware as far as we know it was entirely a process issue on March 21st a lookalike domain was registered in late March a fake password not this one but we cannot show the actual one it was sent uh was generated along with other fake documentation and sent over to Trafficcom.

Trafficcom asked Gandhi to confirm like hey what's going on and obtained no reply. On April 10th there was a discussion between AWS and Gandhi where uh it was asserted by Gandhi as far as we know that they had to take some sort of action based on documentation that was given to them. On April 13th the domain was moved from AWS over to Gandhi. So it was just detached as in the responsibility was now entirely on Gandhi. And on April 14th, the holder details were finally changed to the attackers and a transfer key was generated and sent to them in some way. At this point, they had full control over our domain.

At no point were we actually contacted by any party, which a bit unexpected, but yeah. So, now is the point where I ask a series of questions which you may or may not know the answer to about your register like how well do you actually know them? Because there are a bunch of things to know about them like do they support MFA? Do they provide out ofband verification? Can you get a hold of them quickly if you need to? Do they stop domain transfers when they change the contact holder information? Do they support change notifications, account pins, DNS sec? Do they contact you when they receive a transfer request which well we weren't contacted so that's a bit handy. Uh do they support role-based access control? Do they have audit logs?

Do they support register locks? Do they adequately verify documentation that's given to them. Do they have a cyber sec cyber insurance policy? Like these are a ton of questions. You probably don't know the answer to all of them. We didn't either, but you probably should. So what would be the very first step? Well, you should check your own domains. You should just figure out who is actually your register. It may not be the entity you are paying and so it may not be the one you expect to be able to reach when something happens. We could reach AWS, yes, but we couldn't reach Gandhi. That's marath when the events happened.

Let's go back to that afternoon and see what actually happened on April 14th. We have a rough timeline for you over here for the first 6 hours. Um, so we know at 13:38 is when the hacker got control of our DNS. Uh, they switched the name servers and they issued new wild card certificates for our domain. At around 1421 is when our on call was paged and we started looking into the errors. At 1457 is when the certification errors started popping up and a war room was open inside the team and everybody sort of jumped in. Uh at around 1516 is when we could see a fake site asking for unlimited approvals on our domain. Uh our users were warned on X and the blockade and C911 team also flagged the cow. domain at 1620 is when

the attacker moved the domain from Gandhi to Netim which is a second register in this role. The team in the background worked on getting the website and the app back on a separate domain which we also controlled which is car finance. So we bought the API back up. We brought the sp swap interface also back up around 625 by 1549 is when the seat phase phasing started. So basically the wallet started blocking the attacker u and they started blacklisting him and the attacker sort of moved the attack from uh a wallet trainer to replacing with the or asking for the seed phrase of the users.

At 1937 is when we were able to finally get the attack blocked. Uh we were able to get a hold of netim and they uh blocked the name servers and the website itself. So this is when the attack actually stopped. We still haven't gotten our domain back but we were able to at least stop the attack. Getting here was not really easy. Uh we had to move to a bunch of providers, talk to them and get the domain to be stopped. Um so AWS was our official contact but they were not the registar. So the only thing they could do is contact Gandhi. Trafficcom which was a government agency worked at a government pace. Netim which was the second registar where the attacker moved the domain to was the one we were able to finally get a hold of

via a LinkedIn message reaching the right person. Gandhi which was our official registar uh we only got were able to contact them the very first next day that is also why introduction from the etherfy team because they also had some incident a few weeks ago um and everybody was sort of uh everybody sort of knew something had happened and everybody was sort of blaming each other but nobody knew who were the problem of contact so it's important to you know know how to reach your registar in minutes and not days because uh at a time of an incident you don't need to really rely on a lucky LinkedIn message to be able to reach the right person and while we were able to trying to get the contacts of these providers the attackers was also starting or to move very fast. The attack started with a

wallet trainer or unlimited token approval. It moved to a seed phrase freezing. They also tried to um change our MX records to intercept our emails and eventually finally they also redirected the domain to another domain which they hold completely control over. So yeah, we could try to block the tricks but the attacker was moving fast and these providers were very slow. Um, so the only thing we could do is try to block the attacker and hopefully hopefully that we can get our domain back. Uh, one thing that we sort of learned was that uh everything sort of relied on the domain itself. Um, so our emails or our coms were also on C. That means when the MX records were changed, we had to move our coms to our personal emails.

Our monitoring also relied externally on the DNS records. So we had to change that internally and our partners where an integration was lying also was using car 5. So we had to help them move to the new domain which is car finance. A good thing over here was we were able to get back the applications on a new domain quickly. Uh so everything that we had was in infrastructure as code. So we were able to get our API back online in 2 hours and the swap interface also in about 3 and 1/2 hours. If you see over here, a lot of time was just spent waiting for these providers to come back to us and our domains do get back to us.

Switching to a new domain helped a bit, but not a lot. Only one in five users uh followed us. Um so if you have 41 users every hour on mainet cowfi, uh we only got about eight users on the new domain. And this is sort of expected when you have a DNS hijack, nobody really trusts the new domain. But at least we were able to get our integrations back up. Getting the domain back took about another day. Um so at R0 when we saw the incident uh about 6 hours later we were able to get the attack to stop by getting a hold of Netim. Uh we were able to eventually get the name server switched back to u our old name servers which was before the hijack. At R25 is when basically Gandhi got back the domain and at R 26 is when

we got the domain back in our AWS account and we had full control. So it took about 26 hours to get a domain back uh which was lost because of paperwork and not an exploit. So what can you do uh if this happens to you? So as an what can you actually do [clears throat] after the hack happened? There were a bunch of postmortems propping up within the next few hours. You've probably seen the same happen to a majority of major events and it was out. It was basically completely wrong as most of them are.

The first few ones pinned the blame on DNS SEC and CIA records. These are about protecting your domain against unauthorized contract emissions and against uh DNS changes that are not authorized. These had nothing to do with it because the DNS changes happened not at the name server level but at the register level. None of this matter. The the attacker had full control over all of this. The same applies to CSPs and SR content security policies and the sub resource integrity. Those are set by the attacker as soon as they have control. So doesn't really matter. Then we have transfer lock. This one is a bit more interesting. In a typical use case, this will be handy. But the problem is it wasn't uh the typical scenario. It was actually our registar getting compromised. So it was our registar who gets to manage this was compromised. So

they could just transfer domain anyway. This doesn't matter. Another thing that exists though is a registry lock. That one is a bit more interesting because instead of being on the register in our case Gandhi, that's actually set at trafficcomiccom and as far as we know trafficcom did not go through with the did not finish the process of transferring anything over. So this would have very likely helped if it had been supported which unfortunately for us it wasn't. Then we have your choice of registar that barrage of questions from before. If you can answer them about your registar and you are happy with the answers that's great. we couldn't answer them about ours and if we could we probably wouldn't have been as happy as we were. So if your registar goes through with strict ID checks, they have change alerts, they you can actually

reach them fast, they do all of those verifications I talked about before, probably a good choice. Last but not least, monitoring. Yes, we knew that things went wrong, but we learned about it when the users did. We saw some things exploding before then. We saw some alerts that we didn't fully understand, but we weren't fully certain what was going on. If we had alerting monitoring exactly that, we would have known something was very, very wrong. We were not the first ones to get taken this way. Domain hijacking has been a thing for quite a while. In 2013, you had New York Times. In 2018, it was my either wallet. We had seller network and compound finance. We had Etherfi. We had Zapper. We had Curve. We had Hyperfi. And we had ETH

limo the very next week. This is not something new. This will keep happening. The best you can do is protect yourself for like the best way you can and make sure that the impact is as reduced as possible. But this will keep happening. This is not something that we can mitigate fully and so unfortunately the best we can do is prepare for it. So what are some common types of DNS related attacks because these are one specific uh portion of them. We are talking about unauthorized changes. We're talking about either stolen credentials, a register misbehaving or being compromised, a registry being compromised, INA being compromised, hopefully not, but all of those types of attacks. Then we have DNS poisoning.

That's a very common one that targets when I talked about resolving a name server. There was um sorry, resolving DNS query, there was a resolver you spoke to. This tries to trick that one into providing incorrect answers. Then you have BGB hijacking. This one is a in a layer below. So instead of just trying to trick the DNS resolution, it makes it so that the traffic you are trying to send to the correct destination actually goes elsewhere. So you're trying to visit a website in Mumbai like that is supposed to be hosted here and actually your traffic is being sent to North Korea. This was what happened in 2018 with my ether wallet.

And last but not least, you also have dangling records being exploited when we got our domain back. We we had some pages that were being hosted on GitHub pages and a very uh funny slash unfunny thing happened which is since our domain was detached we lost the verification status. So when when the domain was moved back into GitHub it was unverified and so anyone can claim it and an automated bot claimed the domain as soon as it was moved over. We had to reverify our domain to then get it switched back to our control and we could have our status page back. So what did we actually change after this happened?

The first thing we did was we switched to mark monitor for our registar. They are a very small company who has very very small customers like Google, Amazon and Microsoft. So hopefully we are in good hands with such a small company. Um we enabled registry lock. So this time we could actually get trafficcom to enable registry lock. So if something does go wrong with mark monitor which we hope it doesn't, we have another fallback like hopefully trafficcom will then catch it. And lastly, we significantly improved our monitoring. We now get alerts on basically anything that gets changed.

Our DNS gets changed, our who is information gets changed, our register gets changed, our name servers get touched in any way, we will hopefully find out about it very very fast and we'll be able to start reacting when we learn about it and rather than when the users learn about it. So what are the takeaways? What are the few things if I could tell you to take home from this I would ask it to? First of all, who is your registar? Run a who is check. You just need to type who is then the your domain and you will find who your register is. It may not be the company you are paying and if it isn't you should probably figure out how you can reach them and whether they comply with all your security needs.

You should lock your domain at the registry level if you can. That means enabling registry lock. Figure out who your registry is. See if your registar supports the lock. If it doesn't, maybe try to get in touch with the registry. Figure out what you can do about it. But ideally this will offer another level of support. Uh [clears throat] you should monitor and alert on everything you can. Anything DNS related always good to monitor and alert on. And last but not least you should prepare for the worst. This is inevitable like at some point something will go wrong. It it may not be at the level of DNS but something will go will go wrong somewhere and you should be ready to prepare for it. You should know who to call. You should know how to scale your instant response process and

you should be able to actually create a valid response to the attack whenever it happens, however it happens in a good uh time frame. [snorts] If you do only one of these, please do the first one. It takes like 10 seconds and you may be surprised by the answer and it may make you want to take uh certain actions. Well, thank you all for coming. Uh, we are now open to any questions that may be out there. [applause] Wow.

Yeah. Wow. What a story. Thank you guys for sharing. And that that just frustrates me that someone can AI generate a passport and steal your domain. Um, crazy. So, uh, we're passing this open to questions in the crowd. I think we got one up the front here. If you could please just hold that box up to your mouth. It's got a mic in it. >> Sure. Um, ultimately who paid for the loss? And then a second following question. It seems like curve.fire your domain and ether.fire. Is there a correlation between the dotfire domain and the registry? >> Um, so I can answer the first question.

Um, so the card did discretionary grant uh CIP86 to reimburse the users up to 100% of the verified losses on a case-by case basis. >> For the second question, can you repeat the question please? >> Uh, it seems like five Uh there's a correlation between your domain curve.fire and are they all Gandhi in the same attack or ether.fire too? >> Not fully certain whether they all are Gandandy but the thing that happens that typically these lesser used domain names that these lesser used top level domains have [snorts] registars that may have let's say less standards. If you're for example if you have an org domain the likelihood that the registar you are with will not have some of these checks at the very least is very low. So it's

not necessarily the blame on the top level domain itself like it's not because they're on but it's because the registars are typically not uh don't have the security features you would like them to have. >> Got it. Thanks. >> Lovely. Any more questions? Yeah, if you could throw the box up the back. >> Hey. Hey. Uh thanks for sharing the story. is really impressive and community benefit will everyone will benefit a lot from that. Uh I have a question. Did you consider any alternatives to web to web two and using just like uh infrastructure which everyone got used to like for example I don't know something comes to my mind

like onion pages like uh onion domain which you cannot like I mean to onion or for example distributing just like the front end it uh uh you have just a front p front front end page right so you can distribute it in in any way like through torance are like I don't know uh so yeah not not normal infrastructure something more secure >> yeah um so I think we do support ENS domains but the problem is you really have to be when you're a web- based business you have to be where your customers are and if we are not on some of these domains we will not be able to reach our customers um that's what that's what they use that's what the domains they are comfortable with um so

if we are only on the ENS domains it might not really work for our business so M >> I think I think customers would benefit because I would trust something uh better and like uh many people will trust something better than normal infrastructure. >> Agreed. Agree. I think it's uh it's good for like the technical users but we also have like a wide range of users for co swap which range from like say normal traders day traders to uh Dows. So we have to like serve everybody. So >> yeah I mean not to replacing but just an alternative providing some secure alternative to the uh normal infrastructure. I think we we do support domain. So yeah, >> thank you.

>> Hello. Thank you. Uh very happy cowop user. So shout out to you guys. Um question on the investigation side afterwards, right? So you have this attacker. I assume you spoke to FBI and other folks. Does this just like go nowhere? Like did anything actually happen to to this person? Did you find out, you know, who or what it was? Um, so I think we were not able to find who the attacker was. Uh, but there's like some legal things going around where we're trying to get more information into it. Um, so legal things take a long time. Um, so that's why we're still struggling to find a lot of information. Uh, but the process is still going on.

>> I see. All right. Good luck. Thank you. >> Yes. [music]