How to turn confusion and churn into adoption and retention

Transcript

coming um this presentation this session it's a workshop right so we are hoping to get interactive with you all uh is about how to turn confusion and churn into adoption and retention um my name is Oliver Renwick um and uh I work for drawer over here. Um I'm the head of the customer success content team. Um and Dor can tell you a little bit about what that means. Uh many think many people will say that the content is boring, but you will learn later on how important it is uh for this topic and in general. Uh my name is Jorelli. I'm the head of customer success uh in consensus. I've been in the company for a bit more than seven years or seven and a half years.

And uh customer success is basically touching all of the different uh product and services that we have in consensus. So servicing uh everything that we offer. Thank you J. So uh the thesis that we have is that crypto isn't failing because I don't understand it. It's too confusing. H it's new. That's not that's not the reason. The reason is because we as a community, as technologists, as builders, we need to do better at explaining it, supporting our users, and ensuring they succeed. Don't worry, I won't just read every slide to you, but this one was really important. So, I wanted to go ahead and emphasize it. So, that's that's the bottom line, right? We

can make this adoptable. We can make it usable. We just need to do better. So, we're gonna talk a little bit about how we do that, but first, I think we're going to have a little Q&A. How's support in web 3 doing everybody? Uh going going good. Um Drew and I got on a call a couple weeks ago to prep for this talk and it's like he he immediately gets a call as soon as I joined the Zoom call right on his phone. He's like just a second I'm going to take this and he says hello. He talks for a couple seconds then he hangs up and I was like is everything okay? That seems serious.

And he goes oh no it was a Coinbase support scam. And I was like I love this. The head of customer success for consensus is getting Coinbase support scams. So, uh, how many of you get those scam calls? I I literally got one as I was walking up the stairs to this room, uh, from the United States. So, so love that for us. Um, so yeah. Uh, how do you how do you guys do support um for your for your products if you if you have it? Um, feel free to raise your hand and we can get you on a mic or Yeah. Uh so we have a telegram group. Um there's

things there's an autobot. So whenever uh someone makes a transaction um we get a notification and also there's a help channel. So if people need support they can just drop it in the telegram and we see it right away. Okay. Okay. Anyone with any Discord supports infra in the room? Amazing. Yeah. I'm surprised and thank you. Thank you. Please don't. Um uh yeah the the number that's even for scammers uh discord server is uh and in some respect also telegram is the best place for the scammers just to be aware.

But yeah but this code is really to avoid. Yeah. Um you know somebody looking for help goes to a Discord server and there's somebody that DMs them offering to help. That seems like good support, right? It's it's not. It's not. It's a scam usually. Um so that's one thing. Uh what what about challenges for explaining and supporting users? What what challenges are we feeling most strongly? Like what what are the pain points in explaining and supporting your users? Is it the concept? Is it uh you know implementing open-source libraries and multiple you know multiple different sources of truth across chains? Anybody want to

throw some out there? Yeah. I'd say for many cases I've seen it was mostly about trust rather than understanding that the trouble they had. But mostly it's not that they didn't understood but that they would rather refuse to understand because they was not trusting. They were not trusting. Interesting. Crypto is trustless. Did you have something like the mic? No, I think it's it just depends on the segment we're talking about. Yeah. Um because it's uh if it's people that are not in web three, then it's difficult to explain them because they don't understand what do I need

that for? Like for example, the basic thing as registering a crypto wallet, you know, if people don't use uh crypto, you know, right? They either they don't know anything about that. There is a lack of education there. So, yeah. Yeah. Um, want to go a couple more? I' I've got more questions. All right. Couple more questions for you. Uh, where do you guys think So, so MetaMask is, you know, our flagship, one of our main products at Consensus. Um, so where do you guys think most of the MetaMask users are? Is it you can shout it out and I'll repeat it.

Africa. Africa. Is this something over here? USA. Okay. Turkey. Turkey. Okay. Well, yeah. Yeah. I mean, you guys nailed it. You guys nailed it. So, interesting little fact that we've discovered over the last couple years. Um, uh, like if you look at regions for the extension in the mobile clients for MetaMask, um, Nigeria is by far number one for MetaMask Mobile, like by a wide margin. Uh and if you look at the numbers for extension, the last time I looked they were number 114.

So that's an interesting thing. India has very broad uh adoption on both platforms. Turkey, we've had a long time very dedicated user base as well as Indonesia. Um Indonesian users have the Indonesian user base has hovered around number four, number five for a long time. And then uh linguistically speaking, I don't know where they live, but uh we see a lot of Chinese and Russian speakers as well using MetaMask. Um the United States is probably the largest uh single national market, although a lot of that is probably VPN traffic, probably 10 to 15%. We can't tell. Um or it would be really difficult to tell. Um but they're by far the minority, right, of overall users. So,

all right. Should we move on? Yeah. All right. So, I'm not going to read this one, I promise. Uh, but here's an agenda sort of outlining what we're gonna what we're going to go through today. Um, and uh, I've got some notes here because we're going to start. This guy over here has got a lot of experience and uh, we we wanted to share some of it with you. So, this will be a slightly rehearsed, mostly improvised interview with uh, with drawer. Thank you. Yeah, maybe we'll stand here. Yeah. Okay. Okay. Yeah, I will. Okay.

All right. Drawer. So, Drawer, for those who don't know him, uh he's got an impressive career like running support organizations for huge legacy businesses, right? Like worldwide businesses. Um so, what are you doing in crypto drawer? Yeah, I've been in a legacy software for like more than 20 years. Then around 8 years ago, I got a very weird email from a weird company in a ecosystem that I didn't know anything. It was an email from Consensus. Uh I decided to respond in a semi nasty uh kind of uh uh message to the recruiter asking why the f you are contacting me. And uh and basically I learned quite quickly that they were

realizing eight years ago that crypto will reach the volume that we are seeing today. And the visionary people inside consensus uh starting by Joe Lubin realized that they need to start implementing things that will scale uh back then and the rest is history. I'm obviously uh fell in love in crypto during this interview process. didn't didn't sleep at all and found myself within few months uh inside consensus. And for the record, drawer has told me on multiple occasions that Joe is the his favorite CEO that he's ever worked with. So, um just putting that out there. Um so, uh we've been talking about success. I don't know if anybody's noticed that we're not we've talked

about support, but we're making a distinction between the two and we want to emphasize that. So drawer, could you explain to us why we call it customer success at consensus and not just customer support? Yes. So the concept for me support is a very reactive uh approach. uh you basically are waiting for the user to fail and not only to fail to realize that they want to reach proactively but from it's a reactive to an issue that they have but they proactively reaching the vendor seeking support. So this is really a kind of a desperate move by a user. So the question for me was always what we need

to do as uh projects, vendors, companies to simply make sure that the user will be very successful. Um and there are different types of mechanism on top of the product that need to be done. Obviously the product need to be the basic of that. uh and we implemented different functions inside my organization to make sure that the user will be successful and then when the kind of the terrible time come and they need to reach support that they will get the best support that they can get in crypto.

So what what like structurally you mentioned that we put other things inside of customer success at consensus not just support uh is there anything that specifically that we do differently? Yes, we we have two additional important function. Um and um the first one maybe we should uh jump over. Yeah. Yes. So uh you can see the building blocks. I will not get into all of the details but uh to the left you see something called SR. This is site relability engineering and in most most companies this is either part of CTO, part of the product uh very close to the product. Uh for me it's very important

that it's in part of the customer success organization because when you have availability issue, scalability issue or security issue you have massive disruption to users. So many users uh will feel it and many user the first thing that they will do they will approach support when things like that are happening. So I believe that there is a very close tie between the support organization and the SR. Obviously they need to be involved with everything that related to the product but this is extremely important and a proactive making sure that availability is the highest and scalability is the highest to make sure that there is no major disruption uh to user. On the right side you see a builder engagement organization.

Uh and the idea here is that um builders are part of our flywheel. They bring most of the user to our uh product and you simply need to manage the relationship with them, manage a lot of the technical integration and technical effort with them and that's again at the end of the day contribute to the user success to make sure that what they were trying to achieve using let's say metam mask or infiora or linear they will be able to actually achieve and I think that uh most of the discussion from now on will be focused on support and we will continue to zoom in on the different some of the different boxes inside the support

organization. Thank you, Jar. Um yeah, so that is the high level right here. Here's our org chart. Um and so I I work within the support organization in consensus. My that's where my team is placed currently. Um, and it's a good place for documentation to be, but that's that's uh let's let's move on from the org chart nitty-gritty. Um, and zoom back out to the big picture. So, you've got this complex but not complicated. You know, everything's in its place for a reason, structure that we've outlined for you, but does it scale? Um and as George mentioned like if and for those of you who sort of know consensus and the kinds

of things that it's put out over the years there's been a lot of stuff that's come out of consensus and not just one user profile right consumerf facing dev facing uh sort of business uh integration business stakeholder is sort of the user profile for some of the products right so how do you scale support for all those different user profiles as a technical writer I am here to pump my bags and tell you that you need to explain your product to people Um, and uh, and and I don't just mean a cool marketing landing site with some weird shapes and like a white paper that the founders wrote two years ago. You should really explain it a little bit better. Um, so, uh, I'm I'm going to keep it high level and I'd be happy to dive into details of of different options and integrations in the Q&A

later, but um, in the process of spinning up helping support spin up documentation and processes for all these products over the years at Consensus, um, I came to sort of this this realization that you're always going to need docs, whether you think you do or not, or whether your product manager says the UI is so intuitive, no one will have any questions, there's no bugs, no one will have have any questions? It'll be great. Um, we have had one product like that. Um, but uh, you're always going to need docs. And I spun up some sort of this is just fake data that I was trying to use to illustrate the point. And you can decide where the point is on these graphs that you're going to need the docs, right?

But, you know, the the one on the left is sort of your normal software product. You start out with an initial feature set. You get some adoption. Uh, you add more stuff, right? Um, and uh, and then you iterate, right? and you get more adoption. So, at some point there, you're going to get enough users who are going to figure out how to use your thing wrong or your feature set is going to get confusing or complex at some point that you're going to need to explain it, right? Um, this is like I was thinking, think about SMS, right? SMS and two 2005 is the thing you just use the T9 and you send a message, right? But SMS in 2025, there's a lot more to it, right? and you actually kind of need to explain how to do different things and how it works in

different contexts. And this is maybe something higher friction, more niche, uh that that a dedicated user base is going to use, right? Uh PGP encrypted email, you know, that kind of thing. And you're going to have to explain it up front. Um so yeah, so so you're going to need docs at some point. And the bottom line is oh there's there's one important point when I say docs. Uh how many of you are thinking of API specs right now? Right? This is I'm not from a tech background. I was I was a translator before crypto. Um and so this has been a different mental model for me. Um I would challenge you all to expand your

concept of docs uh beyond just an open API integration from the code comments of the API spec. Right? Um when you buy a product at a store, I bought a mobile power bank at you know the the corner store the other day. It had a user manual in it. It had a getting started guide, a troubleshoot. You know, you need to ship something. Even if it's consumerf facing, even if it's dev facing, even if it's for business people, you need to explain what you're doing. And why is that? Because when you succeed uh and you gain that adoption and you hire support agents, they're going to start answering the same question over and over again. And they're going to write macros and canned answers and stuff like that. And a lot of the time they end up writing a pretty solid set of unhappy path documentation.

Um, but it really helps to complement that with upfront proactive like Dor was saying, educational resources, explainers, right? And you do it upfront. You write it once. You maintain it forever. That's true. But you do it because you can use it 10,00x. And drawer is going to explain to you why and how in 2025 you can do that. Yeah, someone can guess how we are using now this in a multiply way the content. Any guess? Probably the most sexy word in the last two years in tech in the world. AI. Thank you. Uh so two years ago,

literally every other day, sometimes every day, I got uh an email or LinkedIn or Telegram from a new CEO of a new company promising the world doing LLM solution for support. Uh this is kind of a classic use case for LLM. uh and we were kind of a little bit concerned because you know managing the biggest support organization for us is obviously our own meta mask millions of user the support volume is huge. This is a real production machine environment making sure that we will have high level of support and over the years we developed a lot of automation using the old technology of the past and this is technically a bot. Uh to do a bot you need to design a workflow a decision

tree. Uh this is the basic decision tree for our previous bot um managing the MetaMask support. Now compared to the regular um support procedure that we had before this amazing workflow, this is obviously provided better results and uh linear organization and everything that related to that. But this was quite limited. uh a year ago basically we decided to give the market to choose some winners in this crazy LLM race and then we decided that we will do a race between three vendors and one solution that we decided to build ourself on top of LLM and the reason to build ourself we knew

that we this is going to be core to our future so we really need the knowledge inside we took few engineers from the support organization task them okay you learn everything about LLM And the beauty of using some type of a race like that on production data quite quickly we were able to find the winners. We had one vendor that won big time and our own solution and uh we went into kind of real production. Uh we are running it already three quarter I think something like that and the results are um shocking. Okay. uh we are processing 90% of our volume today using uh LLM. We

have resolution rate of more than 70%. We have seesat that is in the 85 90% better than the seesat that we had with our support agent that that are very good and there is a reason for that. LLM for example will never snap. They don't the LLM agent doesn't have a bad day. Support agent can have a bad day. a user can trigger something in a dialogue. Okay, all of that is not happening. So amazing result uh from all of the statistical information. The additional thing I have an habit I'm reading we have a slack integration to all of the support ticket that user are providing seesat and also open text

message. So I finish my day reading that it's kind of nice way to finish a long day. uh because you see people that got great support and they're happy and they are communicating about it and then I started to notice some trend first of all you can see on the left the people that don't really believe in LLM and they are sharing that and many cases they are happy when they got uh a real person to provide the support and on the right you can see two things first of all we are not hiding the fact that they're talking with an AI agent you see the name Finn AI I and by the way in the ticket itself they see AI agent that's the name we are not there's no fake name or something that they know that they

talk with AI and in the last few quarters I noticed that many of the user are first very happy willing to write open text comment to an AI agent and also when you read them you sometime can see even a personal attachment that they created during the the interaction. So all of these results we started to talk with the CEO of uh the main vendor and first of all he acknowledged to us that we have the best result for them and it's a regular web to tech company that serve thousands of customers most of them traditional support uh there are hundreds of customer already using the

LLM and we are the best and we were trying to do analysis what is really what are the key ingredients that uh drove this success One of them and one of the biggest one is having very solid content. Obviously we have access to the content. We have access to the history of the all of the tickets. We also use GitHub but very solid content gave tons of power to the brain of LLM to really provide great support. Um and there is an even more and I think I will give Oliver to talk about it. Thank you J. So briefly on that point just to give it a little flesh it out a little bit. the the content we're talking about is support.amask.io.

Um, if you've never been there, I love that for you. Um, uh, but we designed it, we started writing it four years ago, and the help desk agents were like, "Hey, can you explain what an account is? Can you explain what a wallet is?" And I challenge any of you to explain what a wallet is. Um, uh, you know, things like that. So, there are a lot of highlevel conceptual explainers. What's a transaction? What's a token? That kind of thing, right? But then we've also got those really nitty-gritty product flows and troubleshooting flows. So the LLM, it's it's like the perfect recipe for an LLM, right? You give it that broad context in which it can operate and then you give it lots of certain specific cases and then it can make inferences and whatever the other thing is, you

know, inferences and uh and other logical jumps, not hallucinations. Um sometimes um so the LM can do other stuff too. um that bot flow that you saw before many many many hours went into making that and translating it had to be a very manual process as well and previously when we had in chat support I mean I mentioned previously we have a very international user base right so we were and we don't speak every language right so our support agents are using a custom Google translate API running in a in an AWS instance whatever right to translate all the support conversations brittle to maintain pain expensive to run um and it's Google translate so but LLMs most

of them have that baked in some of them have better language support than others of course um b based on the training set but this is the seesat score from the past couple past three months um that's customer satisfaction if anybody doesn't know the the industry jargon um so you know look at the the customer satisfaction between the English and the non-English like clearly the LLM is doing a good enough job that the user can get their issue resolved to their satisfaction. It's pretty impressive. And to give you a sense of the the scope of that, um this is the numbers in the top seven languages over that same time period. Uh this is weekly data, I think.

Yeah. So um so we've got uh Brazilian, Portuguese German French Russian Indonesian, Spanish, and Turkish. You guys hit the nail on the head. So, it's not uh it's not a trivial question of providing support in in multiple languages. If I've heard a lot of people be like, well, everybody in cryp crypto just speaks English, bro. And it's like, yeah, okay. Yeah, the people who are already in crypto, right? So, do you just want to just want to speak to the echo chamber or do you actually want to get the next billion users on chain? If you want to get the next billion users, you're going to have to translate your stuff and speak it speak in a different language. It's not easy. We keep making up words like danksharting that can't be translated. But moving on. Um, so we've dumped a lot of stuff on you guys

and uh and we did want this to be interactive. So um any questions? Any AMA and how much time do we have left? Yeah. Uh yeah, I just had a little question on whether there is a way you decide on which uh customer question is going to be answered by the bot or by an agent. Is it completely random? Is it arbitrary? Is it anything? I will start from the eye level and uh Oliver can correct me or add more. Uh so uh the beauty of the LLM is that you don't need to decide.

uh it's not like the flow right uh the LLM will either responds the LLM will responds worst case he will respond that he doesn't have more information he will suggest to move you on to a human bin okay so that's the beauty you don't need to design it you don't need to think about it so by default it goes to the LLM and then after it goes to a support agent correct sweet yeah we do we do have specific um cases for things that are flagged as like the us if the user says hello AI I've been hacked that triggers a flag to go directly to not even first level uh support or maybe first level support to do intake and then passes it

directly to second level support. So we do have certain cases like that. Um if you ever have a problem with metamask swaps uh you also get the fast lane. I I will add something on that. So we have few areas that we are really taking seriously. One is obvious obviously everybody that gets scammed. We have a trust and safety team that are basically taking it uh very carefully. By the way, we are considering now because of the success of the LLM and unfortunately the volume of scams in the ecosystem maybe to use LLM in a smart way but time will tell. Second, we are for the future we are looking about doing some type of customer segmentation.

So and and we will based on that decide either LLM and then depends on the different levels of support agent that we have. Uh yeah. Um have you tried instead of just customer support uh product recommendation or product discovery with ALM? Um stay tuned. Yeah. Yeah. Stay tuned is what I would say. Um, I'm I I want the LLM to be as strictly scoped as possible. You know what I mean? Because, uh, as Dora was saying, this is crypto. The scams abound. Um, I don't know if any of you caught this, but when Google launched

their like little preview feature where the AI will just answer things to you. There was a time period where if you type in what is the phone number for MetaMask support, we don't have a phone number, but you would get a phone number and it would go to a scammer, right? So, there's unintended consequences. Um, yeah, I I will add something about it because I told you that you we waited one year before implementing. One of the fear that it wasn't important for the presentation today but it was also there. LLM can with tons of confidence. Okay. Uh LLM will be used by this scammer. These are businesses. We know it for a fact. We are working with all of the different organization in

crypto that are trying to reduce it. We are well connected. Some of our this trust and safety team. Me myself are connected to a lot of the web free application security companies. These are professional organization with OKR and goals. Some of them are nation state as you know. Uh there's a lot of money there and they're not giving up on the revenue. Okay. And the identified LLM. So we need to be careful in what we are doing and implementing with everything and also implementing LLM how to react to what will come from the scammers.

I have nothing to add on that. Um yeah, thank you. And in the same train of thought, uh are there any other areas that you're looking to implement or are testing to have agents, compliance, security that you can share? Uh can I start pass? So uh one thing that uh that actually drawer asked me to do recently was to to start looking at at how LLM how good are LLMs at translating, right? Because we do translate a lot of content to consensus. We do use human translators. It's not free, right? Um translating with LLM isn't free either. So, uh just share a little knowledge. Um we did a test pilot uh with our translation agency partner

and when we pitched this to them, they were like, "Okay, I'm going to manage expectations first and we were like, okay." Um and they uh they introduced me to the concept of the figs, French, Italian, German, Spanish, and probably Portuguese as well, but it doesn't fit the acronym. um those are the languages that a lot of LLMs do better, right? Because the corpora the corpora that the languages are the LLMs are trained on are better. Uh there's a whole, you know, context there, socioeconomic context. So, we're looking into it. For some contexts, the trans the EF, for example, I was talking to their to Luca who runs uh their website. They do a thing where they'll use an LLM translation for the content initially while they're waiting for the human

approved translation to come through. So it's like it's good enough for now, right? So that's that is a and I'm as a as a translator I'm okay with that, right? It's good enough for now. And I will add two more areas that I see kind of coming in the future. First of all, for the product itself, not not on the customer success side, but it's starting to in my opinion to kind of cross the lines. uh but I believe that what we called in crypto as intent manager that people you know doing I I I reviewed myself few intent uh management companies it's all about workflows I believe all of that will move to LLM and it will be designed and built by the product itself you don't need another

company we already have a small team that are kind of checking it so I think that in the future the heavy UI UX that we have on products will go back to the fact that we So giving it, you know, 20 years ago, 30 years ago, we had a command line. Only few people knew how to do it. Uh now we'll go back and we'll do a free text, you know, in your own language. The product will understand what you are trying to do. We'll give you a summary. You will need to do a thumbs up for that. Obviously in crypto, you know, if the LM didn't understand you, something bad can happen. Uh but I think this is the future on the product side.

and um and that and that's by itself I think will be will be uh a big one and I forgot the second one so maybe we'll come back. Sorry it's fine. Any other question? Uh hey yeah so as you said the more content and more docsuh we have right the better uh the LM can understand and answer the uh people right but what about like we are getting questions to the LM and L&M can be used to train the docs and content create content so basically my question is can we use LM to you know or you can suggest LMS to basically um create better docs um yeah it's an excellent question and a

lot of people a lot of people are saying it. Um, so we've been piloting, okay, we've I'll be quick. We've we've been piloting this um with our with our AI. Um, and there's some interesting nuances to it. I think if you get the LLM integrated properly and your content is maybe tagged uh or let me explain. So, we we ran an initial sort of like content gap analysis on the support site, right? And the LLM, we were like, "Yeah, that would be awesome. tell have it tell us uh like look at the support questions and then look at our content and tell us what users are asking about that we don't talk about right fill in the gaps that'd be awesome and it was like okay cool I'll look at

the gaps and then write articles for you and we were like okay fine we didn't ask for that but so it came back with um some interesting results uh and some of the the highest priority in terms of its its list were topics that we actually covered but it was like the users are using a slightly different word or the name of the article is sort of like the general topic and then the specific topic is sort of buried in it. Uh so you're that's that's where there's sort of like a misalignment, right? That the LLM might be reading everything and ingesting everything, but it's not necessarily understanding this equals that. Uh but that's not every context, right? The the metamass support site is a general a very broad audience, right?

And we're explaining a lot of topics. if you've got a narrow focused set of like technical documentation, I can imagine that working a lot better. Um, and and another important thing is making sure that you reync or retrain the LLM every time you push updates. And we'll add one more uh thing about it. Uh, because people are also asking the same question about the support agent. So I think we are moving from a traditional support organization that has a lot of people on the you know the layo zero level 01 to a kind of a shape that I still don't know what it will how to define it uh because for example content team we need to increase the team there's much more demand to content especially when you realize how important it is and yes they

will get more efficient and they will use LLM in the process but it's becoming very important we have a large team that know LLM infrastructure structure configuration everything that related to that. This is also something that traditionally you didn't have in support. So we really have subject matter expert in LLM and all of the different solution and we have tons of new solution. So um it's we are in a massive change because of that. We have time for one more question. Hey um yeah we started this talk uh so we don't Oh we don't okay you can come ask us.

Thank you very much guys. You have our contacts and if you want we will answer question. Yep.