Learning in the AI era

Transcript

[music] Hi everyone. I'm uh Pablo from Bill Gill. I work at uh on product and uh we mainly uh create products around dev tooling and and learning in Ethereum. uh all our sessions today are about uh how AI and agents can do on Ethereum but uh during my sessions uh will focus on education exploring how agents help us learn or not and uh how we are changing the way we learn and how we are adapting our educational uh products.

I'd like to start with a story that I read in an essay by Carl Henrik. In aviation, the line that shows on the screen when the autopilot is on is a magenta line. In the '9s, a captain from American Airlines warned pilots that they were uh becoming very good at following that magenta line, but they were losing some of their skills to flight the planes manually.

Years later, an Air France flight had problems with their speed sensors and they had to switch off the autopilot. the control uh went back to the pilots and they had to uh use the manual mode and they couldn't recover from it and the plane crash and they had an accident. Um that wasn't the only cause the the manual flight but of course their lack of skill uh manual flighting uh had to do with this action. Hendrick points is that automation is not dangerous but overdependency of uh AI and automation is what is uh dangerous

those days and especially for people learning is even worse because pilots can lose some of their skills but they already had uh that knowledge but people uh trying to get that knowledge on first time are are having a trouble able those days. Now thinking about code, we have our own magenta line everywhere like in the terminal, in the IDE, the chatbt cloud apps, everywhere and we just tell an agent what we want to build. He writes the code, writes the test, we accept it. And how you saw in in the last uh uh talk by Austin, most of us

are not even reading the code anymore. Uh this is a real exercise from the learning lab. It's a refund function from a crowdfunding contract. And the AI can of course write this in seconds. compiles, all the test pass, everything looks good, but do we understand the output? Do we know if it's safe or not? And does it work the way we really needed to? Uh we we saw that everyone is writing uh code with their agents. Uh I'm not going to to ask again, but uh a few years ago, working code was a decent sign that

someone understood it. That's not true anymore. And those lines in in this contract, if you take a quick uh a quick uh look, you will find a line that would have stopped the attack that drained the DAO back in 2016. Uh you can pick it and think about it, and we'll check later in the demo. But this goes beyond the code. Let's zoom out a bit and think for learners in general.

uh is a test around thousands of uh students 15 year old students all across the the world and for example in the OECD countries their reading skill is at the all-time lows for the last 10 years and they also track what they call hasty readers who are students who go through the lessons real quick but uh don't get it and uh this uh this number of hashy readers has almost doubled from uh 2018 but uh let's be clear the AI is not the cause uh from this because the decline

started uh uh before chat GPT was was uh born um probably a screens and social media has a lot to do with this problem but what is true is that AI is landing on top of it so we already have the problem and AI might uh might uh amplify that. And also at university the professors at MIT talk about cognitive surrender which is that uh students at the first uh problem or whenever things uh go hard they go to a chatbot and ask and try to get help from him. Uh the answer is a principle I like. Augment

don't automate. These are different signals but there is no single cause but it's all pointing to the uh same issue. Who is still doing the the thinking? So if we want to keep doing the thinking in the future there are two questions to answer today. First what do we still need to know? And second when does AI actually help us learn and when doesn't? So question number one, what do we still need to know? At MIT, a committee is openly asking whether most of the students will still be writing code by hand to learn it. I think maybe not by hand, but uh you need to know enough to tell if it's right or

not. And here are some everyday examples I want to to check with you. For example, uh the AI adds a catch to speed up a page. Sounds really good, but do you know what happens when the data change in your app? Or for example, it builds a connect wallet screen for you uh works perfectly, but will a newbie that first come to your app know what to do with it? Or the example we saw before, the the refund function, everything works okay, we see a bunch of tests, all passed, but do the tests cover what could go wrong with the users's money? You have to understand the contract to be able to

answer that. If you if you don't understand it, you can't really review it. But most days it doesn't go like in the reference story. The AI don't switch off and you don't have to baby uh babysit the the AI like we had to do one year before. But it it still does takes for you a lot of decisions uh adds a lot of stuff that you didn't ask for and uh that's drifting and you have to understand what you are doing to know if those ads that uh the AI did the the decisions uh he took you need to to know if they look right or not and you need

experience and judgment to be able to do that. First you need to notice is this what I ask for then you have to judge it does it fit what you are really building and then you have to fix it if it's needed so you have to tell the AI how to do it the way you you want it or you have to do it uh yourself that judgment is what people call taste but it isn't talent you get it from making the stuff yourself making your own mistakes facing problems and trying to to solve them. And that's starting to matter for jobs. Stanford published uh

data this August and in the jobs most exposed to AI like for example software employment of people 22 to 25 dropped uh by 20% over the last year. Uh that's uh mostly because companies are hiring fewer journers. It's uh there is no such gap yet for experienced people but uh we will see. So that's what worries me like if juniors skip the learning and they going don't get hired to do it. Where does the next generation judgment come from? You can't take over from the autopilot if you never learn

how to fly. And question number two, when does AI help? Let's start with the good news. Back in 1984, Benjamin Bloom found that average student with a personal tutor did better than 98% of the students in a normal class. The problem was that nobody can pay for a personal tutor. But those days with AI makes it almost free to have personal tutor that is there anytime in your language. It will explain something as many times as you need and you can ask the questions you wouldn't ask in a class.

But there is a catch. For example, I heard about a study recently in a podcast that they did on China. They look at thousands of students of secondary school and uh once they started using AI, their homework took them less time. They had better marks in the homework, but then they had 20% worse marks on the exam. It also showed at the university exams. Carl Hendrik calls this illusion of learning and it feels like you got it when you are using AI, especially if you are using it in the wrong way. So you put less work and then this less work makes you have worse

performance in the exams. Uh the good news is the students that use AI but kept spending the same amount of hours uh for the exam, they had the the same marks and as those uh not using AI. Uh like I said, it depends especially how you're using AI. If you are using asking him uh about stuff or you are using him uh AI asking just to to make your homework. So how you use it matter? The best evidence evidence I found is from Anthropic. They took 52 developers mostly junior and had them learn Python

library and half of them use AI, half of them didn't. Then they took a quiz right away and the group with AI did clearly worse around 50% against 70% from the people not using AI. And the biggest gap was debugging which is what you need when the AI drift. They were s they weren't significantly faster either. But inside that a group it depended a lot on how they use it AI. The ones that mostly delegated like write this fix that that group score between 24 and 39.

But the ones that ask the AI like to explain things and uh create the like interactive stuff to to learn the concept concepts better. A score between 65 and 86. Same AI much different results. I know those are small groups so it's just signal. It's not a law but it's not the the only study that showed that kind of behavior. So what seems to matter isn't so much whether you use AI, but whether you are still the one doing the thinking.

So from all of these lessons, we came up with four rules for building our tools. The first one might sound a bit strange, especially for us that are builders and always try to remove friction from the users from our apps. But when you are learning you need some friction. You need to face problems and resolve them and think how to resolve them. You don't need friction on the setup probably but yes in the thinking. So rule number one the thinking stays with the learner.

Rule number two hints and questions before answers. That comes from the Socratic method. And three, you need to explain your choices in your own words. An MIT committee recommends the same for the exams. Ask the students to explain. And four, if you fail, it means not yet. You need to try again. We'll showcase each of those four uh rules in the demo that we'll check right now. Okay, we already saw speed on Ethereum I

think but we'll open the crowdfunding contract. Okay. Uh this is the website. One second. Let me show the screens better. We'll uh go through uh our um code not uh the website for now. Uh this is uh uh the code from uh agents.md is the first step we took to get AI into

skitter ethereum. So we created added all the context from the challenges in those empty files. So the agents had all the context and could explain you the questions you had along your challenge. That was the the first step to uh get the AI to help us. But then we add an interested interesting uh function into the the challenge which is which is the slash start. When you do a slash start in your agent or in your IDE it will start explaining the the concepts to you. It will ask you questions. You will have to explain uh

your those those answers and you will go through the challenge instead of just writing code and uh checking other examples and stuff. You will follow whatever your agents uh tell you to do. You will complete your code. He will re uh review it and uh you will get hints to get the final right answer. He will also hint you how to set up your local environment. And when you are ready, you will start getting questions from here. Then I then I will show you real quick

how it works on the website. We added uh an AI tutor also here because we found out that a lot of people were coming to speed Ethereum with no prior experience in development. So they had troubles setting the the environment. They had troubles starting with the challenge. So the same agent that is in the in the IDE or in the in cloud uh you find him here but only guides you through the concepts and whenever you are ready you can uh set up your uh local environment and this agent will help you to get through it.

Now let me show you one sec. Okay. Uh this is the learning lab. It's the last tool we build on education. We realized that as I said a lot of people was coming to speedrun Ethereum not uh prepared yet to uh like have their IDE their setup their everything you have to you need to have to to start coding. So we decided to create like a a previous step to to this. So everyone getting to the learning lab can start uh learning

the Ethereum concepts. We created three labs to start with it. You can go from the concepts to uh configure your first wallet and sending your first transaction to creating your first contract. But here everything lives in the browser like the coding the setup the EVM everything is in the browser. So you don't need any local setup to start uh creating code and creating your your first contract. The labs consist on different cards.

Okay, great. Uh labs consist on different concept uh contracts. Uh you have concepts then you will have for example interactive experience like this one here. You will have an interactive experience to realize how secure the private keys are. I think the internet is a bit slow. I don't know if yeah will load. So with this kind of interactive cards, you play with some information with some uh interactions to get the concepts more easily. For example, if you try to crack uh one character private key, you realize

it takes just one second to to crack it. If you try to crack a two character private key, it will take a bit longer, but not much. Let's see. Maybe I don't know. It says 16 seconds. We'll see. Okay, it took I don't know but 93 guesses. But if you try to crack a full length private key, you realize it's almost impossible to crack. We will see how many billions of years it will take a computer to crack it. But let's just give up and we'll see that yeah, you need

I don't know how many times the uh lifespan of the universe to crack a private key on Ethereum. Then we have a contract you can build. Here is the same or a similar crowdfunding contract that we saw uh in uh in Speedo Ethereum. But here you uh start with concepts. You uh then have some questions that the AI will uh will assess and tell you if you are right or not. and help you get uh to the to the answer, but we'll always give

you just hints, not the full answer. So, if we I will go quickly. For example, here you have to write this refund function that I saw in the screenshot at the start of the talk. And if we let me copy some wrong code to paste here, we will see how the agent well the AI will assess it.

Okay, some line here is uh really wrong and will end up in in a hack. And the test that are not AI, they are regular tests. They assess the code, they don't pass. So here we see that it's sending the ETH before it clears the the balance of the of the sender. So, this will end up in the in the Dow hack. So, let me fix it. Let's move the Let me If I can see it.

Let's move this line to the correct place. And everything works now. And if we want, we could deploy the contract in this browser EBM. And here, for example, it will ask us what the problem was with this code we sip before. And we will have to write our own answer in our words and it will assess it, help us

explain whatever is missing or whatever we got wrong and we will continue until we master this knowledge. Okay. So, did you spot all the four uh pillars or the four rules we we talked about before? Like the test rate the code, not the AI like those unit test we created are uh going to to assess the code if it's right or not. You have to

explain it yourself with your own words. And if you fail, it just means not yet. You have to continue until you master the concept to move to the next concept and uh continue your learning path. And that's more or less what Bloom uh saw working 40 years ago. A tutor who has questions and mastering questions, mastering concepts before moving to the next concept. So just go try the learning lab. It's really new. We launched it in in uh October. We would love to receive some feedback from you and continue uh iterating on it to make it like the next

great uh education platform in on Ethereum. So that's it. Thank you very much. >> All right. Thank you, Pablo. So then time for questions. I can see three questions here. So uh first of all you said you launched this in October. Do you have any stats of how many people have been using this so far? >> Uh not many. Uh we realize uh the the people looking for education is uh is reducing. We are seeing a less number of people. uh our uh speed ethereum already still gets uh decent traffic but we just

uh add like in speed in Ethereum we just add the link to uh the learning lab so we are still uh building this this traffic in the new platform >> cool uh then there's the question uh learning using AI is really helpful only issue I find is getting into a rabbit hole how to how to be more like selective and uh kind of avoid getting into the rabbit hole. Do you have any tips for that? Yeah, it depends if you already know uh like what you want to to learn and you know the topic and you can ask the AI like also all the main uh labs like

cloach and stuff have like an education a learning mode that you can activate and uh it will it will build you materials to uh help you to get through your lessons. But yeah, you need to to have the topic that you you want to uh to dig in. >> I I think by rabbit hole the person asking this is probably like referring to the fact that you get an endless stream of like information you can do whatever. Do you have any like technique? How do you kind of like scope yourself or limit yourself that you don't do too much or go too deep?

>> When to stop? >> Yeah, I know. I know. Might be hard. I I don't know which uh what to recommend to you but yeah just maybe like build the materials before starting to to check uh them and once you have the materials like I like for example personally interactive stuff and yeah building whatever you want to learn letting the agent like get the best resources and and yeah scan for example the deep research searchs uh work really well. It goes through hundreds of websites and just grabs whatever he thinks is the the

best and most effective uh uh sources for you. So, >> right. So, maybe write a plan first. >> Yeah. Yeah. >> And then stick to the plan. That maybe helps avoiding the rabbit hole. Uh then the next question uh except for except from solidity coding, what is the developer tool or smart contract skill that stands out for a junior developer? >> Yeah. Well, I guess of course uh uh more broad uh um like developer language like

I don't know whatever JavaScript uh whatever react but uh I don't know I guess overall those days especially overall uh security not just solidity security overall security and good practices on security will help all of us like avoid or try to avoid the recent uh grow of uh incidents like all the hacks, all the everything all it's all about security, not only solidity like yeah know how to hold your private keys. Never set them up on your code. Be careful where you set those like in your environment files. never

commit them like juniors especially back in the days uh had a lot of problems compromising their private keys and losing their funds. So I think that's still a a rule that they need to to to get security knowledge and how to store the the private stuff. Uh right >> so the specific skill is like go deep on security. >> Yeah. Yeah. >> Right. Uh then somebody's asking how to improve cognitive ability. What should we learn apart from tech to test to improve cognitive ability? How to automate with speed and scale with our own thinking?

>> Yes. So I would recommend uh like master the multitasking those days that you can have like 10 different terminals uh working on 10 different stuff. uh mastering this multitasking. I don't know what to recommend to you. For example, playing uh video games uh strategic uh realtime video games uh would be a a good excuse to to increase this multitasking abilities. But I think multitasking might be a a good skill to have.

>> Right. Cool. All right. Thanks, Pablo. >> [music]