Building Gensyn with Harry Grieve and Ben Fielding

Transcript

[Music] super excited to have Harry and Ben here we've been working together for uh a little over a year and they are I think just a a really Dynamic interesting Innovative team and so we'll get into not only what they're what they're building but how they work together uh and of course is always open it up to all of your questions um so just to to get through their formal bio so Harry and Ben are the co-founders of Jensen which is building a decentralized machine learning compute protocol that acts as the rails upon which AI models can train on any compute device in the world and we'll talk a lot

more about what that means Harry previously led the data research team at an applied machine learning startup and holds a postgraduate degree from Brown with a focus on econometrics Ben previously co-founded D data privacy and startup and holds a PhD in neural architecture search for deep learning and computer vision so let's maybe start with uh like how did you guys meet each other and decide to start Jensen together yeah so it started uh on entrepreneur first which is maybe a dirty word here because it's another accelerator but um we uh we met in 2020 and uh from my side I'd been working leading that data research team for

approximately 3 years solving kind of large geospatial data problems where we were predicting disaster events for insurance companies um around the time we were hemorrhaging money on compute uh to to train these models and two it was just after a gpt2 had just been released so you could kind of see the sounds shifting beneath our feet you know the model parameter density was increasing the costs were spiraling and it felt like a good time to kind of you know start a company turned out to be the jaws of Co which was an interesting point so Ben and I met on entrepreneur first which for anyone who doesn't know it's basically build as sort of a combination of Shark Tank meets love Island so you join and you um you know

as a person with the idea of finding a co-founder and they they kind of pre-select people to join the Excel writer um and yeah that's where I met Ben yeah and my I was coming out of uh my previous startup which I've been running for a couple of years uh as Jason said in the data privacy space um so we were a very ideological um startup focused on giving people control of their data it was kind of it we started around when gdpr kicked off um and then learned that building a startup based on regulation like that is a very tricky thing to do um and puts you at the whims of regulators uh which is tough um and we also just generally kind of made a lot of the startup mistakes so I kind of speedran the early startup Journey

realizing that when you pitch invest tors and they give you information that's through a certain lens um and we made uh a lot of decisions based on that that now I kind of learn that actually potentially those decisions were taking us down a path that that wasn't optimal for the business which ultimately led to having multiple products uh in a B2B space for uh managing gdpr Administration and nothing to do with the ideological people taking back control of their data which as a Founder is a horrible position to be in because then you think well I could grow this product that I don't really care about uh but can't actually achieve that kind of vision I had anymore um so took the difficult decision at that point to shut that startup down we had funding we had offers to kind of have more funding uh

but it wasn't leading in the direction I wanted um an entrepreneur first represented a way of basically resetting uh going back to kind of what I knew which was machine learning I'd finished my PhD in uh in deep learning a year before uh and basically look at the space of possibilities with somebody new and and figure out what I could do and then that's exactly what happened with Harry so joined entrepreneur first like Harry said got there on the first day they said okay welcome to the inperson part of entrepreneur first and also say goodbye to it because you're all going to go home and not see each other again in person um and then uh we we did that and I think we we skipped a lot of the sort of Love Island aspects because we teamed up very early um and then just

essentially work together from the the start uh in probably 14-hour Zoom calls from our respective Flats in London periodically getting walks along the temps uh but not as often as we'd have liked yeah and I'd say the the kind of three things which we unified over in those early days were one we wanted to build something enormous in the machine learning infrastructure space that's kind of always been consistent the second one was we had both just finished reading Edward snowden's autobiography and we felt very aligned with some of the cipher Punk principles and that this you know similar to like Ben's previous starts off but also these ideas around like the ongoing fights R and encryption Etc bonded over kind of our support of like the Torah foundation and stuff like that uh and then fly finally being from

the north of the UK both of us we just had a similar sense of humor and I think like the single like largest like piece of advice I would like give like any new Founders like if you can't laugh with your co-founder it's like lights out eventually because situation like I think the biggest one for us was when like Signature Bank collapsed like last year and we lost all our money for like 24 hours and if you if you can't like find like humor in it you're you're uh it's just not great it breaks the tough discussions as well cuz like you as a founding team you are going to have um discussions that you both feel really passionate about and if you're not able to kind of like just like collectively look in the mirror and

be like okay yeah we're going to have this really passionate discussion but at the same time like it's there's a bit of humor in it like we're not going to take it fully to heart and end up kind of taking something super personally so I think there's there's a lot of discussions we'll get into and and we'll probably talk about this a bit later but Harry and I do have pretty robust discussions we're like quite proud of that it's a a good way of of working in our opinion but there'll be moments in those discussions where we'll both just like burst out laughing cuz like if you don't you just end up getting way too attached um and I think that it's quite nice to like when that moment happens basically because it just cuts any like tension that might have been there if there was tension basically amazing uh

so talk a little bit I mean so so what what is Jensen why does it need to exist uh what what what's it going to be tomorrow and then what what do you think it turns into two three five years from now yes there's probably two ways of view or like reasonably simply two ways of viewing Jensen one is uh much more immediate and it's a kind of backward looking way of saying that the market for machine learning compute so the devices that actually do the computations behind machine learning uh is a highly constrained Market uh access to chips is just limited right now and it leads to um incumbents people like AWS Google Cloud a your having a strangle hold on top of that market and being able to charge essentially whatever they want uh and then it stifles innovation you get all these

startups who think hey there's something really cool I could do in machine learning but I can't get access to the chips to do it and it's a problem we'd both felt really strongly so I felt it in my PhD I could only buy four gpus and put them under my desk um but I was competing with like Google research who would have thousands of them and run these huge tasks I knew exactly how to do those tasks but I couldn't access the compute to would be able to publish papers in it which was frustrating but also when you kind of zoom out on the world you think well there must be other people in that position and they can't do these cool new things it's only one entity who can and that's not a good state of the world Harry had experienced it um with uh in the kind of industrial

context actually accessing like being able to pay for the compute but not being able to get it I couldn't pay for it Harry couldn't even get it with the money there in the funding that they had so it was clearly a constrained problem uh that went just beyond kind of cost um we solved that problem by just creating a completely open market over it so we build open source software that anyone with a GPU or TPU can run and they can join This Global Network where they can rent out access to that device um it removes those kind of limitations of if I want to become a compute provider I have to basically become AWS in order to do it and says let's replace all of the kind of human world things that AWS has to do account management infrastructure um things like that with technology and

crucially replace the trust part so with AWS you have this kind of trusted relationship you establish legal contracts with them with slas that's a very expensive thing to do in general for the world it requires people to do things Jensen replaces all of that with tech so cryptographic probabilistic game theoretic proofs that the task that you've sent to a device has actually been done correctly and then you can have settlement of the relationship of supplier and demand side participant without ever having a human do anything in that transaction that moves us on to the kind of forward-looking second way of viewing Jensen which is we think the machine learning world is moving towards uh much more automation models themselves will be doing those interactions not people uh and when

models do those interactions they can't rely on a person or a physical legal contract or a court system they have to rely on technology and and uh computations themselves to settle the transactions so we view the world relying more and more and more heavily on machine learning technology essentially uh as it is now with imperative code being replaced with models in basically every aspect but as that happens those models need agency in the in the world they need to be able to interact and the first thing they need agency over is their own resources and compute is the most constrained of those resources and we unlock that with this technology um I don't know if you'd add modify anything on that Harry yeah I I could kind of present it just in like three priors that we have which informed Jensen the first is that the machine to

machine economy which is going to emerge in the next 10 years will eclipse the kind of human economy so just agents working with each other you know you know transacting value with each other um we have a kind of human to machine economy emerging just now but the machine to machine one what will eclipse eclipse everything the second point is the notion of kind of computational Liberty like this idea that you know in the next say five years when we have consumer brain machine interfaces which are noninvasive and you know your for the first time ever your kind of biological neurons are interfacing with artificial neurons in a way which you know you don't really have much control over because it could be closed Source it's very important that people have the ability to train models which go in those devices and kind of read your

thoughts and allow you to interpret the world differently in a way which is um aligned with your values and you don't get that if someone has the ability to turn off a kill switch somewhere and the third and final point is just this notion that you know the substrate on which machines are going to train and kind of exist it has to just be permissionless it can't have these kind of real world tie-ins which is why many of the verification systems and many of the kind of web 2 approaches so far don't work for this new economy with machines they require a human in the loop or they require some type of trust which fails that scale and um the majority of research to date been on solving that final Point around trust verification and and Harry you you have a personal interest in uh in censorship

resistance talk talk a little bit about that where it came from and and and I think you you partially answered this but how that ties into what you're what you're building with Jensen yeah I think if you this is like you know the trigger War quite like philosophical but like um if you if you if you care about um if you care about the idea of Liberty so the idea that the rights kind of start with the individual and you know you give away rights willingly to to other people um that you you're probably on at least in the west and at least in the last kind of few centuries you're in the majority what we've kind of had with the Internet is a speedrun of kind of capture where large entities and governments have managed to lock down

entire sves of this kind of Digital Universe that we frankly spend most of our time in now so you know the ability to kind of control the scent to sort of of manufacture you know content with you know um advertising and overall just the the kind of ol oligopolistic control of of certain kind of cloud institutions have made it very very hard to really look at the the internet today and view it as as truly free um there's been push back against that it happened kind of in the late 90s with the wars around cryptography and this idea that you know the US government was trying to make um elliptic curve cryptography ammunition so you'd be like a terrorist if you were using like http guess um and then you know in the kind of late 2000s with

Satoshi when they released Bitcoin he kind of gave people their their money back in a way um we think that right now we're at another kind of critical juncture where it happened with encryption it happened with money the next big one is intelligence and the ability to kind of freely train models and to freely kind of you know interact with open source models is is coming under threat you see it in you know the US with various um executive orders from the state level and the and the federal level you see it in Europe with the AI act and you know there's some good stuff in those bills for sure they kind of provide Clarity they they you know help you know businesses kind of navigate navigate issues with consumers But ultimately they're also regulatory

capture and they're bankrolled by the kind of Usual Suspects if that doesn't work if we want a kind of open world where Liberty sits at the center of everything and that's why when we building Jens and it has kind of two key qualities which relate to this one is permissionless so anyone can access it there's no kind of kyc at the protocol level like https and two it's built with the idea that the world won't be a kind of one universal model which is sitting in kind of California somewhere owned by open Ai and kind of lobotomized in a way which is you know suitable for the people who've created it it will be a tapestry of lots of decentralized models around the world which are all open source and all

reflect the values and belief systems of the people who created them so that's going to help it interfaces yeah there's an interesting pattern match you can do there in the in the past where uh one where one area where it did work particularly well is the outcome of the protocol Wars of the late 20th century for the internet so like the tcpip stack uh does effectively as long as other things aren't kind of allowed to capture it um provide that equality of everyone like the way that our devices talk to each other is a standardized way which is completely open to people to use it's not owned by a certain company but if you look at how that developed we developed that standard and we allow these devices to communicate but then there's certain things we didn't make

into that standard and one of those crucial things is the accounting for the resources used So within communication it's a bit different from computation but you're still using bandwidth every time devices talk to each other and that accounting of bandwidth was taken by telecoms companies and those telecoms companies then run with that and they capture the kind of monetization of that core resource uh one way we see J is building the tcpip stack but for that next Kind of Revolution like Harry mentioned for intelligence rather than communication but learning from the past we say hey that resource accounting should be in the open protocol itself uh we couldn't do that before we didn't have the ability to have like open kind of financial systems before now we do it's a kind of Step Up in our ability to build a open standard uh that won't be

captured by somebody else there'll be other areas where we'll have to battle on it inevitably there'll be unknown unknowns uh but we can do a better job now with new technology which is cryptocurrency and blockchain essentially so it's kind of stepping up our ability to build fully Open Standards that allow devices to communicate and then eventually allow machines to communicate and create a uh an entire economy and and talk a little bit about uh I mean you you very explicitly chose to focus first on uh training rather than inference and I think conventional wisdom is that inference is just a much bigger market right because it it needs to be done every time someone wants output for a model but talk about uh why you chose to go that direction I think the the kind of core reasoning was freeold so the first part was typically

trainings compute constrainted as opposed to latency or bandwidth constraints so if you're building a decentralized system it makes sense to you know trade in favor of compute um it's not you know strictly true like you can have decentralized inference protocols that that make sense but it's just you know more hand in glove on the training side the second kind of more consequential point is when you get back to kind of philosoph iCal reasoning that that we had the training accounts for so much more of the perception you know like it's the way the models interpreting the world is determined by the training the idea that you know you offload all the pre-training to a few companies just felt crazy to us so it was it was starting there um and thirdly and this is probably like maybe even

more relevant it was a problem that we had so like we came from this world of training models it was why it was costing so much and I think you know inference has definitely stepped up a nously recently um but we had we felt the problems ourselves so back when we kind of co-founded in 2020 and kind of before you know chat GPT and everyone's kind of talking about it in the video stock was going for the roof uh we were like yeah training is the problem um I don't know if you would yeah I'd maybe offer a different lens in the same way as the like backwards and forwards looking kind of two-piece thing again like in the backwards looking way I think training is just a uh an easier problem to bootstrap a network on um cuz like har said training is asynchronous

it's heavily compute dependent um it it's a better place to start in terms of providing value to somebody if you have another kind of device somewhere on the planet um for that person to use inference is latency dependent like the kind of trend we see with inferences getting it as close to the user as you possibly can which a decentralized network could be effective for that because you can incentivize kind of like Edge inference and things you can incentivize people getting models close to the user but its value is dependent on the size of theor Network which is meu's law um having Met's law be a bootstrapping problem for a blockchain network is a really horrible place to be because it essentially means you have to bootstrap the entire network before you realize the value of that Network it's

thing things like helium had this exact problem when they were doing their first use case and it means you have to pour incentives into the supply side with the hope that the demand side will just pick up within the time window you have to create incentives a horrible place to be if you can bootstrap the network some other way if there's another use for the network IE a CDN and then you can swap in the inference that could potentially work but that's the way we see that market going so centralized CDN providers are in a much better position to create an inference Network than somebody who's trying to build one from scratch whereas training if you have a single device somewhere in the world that's useful to somebody somebody can send a model over and have it trained that's a kind of good value transaction

uh and then as you increase the number of devices that value obviously goes up but it's a a more linear relationship than the other one uh so that's the kind of backwards looking way the forwards looking way is saying well training and influence in our view are just going to merge uh if you look at the way and this is kind of another analogy to the internet the internet is this huge shared knowledge base that emerged over our ability to communicate between devices from those early standards that we mentioned before um the intelligence Revolution that we're going through right now in many ways just represents a richer interaction with that knowledge base it gives us ways to do it that we couldn't do before we can distill information into it much more effectively because models can compress images and text and audio everything into this shared embedding space that's

really exciting but in order to interact with that big shared knowledge base you have to go via models and we think essentially we're in the kind of early internet there's just a single server and I'm reading information from it kind of phase right now where it feels like the read activity is the most important one but very quickly that gave way to us creating content on the internet as well so when we started with client server I just read websites and that's it and then moved to oh social media now I'm creating the content and I'm up dating it constantly we see the same thing happening with machine learning where inference is the read but we also have create and update and delete and those things will need to exist and arguably they will overtake the read portion because we'll be creating that shared knowledge base together we'll just do it

via machines rather than directly and that's why you need this protocol to allow the machines to do it on our behalf for for any startup it's important to think deeply about who your first best customer is and I think you know particularly in a market that is as new as fast moving as complicated as what you guys are doing where there's like you know lots of big players uh and and a ton of centralized startups who all want a piece of it talk about what that customer looks like for Jensen and and how you came to realize that that was a a place to start I say there's there's kind of two two points to this so the first is the just as a preliminary the customer that we don't want are the kind of liquidity mining people who kind of come into networks

and they sort of try and suck value out then we see that with some I protocols just now in terms of the the ideal customer it would be someone who's a machine learning engineer or who's training um sort of 7 billion and up models who is Cash constrained number one or scale or scale constrainted so either they want to train larger models which they can't currently afford to train um because just economically unable to or like I was in in my previous role there constraint uh this Maps typically to like early very early stage startups where Iver are trying to do one of two things so the key use cases they're

either trying to fine-tune models with for a certain use case so that they can include it in their product and their constrained um on price or they're trying to build or pre-train larger models with new kind of decentralized architectures but they're unable to get the scale required um on the second point we do a of work in the kind of model architecture space itself to work out how you can retrofit larger model architectures in such a way that they work across decentralized Hardware our feis is that basically you can take models um by doing asynchronous training and sharding across lots of devices you can beat state-of-the-art results provided that the unit economic cost of

the device is sufficiently low because that means you can just run it for longer you know you can get more of them um so yeah it's cnly machine learning Engineers who are either skill constrained or cost constrained and who are typically in sort of very early stage kind of seed startups I i' just um emphasize a point that that Harry made around like the early like the liquidity mining users um similar kind of vein an anti- pattern for an early user is somebody who is highly ideologically aligned with what you're doing but not necessarily actually value aligned you see this a lot in crypto projects where getting crypto native users is much easier so you think hey I'll go after those users first because they're the

easiest to acquire and then I'll kind of expand out from there but there's a classic trap there where you end up building a product for those users and those users don't necessarily actually care about the thing that your product does they care about the fact that it exists and the signals they will give you won't be the same signals that you'll get from uh the real customers so in our example we could go to um people within the web 3 ecosystem who do a bit of machine learning and we could say Hey you can do machine learning and decentralized way now but the signals they'll give us won't necessarily help us going to Pure web2 ml users and saying hey we've got something valuable for you now because they see the kind of decentralization aspects and they're like I don't care about that what I care

about is the price or what I care about is the scale I'm I'm able to achieve so we take a kind of difficult decision to say even though those users are much more accessible we can't index to them we have to find the um more typical representative users of our larger kind of group you can get them as well but focus on the users that you expect to have in the future and you actually want to capture because they'll be the people whose feedback you use to grow your product it putes up the relatively gnarly intersection of web 2 and web 3 where there are people in the machine learning world like which Ben and I used to inabit because we come from web to who are completely crypto F phobic like they think about it

and they just think of like SPF or like Lamborghinis and like it's just like a um it's a changing thing from like the front lines of you know machine learning research conferences for example there's increasing kind of development of decentralized training Frameworks so the Paco from Deep Mind a few weeks ago was like the most recent kind of step forward in that direction so there is this kind of emergent sort of it doesn't use the word crypto but for all intents and purposes it's so aligned as to be you know hand and glove um of decentralized machine learning so it it is changing but yeah it's definitely a much harder sell and this is wely true for hiring as well like trying to hire

people out of these you know web to machine learning um shops into into Jensen you have to kind of you have to almost teach them about like tokenomics and stuff like that which is not an enviable position to to be in and actually perfect segue into what I wanted to talk about next which is uh and and you can hear this in both your answers but you guys have like a very unique way of organizing and running the company right so you're about 15 people uh total today uh if I'm if I'm remembering correctly nobody officially reports to either one of you uh you you truly co-lead and co-run the company uh and um and there there's no other people managers uh in the company other than

the two of you talk about uh I think one how's it how's it going uh why did you choose to go that way and uh and how's it changing you continue to scale I I think it's based ultimately on the like the principles that we share like the place that that we started like Harry mentioned uh that kind of like individual liberty piece but then with that comes a few things and we've codified those within Jensen into three areas of autonomy focus and rejection of mediocrity and those are those are kind of like nice Snippets of describing the three buckets but we spent a lot of time thinking about what they actually were but ultimately it's this drive to empower the indiv ual as much as possible um and that's the way we've

always operated so like you said we don't kind of split our roles we don't explicitly say that one of us does one thing and the other doesn't do that thing we prefer the sort of emergent um coordination that comes from just us having constant um robust feedback between the two of us and being able to navigate situations in that way uh and it's worked fantastically well for us as a co-founding team I think it it wouldn't work for everyone it only works particularly because we have such similar backgrounds and such similar skill sets we've both done highly technical things highly commercial things we can both essentially sit at any point in the business and trust each other to be able to just navigate that and deal with it in a similar way to what the other person would um and we've broadly up to now kind of extrapolated

that out over the the business and said okay um we will expect that to work in exactly the same way as we expand the team and as as everything grows um when you do that though there are certain things that don't work at larger scale that would work between two people and there are also certain things that we realize that maybe that works for us but maybe that's not a kind of universal um guarantee it works for us in a more unique way than we'd maybe expected um so we're we're going through this process now of thinking about how do we stay true to those principles but also allow the team to grow and don't kind of hamper it with uncertainty because there could be a lot of uncertainty in the way that we operate like we don't have those rigid structures we're okay with that

but maybe not everybody is and maybe when you have 15 people instead of just two that's even harder to navigate so um it's something we're going through at the moment we are bringing in a bit more structure and we're thinking a lot like the the two of us and with the team about what that structure should look like and which gaps we need to fill staying true to our principles and which ones actually the emergence of those is acceptable it does work um and I think just being very deliberate about that should allow us and allow any company who's doing it to experience the benefits of that kind of free open way of of working where anybody can just kind of take control of something and drive it forward it doesn't have to be that one person that you put in that position because that's a very fragile

way of working if you didn't pick that person right you're a bit screwed at that point because everybody else has just left it to them whereas that if you let people figure that out themselves you end up with if you go too far I guess a bit darwinian but you end up with the kind of best ideal wins sort of um environment and we think very early for a company that's far more performant than having that rigid structure but it definitely changes through time um I don't know if you'd modify any of that and maybe it'd be good to talk a bit about the the structure we're talking about we're the rubber hiter roads in the engineering team basically we have obviously like a highly technical product it broadly um quarters into compiler design and Engineering distributed machine learning the like blockchain specific

architecture of of the node and then what we're doing on the on the verification side so the the cryptography uh um those four things are they're not like oroginal to each other but they're quite distinct for someone to kind of emerge in that team with enough context over each area and also have the requisite you know ability to be a world class I see in that area and to also be a you know engineering leader and to share the vision and to kind of you know you just keep adding these things on and you eventually there's nobody nobody in in the [ __ ] world who can do this so um we made a decision to bring in a VP of engineering um with the idea that that person has a not

necessarily a kind of hierarchical um command and control style relationship to the team but they have a very explicit mandate in a way that other people don't so typically when we hire people don't really care about titles that much we just you know hired as a distribute systems engineer theoretically you could work anywhere in there and you're incentivized to um but for this person it's it's relatively static so it's you will focus on you know making sure that the team performance is high making sure that everybody's coordinated and helping kind of compress a lot of the vision that we've talked about today into the engineering team such that it's aligned um with the both of us um and that's been a big step for us because it's the first kind of middle-ish explicit layer

that's been put into the company and it's a bit of a departure from the kind of heady days of uh yeah that we had previously uh so last last question and then we'll uh we'll open up for questions but so one one thing that uh I see a lot on our calls and that's always impressed me is that um there there are many times when uh there there's a a company critical issue that you're both wrestling with you don't uh always at least at the outset uh both have the same perspective or agree on on either what the issue is or the right path forward but through deep questioning and conversation you're able to often uh form a

perspective that that actually is almost never where either one of you started but is like a fusion of both of your best ideas talk about like I mean was was that something that has always been part of your relationship from day one was it something you had to work on together like how did that how did that come to exist because I think it's quite powerful and you don't see it that often and I think it it's both of those things it's always been there from day one but it is something we have to constantly work to maintain um at at the end of the day it's I think it's down to a absolutely kind of like unwavering belief in the ascent of incentive alignment between the two of us like I 100% trust that Harry's incentives are to grow Jensen as are mine and so the it's kind of like from

both of our sides we think I might think I might have this position that's not necessarily correct Harry has this different position the outcome of the merging of those two positions will be the optimal for in so we're then uh willing to put in uh enormous amounts of kind of effort to make sure we get to that outcome rather than breaking it and saying okay we'll do that thing but actually I think it should be the other thing we root out the definite kind of end State shared position because it should be the best one um I think it in some ways it is expensive like it's expensive on our time and it's something we've realized when we were in person together we had an office for about a year before we gave it up and went back fully remote again uh and we spent loads

of time in the office just talking together about uh issues we were facing but also philosophical issues cuz it figur it let us figure out where we both stand on certain things and to understand the other's position and the likely position in the future uh as we then went remote again uh we had to figure out how to still have that and that's been something that we continually work on how do we make sure the atomic information flow between the two of us is as high as it can possibly be in order to keep that um communication kind of Channel open and allow us to make those Collective decisions the end state of it um I think at least is that we now have this ability to act independently in the vast majority of situations and Trust the other to make a decision that is aligned

with the center and then in those Mission critical situations that you mentioned Jason we come together and we can very quickly now uh maybe not earlier on get to the shared position and move forward so it allows us to be very independent but also act together as and when we need to and each person has an idea of when that is so we'll know hey in most situations in this situation I can just act it'll be aligned with Harry that's fine but in this situation actually I know he's going to have a different position I don't know exactly what that's going to be but I don't want to just act on my position I want to act on our shared position because I think that'll be optimal um so that's that's the way I view it it's down to that incentive alignment I think I I totally agree I

just actually give one bit of tactical advice which is advice you gave us um because we were we're in very different time zones so Ben's um California based I'm London based uh we basically bookend the whole company because we're split between the US and Europe which I would actually Rec recommend just like generally if you're hiring founder at either end it means that the center of gravity doesn't shift to where the founders are um but obviously the time zones are ridiculous so we um we have a shared uh private slack Channel which we only put the kind of I guess well we have two actually we have one where we only put the kind of mission critical thing so you know if there's some big decision which is coming up or some state that we want to push to the other person we put it in there we have a

separate one for team as well so when we do the kind of onet to on with a team or if we're seeing any performance issues we put that in there and the consequence of that is basically our normal channel can be kind of fun games and like you know going to keep the sort of like you know banter and everything going without having to drop a kind of nuclear warhead in the middle of it like every sort of like hour um so yeah that's just a good tactical thing if you find yourself uh working in a very asynchronous way there's a I think the state update piece is probably quite interesting to people as well because we explicitly have like different ways of communicating between the two of us um where every so often we will explicit L communicate this is a state update for me I'm going to tell

you some information I'm going to tell you that I have changed my internal position on this because a lot of the time you'll be telling somebody information and that person will listen to it and they'll think okay sure some some external inputs come in but I don't know where you stand on it and a lot of time you actually haven't changed your position you're just saying hey there's more external information you need to know about it but now we're very very explicit where there's been some external information and explicitly I used to be here and I am now here and so that lets us in the future situations that might coincide side with that to think okay I know how he changed his position on that I have to talk to him where before knowing his position I'd be like okay I can just act so we kind of

very explicitly have this like separation of this is a state update versus this is just some new information that you probably need to know but like take it however you need to Fantastic good advice all right well with that let's uh open it up to uh to the room how do you feel about economic e economies of scale uh so like someone with a single GPU at home might not achieve the same efficiency um or power consumption um cost as as like a large um center with which hosts a bunch of gpus and that's kind of similar in spirit to how Satoshi was imagining the original one one CPU one vote right um I guess the mission is a little different because the mission

would that I assume your mission is not once one GPU one vote but rather decentralizing and permissionless training but that if you take this sort of extrapolate this this could sort of lead to concentration of power because these large centers can actually out compete the individual players right so you actually end up with a more centralized Solutions yeah I'd say kind of there's like a macro and a micro response to that the macro is that's the default already like in web 2 right so like open AI is that thing and it can't be worse than that so it's it's always like a step back when we're prac ially on the micro level I think there always will be kind of petal style relationships to

things in in decentralized networks but I would say that because the network is permissionless the idea is that at least not at the kind of staking level which I guess is like a different question but at the actual Market order level if you're trying to you know purchase flops on a device you're able to just do that with every anybody on the network which keeps it open it keeps it free it keeps it kind of ated um I also think that there's an argument that what when the scaling continues into kind of gp5 and it everything keeps going up like an order of magnitude both at the data level and at the compute level there will be even more demand for people to train very large decentralized models

which in turn will make an individual with kind of a GPU more economically important so the idea there is that basically there'll be more and more demand for Solutions like this they'll increasingly need to be more and more decentralized and activate the longtail so there's that kind of a push in that direction coming from just the economics of model training we're not kind of in pursuit of a perfect solution it's not like we're trying to present something that perfectly solves all these problems I think the idea from our side is like rely the current system is skewed for lots of reasons that aren't necessarily emergent from like a free kind of Market there are reasons that people have created for themselves in order to have pocket markets that kind of control

Power and ability within that market we'd like to move to a system that is much much more accessible and much more free the emergence system from that like you said in some ways may be still constrained in terms of power it may Trend in that direction but that's a much more natural Direction than the current one that's been created specifically to have those power structures so it's not saying that like creating a free market over this just is like a Panacea it's saying that creating a free market over this is inevitably better than the current situation that we we have on the uh economies of scale piece and they kind of if you look at what happened with like Bitcoin Bitcoin became this almost like Energy Efficiency mechanism for the planet we'd like to see that happen for AI comput as well where it does push um the the

performance and the the creation of these compute um data centers to areas of lower electricity those will move around people will see opportunities within that if somebody can build some green um very cheap electricity Source then they'll be able to build a data center there and they'll be able to actually kind of realize the benefits of that maybe that Trends upwards but it constantly changes so every discover new things it can change right now it can't change it's very very difficult for somebody to come along and say hey I've got this super cheap source of electricity I could build a data center here but I have to become AWS before I can do that um so yes it will end up in Pockets but those Pockets will break and reemerge and reform and it's this system that constantly evolves rather than just being stuck where it currently is I have

more of an engineering side of uh question so are used to work in machine learning and from what I understand correctly you need to run very specific models on uh V chenson right because if I think of a very large model um I think you said latency is not important but in my experience latency was everything like as soon as I need to split the model over 100 gpus as soon as they're not communicating over NV link um I wouldn't see how I could possibly uh train anything efficiently so would I need to come up with specific model architectures that don't have communication on the batch level or how does that work yeah in the in the current way of training

models that's true um we see two solutions to that um and we work on both of them so it's not like we're saying hey yeah you have to come and build an entirely new model we're designing the protocol to be more performant for those situations but it'll be a bit of both there will also be a drive from the ecosystem to build new models for this um the two different ways are one just treating uh the Jensen Network as a big problem to optimize when you have an existing model so you have this model which currently operates over 50,000 gpus say and it needs those interc connects it expects homogeneous latency bandwidth and devices uh if you then throw that model over a network with heterogeneous latency bandwidth and devices you create an optimization problem for yourself so where are you moving the compute where are you doing

more communication and more compute that is just one big problem to solve uh and we can do a lot of work in order to get closer to state-of-the-art and doing that you'll never get better than a centralized cluster but you can get much closer than most people currently think um so that's a maybe shorter term approach because it takes those existing models and just optimizes them the longer term approach is entirely new model architectures in our view but if you look at the way the architectures as we currently have have progressed they've not been built for very long like we've not been doing these like distributed training runs um for very many years it's not necessarily the only way to do it but I think the machine learning space gets this like really short memory where it thinks that just because we've had uh success there

that's the only way to do it and we should just continue down that path actually we're seeing the scaling limits of physical data centers now not from a cost perspective but from a geographical perspective like how can you build a bigger data center somewhere on the planet that you can cool that you can put 100,000 or like a million gpus in that becomes a problem that's insurmountable at some point and so we say instead of increasing that arms race and carrying on trying to build bigger data centers let's shortcut it and say Hey you now have access to every single Compu device in the world but yes it's heterogeneous later see bandwidth so we'll have to think of a new way of building models between it but we're more bullish on that than just continuing down this path of trying to build a bigger data center if you want

to uh read more about that uh Delco depaco veruna and Alpa research papers cover the variations of optimizing for that uh for that problem I'm Tim I'm also building an AI aligned protocol not competition for you guys don't worry um so one thing I really love about the light paper is that you took three really important research pillars and you managed to actualize those like probabilistic proof of learning and the true bid incentive system and we've had lots of discussions about incentive alignment and Game Theory here I would just like to know like so I can follow in these footsteps the concrete steps you took from I have some cool academic papers that can inform my protocol what did you do who did you talk to what were

the steps you took to turn that into the actual I guess to to start the very kind of like beginning with pre- light paper we just discovered that solution when looking specifically for a solution to the trust problem we weren't in the crypto space we were completely outside of it we were doing verifiable computation kind of like literature review essentially and as we did that as we came to solve that problem we put those Solutions together but it was purely in service of solving that problem um which I think is the kind of purest way of doing it essentially but yeah that that was the kind of construction of the light paper after that it was figuring out what do we need to do on each of these pieces like where the and there's a a principle here that maybe we're giving it away from

entrepreneur first that they have which is run into the spikes and that's the idea that when you have an idea or a business idea whatever you have a a thing that you want to create there'll be something in that thing that worries you the most there'll be a part of it that you're like ah if this is going to fail it's going to fail for that reason uh and the running to the spikes principle is like run directly at that thing don't think about the things that you're like excited about think about the things that worry you and go after them and that's what we did with the light paper so one of the things we discovered quite quickly is that proof of La learning is actually flawed uh it doesn't work in practice it's uh flawed unless you add cryptography uh and that was something that we were uncomfortable

about we talked to the authors about it we talked to other people in the kind of research space about it because it was this Spike for us and in doing that discovered hey here's some solutions within cryptography we then chased those down and ended up at the uh interactive proof system that we have now which sits alongside it so it's just been this constant process of saying okay we're happy with this light paper construction but a few spikes on it let's go directly at them and it's how we operate now it's every single kind of decision we make is basically driven by a kind of consideration in our minds of does this is this likely to be a limitation for us is this likely going to be the reason why this thing fails if it is let's find out immediately uh and then we can go in

a different direction or let's smoke it out so that it's not actually real um so yeah I I think it's been that process of just running at spikes continually I'd add like one point which just be willing to make reasonable assumptions as well like one of the things which we struggled with maybe two years ago was the idea that you know if you don't model people as financially rational everything doesn't work SL you know if someone's just willing to like spend infinite sums of money to like break a protocol or if it's like for you know like the US government or like the CCP just want to crack like Bitcoin you know and buy up all the as6 and stuff there are ways to like you know Fark it and stuff and then kind of solve against it but you know you have to at some point

just make assumptions we had something in where we still do in our internal literature called like big G which is just like a government attack basically where you know if you're modeling something endogenously in Game Theory and you're like well you know the staking um kind of requirement versus the probability of being caught multiplied by the the slashing you know effect as such that there's you know negative expected value to cheating but what if someone paid them a trillion dollars like you you just have to like at some point con constrain the problem and I think that that's a trap that some of the best people that we've met in the space have fallen into when we try to hire them and they've just been lost in this world of how do I make it perfect if you think that open AI is asking for

7 billion for Hardware acceleration and doing that with an open- Source model decentralized way and giving data ownership to user is so valuable so my question is how you are planning to incentivize user which running model to contribute decentralize AI it's uh so essentially you know why would someone use us yeah economics is the is is the bottom line you know we there are two reasons that our solution is cheaper the first is it removes this 80% margin that Azure gcp Etc charge on their um on demand instances you know no one nobody pays that margin it's always a bit lower but you know at the end of the day it's

it's significantly cheaper 0.1 and in point 2 by activating this longtail of Supply you typically have a defl stationary um effect on on the price in general there's also the point as well that just the the kind of there's like an Arbitrage between you know say an a100 40 gab and two RTX is for trading like a if you do like a filly sharded data parallel um mral 7B fine training run with with Laura you know we we've done these tests internally it works out um significantly cheaper doing it on on the RTX is when you consider the the kind of you know asymmetry in prices so there's just this kind of economic reason and scale reason I think um that's certainly on the um on the demand

side on the supply side it's a little bit trickier um there's definitely we talked to kind of um mining teams from like the big players like Foundry Etc down to the kind of smaller um people who run like you know a shipping container full of RTX is in Serbia where there's like cheap power and stuff and it's our thesis that with the demand that we can bring from the unit economics it will pay for people some bootstrapping of the network saying hey the network itself the protocol itself needs to incentivize participants some of that is useful but it's useful in a very like contextual way it's not useful if just applied in a blanket way to say hey the protocol pays people to put gpus on and at some point it won't anymore

and the demand side will cover it you can't like build a you can't definitely build a viable Network in that way if you know that you need to beat bootstrap for a very short period of time then you can use it sparingly but our view is that the incentive system should be as simple as possible and it should be as close as possible to somebody has a need and somebody else has something that they can provide and you connect those two people together and the protocol should be minimally involved as possible in that transaction but there's a lot of talk about these complex incentive designs I think as soon as it starts getting complex you're probably actually creating more of a problem for yourself than you're actually solving one so go back to basics and just say hey am I solving that user need if I am it's

probably in the right direction hey guys uh really love the parallel that you guys Drew between um uh essentially AI as you see it currently and the early web with like their Reading Writing and guess owning and uh the comparison between reading and and that being like inference for ML um and eventually models being able to write and if you kind of you know continue that that parallel on do do you see the the models being able to own as well yes uh we think what we're we're going through you can view the kind of progression of machine learning models in a similar way that you view autonomous agents within like the research space when somebody

creates an autonomous agent right now in a pure research context they create an environment for that agent and then they create the ability for that agent to interact with that environment what we're creating with um initially starting just with the crypto space in general is an increasing sized environment that agents can actually interact with um so with smart contracts we can create Financial agents now they can act in exactly the same way we can in that constrained environment we view Jensen as giving them The crucial access to their own resources to continue to exist and that's an increase in size of their environment but that environment is going to continue increasing in size and at some point Humanity will go through this kind of weird process where we say wow these agents are kind of equivalent to us in a lot of places now

and we're going to have to go through a general sort of social governance discussion of like at what point are they human because they have as almost as many rights and abilities as we do and it's an exciting I suppose you've got the verifiable computation to be able to ensure that it's actually from the model right you know decentralized just trustless way yeah I mean that's that's a crucial part yeah if you think about that that agent in the future how can we kind of know what we're interacting with the Providence flowing through for the agent it creates the trust where humans would have different trust structures maybe we've got like attestation from a university or something that shows that we know something agents will need to have those similar things and we can build those with technology but it's not like trying to recreate what exists for humans we're

trying to create what should exist for them in a way that works for them so it's it's parallel and there will be constant parallels but it's not trying to recreate what works for humans it's trying to just increase their uh agency and their environment thank you all thank you guys [Music]