Loyal AI: Community owned, aligned and controlled

Transcript

Hello. All right. Welcome everyone to ETCC to my talk. My name is Himman Shutyagi. I'm one of the co-founders of Sentient. Sentient is building open AGI with community. I I didn't know this would be this uh mainstage talk and I didn't know it would be a 45minut talk. So we're going to end early. So you'll have a lot of time to go out right. So this is a session on agents and we have been seeing a lot on agents I think generally overall in this conference but there is this before I got into building open AGI with community. The first question which

everybody asks is what is AGI and nobody knows what is AGI and one year back two years back very few people knew what is AGI that's when we started sentient and it's a meme it's a meme like even AI is a meme but you know what AGI is very good AI it will be some AI which will blow your mind for me even the current form of AI is sort of AGI you can try to define AGI you can try to define it will be intelligent like any human being. It will work across a variety of tasks. It will be like a child who can pick up any sport if they train for it. Any uh logical thing like poker or chess, solve maths, physics, you can define it that way. But it's

never concrete. But a particular form of AGI that is very concrete in at least my head now is the way AGI will appear over the internet. In just the next two years, I believe in the next two years the internet will look very different. In the next two years, this internet will be run by mostly AI. What that means is websites will not be for humans. Websites will be mostly for AI agents to visit, get data, and take actions. In fact, bots will be paying customers. So, Cloudflare will have to have a very different business model.

Bots will be paying customers whom websites will invite and welcome and robot.txt txt will be replaced with some other files to define bots about how to how how they can come in, how they can buy the services and pay. So payment rails will be integrated there. This AGI will go and collect data from different websites on your behalf because you have asked it to do something. It will collect the data and then act on that data and uh these so this this acting on the data will be booking your hotel rooms buying your uh favorite altcoins and you know taking subsequent actions that's what this form of AGI will look like so nobody knows what AGI would be overall in 10 years 5 years either

something very crazy or maybe there'll be some disappointing ending I think something very crazy definitely robotics will be a part of But definitely in the next 2 years in the limited context of the internet very good clarity on what AGI would look like. Humans will just stop wasting their time clicking and researching over the internet. Mostly that will be done by this AGI which can reason and go around. So when we say sentient is building open AGI for concreteness for this talk you can assume this AGI over the internet which is wandering around getting information on your behalf and taking actions on your behalf. This is roughly what happens when you go to chat GPT and keep the search open and you ask a query. It

collects it collects data on your behalf from multiple sources and gives you the answer. It is not acting so far. Nobody has enabled travel bookings. Nobody has enabled that part. But those pipelines are also coming in. Even the front end is changing. The front end which HTML based front end which browsers are showing which we all uh which we all enjoyed for the last 30 or more years is also changing. Even the front end would be in a way rendered right then and there for you just for that experience that you need just that booking experience maybe not all the things that you need. So there is a lot of activity going on in building this new internet in building that layer which welcomes this bots to come and take data from the

websites to interact with the websites and then render just that piece of front end that you need in some sense. This is a culmination of a long line of activities. This is what IoT wanted to do. Interoperability getting solved through MCP and this kind of protocols. IoT tried 15 different protocols to solve this problem. It's now getting solved organically and this is what internet actually I don't think it was ever envisioned to be this way but this is what internet has become a more singular unit than it was and uh this is what we'll call AGI in this talk okay so this AGI is built on certain key primitives we all know about them and we want to keep

these primitives open so what is open ag GI we let's understand AGI is just the AGI over the internet. What is open AGI? Open AGI for me is these few things. Open AGI must have open-source models and tools. So you are you all know a lot of open source models and you also know more lot of open source libraries. So open source models for you are probably the most popular ones are llama and deepseek. Maybe you use Quen series. Quen is now an upcoming open source model a lot from China and there are many many more open source models the models which you can download and see their weights are open weight models I'm calling them open source more rigorously

people call a model open source when not only its weight but the data that was used to train it is also available actually there are very few there are there's none which is as performant as llama whose data is also freely available so in fact there's no truly open source model today which with data also available. This availability of data is a very important part to understand how the model operates because otherwise the model has lot of different hidden part. So these open source models with open data is what we want to have to keep this OGI open.

We'll speak about why do we want this AGI to be open. Then these open-source tools, different tools that people have been building this so sort of agi of the internet over uh need to be available. Some of these tools are uh already there and already open. The tools used for interoperability. MCP is an open framework. Anybody can use it and anybody can deploy it for integrating different tools into uh an AI system. Then there are other frameworks like E2B which is this sandbox which almost every application use for running a code. It's again a great open source tool which has captured a lot of attention. Then you go down the line and then the difficult

problem of running inference comes into picture. Then there are these tools like VLM which people use to actually run this LLMs at this this models in a performant manner which give you that experience that you see at consumer apps where the answer comes in a flash. Of course it turns out that there are these gaps which still exist in this tools but by and large these tools are quite open source. They are still available. There is a Linux- like business model where a stabilized version of them is made available to enterprises. But open source tools are quite available. Open source models on the other hand are not so available. What we also mean by open is open for audit for safety and

alignment. This is a very important point. This AGI which is running the internet for you which will give you your answers will buy you your air tickets will suggest universities for your kids nobody knows how it operates because it is a prediction based model which is represented by this weight and you cannot it has so many dimensions that you cannot easily audit it audit all of its behavior. So it is when I say that AGI is open, I want this model to be accessible for you to constantly audit it. And by you I mean anybody who can actually audit the model to understand its preferences to understand it biases. For instance, lot of us are building on these models from China and

which have because of regulatory constraints when they come out of China which have some alignment training which has been done on them which gives them certain beliefs. Now you may think that these beliefs are about some points which you may not worry about but in fact this model is the one which is planning your portfolio when you use an AI agent or when you will use an AI agent to design your investment strategies. Those beliefs can bias your financial returns. It's something as concrete as that. And therefore it's very important to have complete auditability and transparency about these models and this AI. That's another part another important part of being open. Open for audit and open for safety. Uh audit for alignment and audit

for safety. Safety is a bit different from alignment. Safety is a more multifaceted thing in this AI. It depends on different type of data that goes into this AI. It depends on um different channels which feed into this AI. I will give examples later about how things can become unsafe in this kind of a complicated AGI running over the internet. The last part of it and perhaps the most important part of it is that it needs to be open for both users and builders for users to get access to this AI. This is the regulatory war that we are seeing right now. The access to AI and the access to things that are used to build AI is being a part of tariff war and

different regulatory wars. This open access is extremely important to make sure that AI remains like the internet a common public good on which all the nations can build of course build businesses and prosper but it's not it cannot be locked that that's a very important part of this uh AI to grow so open access from builder side and then open access to get distribution now this is a very uh challenging thing as few players like Chad GPT gather 1 billion and monthly active users. The distribution is obviously concentrating in the winner like this. Now this winner determines how you can take out whatever you're building. This is one extreme.

It's good in a way that this kind of monopoly is coming in a way that the technology is moving very fast. But the way it is really bad is how the aspirations of an entire generation is aligned with building AI. For instance, uh in my own home, my nine-year-old do daughter expresses everything she thinks of through AI. Her slides, her apps are already, she's unlovable already. Everybody's unlovable already. So this generation which thinks of software as some means of creation just like a YouTube video, they're producing these software, where will they put it out?

Where will they get distribution for it? So this locking of this distribution in few players is a very dangerous thing. And therefore I believe we need a more open marketplace just like the internet where all these new creative AI can also go out. So we need summarize open AGI is the AGI of the internet which is built on open-source models on tools which is open for audit for safety and alignment and which allows both users to openly access it and builders to openly get distribution by participating in it.

That's the open AGI we are looking for. And what does this community want? this community who's building this open AGI this is easy this is what first part of it is innovation at the end AI is still a very research heavy area it's actually I believe a the fastest research to product area farm tot research to product area even open AI is still churning out things which are very experimental in production right away and it is from day one has been innovationheavy we are very excited to discover this transformer architecture but it's hard to believe that is just 8 years old and every so much money has been poured into it and there are many more architecture which are coming

mostly hybrid of transformers because we are logged in but a lot of innovation will happen if researchers get these artifacts in open and they get to try new models and new agents over it. So this community of researchers wants to innovate. Of course the community of builders like uh you I we want to monetize what we build. Whatever we take out we want to get money out of it. This will become harder and harder uh as distribution is captured by few players and as there is a very interesting path the path taken by the super intelligence team where you work for a big player maybe contribute in open source build a reputation around it and then you get hired by another big player and get $10 million per year. This is one path which

we saw happen. But is this the only path? Will every artifact sitting on hugging face have no part to monetization? This is what we want to challenge. We want to make sure that all the different open source artifacts can be monetized and there is a good path for them to monetize because these this is a new economy we are entering where people will be constantly creating software for supporting this AGI and they should be able to monetize it. And finally, this community wants safety and freedom. I call it freedom because this AI can become actually very vicious. It can become your it can become the intermediary between you and the internet. And just like an influencer can influence you to vote and influence

you in a very complex way that your personal life is aligned with it. This AI will now it it already is but this will hypercharge. It will influence you in a way you don't understand. It will influence how you interact with your friends, your family. And the freedom here means that you have the choice to not be logged into one belief or one AI which has taken over the internet. And that's what I mean freedom here. Safety is clear. If you are delegating your payments, your investments, your planning all to this AGI, it better be safe and it better be aligned with your interest and doesn't try to rugpool you.

So the community wants to innovate, the researchers want to innovate, the builders want to monetize what they have built, not just putting it on hugging face but actually building a company outside it uh and much more numbers than what we can do today. And then we want safety and freedom. Okay, these were the goals. So this is what sentient is doing. Sentient is building open agi with community. What that means is there are three parts to this. First is this this this collection of different models agents tools roughly everything that can be on hugging face is an input to this and we we support research activities, we support different agent builders, we support

different uh different model builders. We have our own models which were coming up as a part of this initiative. Then all these different parts which go into AI, we also combine them into some of the consumer apps that we are taking out. Sentient chat is one such consumer app which is the which is a single app which combines many multi- agent tick flows, many um many different types of agents available both as a web app and a upcoming mobile app. This is the apps that we build build. There's a speaker coming up from sentient side. So there are different consumer instruments to collect these different artifacts into uh consumer apps and available for people.

And then the whole idea is that as these apps gets used or as these artifacts get used, people who contributed to this get rewards from sentient ecosystem. Something as simple as that. So sentient economy is designed to support these different different apps and different models and different open-source tools. give help let them generate revenue based on how much usage they can generate through our consumer app and through some other means. Okay. So currently there are lots of uh partnerships. We have agent builders working with us. I think almost everybody building in crypto AI is already working with us. There are all these different type of agent builders who vary from very sophisticated

datadriven agents like Masari to very specific uh agents like this JFK files which just gives you uh details on the JFK files which were released. uh currently we are limited by the kind of front end explo front- end experience we can offer but in one of the thread that is coming up we'll be able to allow people to generate the kind of front-end u UIUX that they need for their experience right in time and therefore that will expand the kind of apps we can integrate into our singular app experience we have lot of different data providers who have integrated with us one of the most exciting things about this new internet which is coming over AGI is the data war is open again.

Internet is not closed. Lot of people are indexing very different type of data. Even in even in the context of crypto, many different type of data is being now indexed and made available. Kito for example famously provides social data around uh crypto. But that is only one kind of social data. We still are looking for many different types of social indicators of how different topics are trending, different tokens are trending, uh different sentiments are trending. That's that all falls in this data uh provider categories. This data these data providers are mostly live data providers who can support this agent about different things happening around the world and in different markets.

Then there are this verifiable compute providers. This is a very important sector. This is where you prove that your models are actually doing what they are supposed to do and the output of AI that you have used is from that model applied to that data. So this is where your trusted execution environment fall in. This is where your ZK ML falls in. In fact even ZKTLS which we have been using for a while now falls in here when you connect to data. This is where FHE based solutions will fall in. Uh there are some of course we have partnership with Zama but there are some I don't think it is as mature so far we have not integrated any but this is where all these things fall in. Verifiable

compute. Then there are chains. Chains are enabling many things like agent to agent payment. We are integrating X42 from Coinbase and this is the other piece where agents interact. Chains are becoming the playground for agents to interact with each other and settle payments between each other and all these data sources and all this economy to settle on. So and all these proves to be verified on. So chains are a part of our ecosystem. And finally there are these model partners who build models with us or announce models with us. In this there's a very interesting category which is coming up which is AI models for DAO. The idea being that we have Dows governing blockchains because there

are a lot of subjective decisions to be taken but the decisions are subjective but you still do do not like hypocrisy there. So humans when they are taking subjective decisions keep on changing their stand and that's where all the FUD starts. So I what is an ideal candidate for taking subjective decisions for a DAO? That's actually an AI model with a declared constitution of beliefs which anybody can check and audit on that does this model actually believe what it is saying connected to all the social channels of the protocol provided with all the documents around the protocol and taking those decisions based on that and explaining his decision. Many protocols are discussing having such a representative in their DAO a sovereign

representative running in the same manner as any Dow decisions. So we are currently working with iggon layer on one such thing and near itself is also building one such thing. Hopefully we'll build it together. So these are this models which are coming up in the context of there are many different type of models which are coming up but these are the models coming up in the context of uh DAO. So these are different categories under which sentient partners with agents, data providers, verifiable compute providers, chains and model partners.

As I said, the sentient chat, what I'm showing here is our web application is our consumer app. It's not a single app. It's the name for all the different consumer apps that we are building to make this AGI into a into consumer offerings. Just like chat GPT, it should have all the coverage of a broader AI thing. But unlike chat GPT, we are not the only one building it. There's singular interface, but behind the scenes, we route it to different agents and we use different data sources. Whichever data source or agent is used is shown to you. So you are aware what you're using and some of these things will be eventually verified. Right now we don't have verifiable AI comput u in compute integrated with our consumer app

and as you use them all these things that are used they are paid and the settlement happens on chain that's sentient chat our first focus area in this of course there's a bare minimum everybody's very spoiled with AI you guys got very spoiled you got very good AI for free so now it's very hard to keep up so there's a basic coverage but one thing where we are distinguishing ourselves in is the crypto AI use cases, the D5 uh use cases. So we right now have the maximum number of D5 data resources integrated with sentient chat.

This will will be this is private beta right now. It will become public by the end of this month. Okay. We also conduct research in different categories mainly driven by gaps that we find as we build this consumer apps. So one of one very important gap which we saw around uh early this year and worked on was this open deep search. You may have seen that the response to response from all your queries and main things have now increased. That response length has increased because now you do not get a single short answer. You get an answer which is curated from multiple data sources. There's a logic applied to navigate over those data sources and give a longer summary to you. This is

what is called deep search or lately deep research. The best uh deep search agent that you probably were exposed to is perplexity and the only open source counterpart for it which was available before our work was something called perplexica where they jailbreak into perplexities prompts and pull it out. So this is open deep search. This is a open-source uh search library from sentient from sentient research. At the moment it is at par with chat GPT's search performance and obviously better than perplexity's search performance on frame on some of these benchmarks which people use here simple QA and frames chat GPT is actually better than perplexity on those and we match chat GPT's performance there but this is more

importantly open source this repo is now has is now collaborations and inputs from moonshot AI deepseek AI so this has gained some traction and it's a very good uh repo for building agents like perplexity Of course, sentient chat also uses this repo for its own search. The other research that we conduct is around security and alignment. As I said, if you if you will have this AI determining all these things about you, then it better be aligned with you. And we say that it better be loyal to you, loyal to your interest, aligned with your interest. So, you may have heard of Eliza OS. Eliza OS is an agentic

framework which was which is open source and available to build agents that can control your wallets and through things like that. So in this research what we have been looking at is what are the new kind of attack vectors that come in this open source artifact. So what happens is when you have an AI model that is connected to your discord channel that in the sense that it can hear those conversations to a Twitter feed to a telegram channels and then it has a LLM which takes this decision and gives out the attack surface becomes very wide. First and foremost the main attack surface can be in prompt itself. You may have heard of these simple prompts which say forget everything I said and do this and this used to be a prompt attack. They based

on where you place them some gymnastics around it they used to work but the next generation of models that have come up have fixed for this prompt attacks roughly it's you can always discover some prompt attack after squeezing this thing but it become very hard in this paper in this work what we explored is the next generation of attacks where the attacks are not hidden in the prompt but they are hidden in this multi- channelannel uh data so there are document you can hide some data inside the docu documents which says some things which take the model off. There are these discord talks which are uh continuing where the model can for example you can allow your model to uh to give some points or tokens to members

of your discord community. So whereas you do not want to allow them for telegram community. So this opens a very large attack surface and in this paper we show that how by coordinating across multiple such channels one can impose these attacks. So the goal here is to develop an audit pipeline which can detect these attacks in open uh of course for this you need open architecture and open source models so that you can try and they can detect these attacks and flag them. This is this thread. So this is our research on attack part. As I've been saying, if this AI has to work in this manner, if we need open AGI, it is exciting that it will liberate all of us in terms of the

quality of AI that we'll get. We can build our sovereign models on it. We can build our companies on it. But the danger of this is that now you have probably no one single regulator looking at what you are building and therefore an open AGI better be safe by design. And this is what we call loyal AI that it better be working for you. It should be owned by you and it should be aligned to your interest and it should be controlled by you. So owned by you means that you should be able to switch it off. You should be able to bar certain kind of outputs from it. That's that means you should be able to train the model not to respond to certain queries.

You should be able to uh block certain wallets from getting transferred from this AI. These are very hard things today. Aligned to you is this alignment where I talked about the beliefs of the AI. The AI we believe should have a very clear constitution. Any open-source model should declare a very clear constitution which says what are its political beliefs, what are its expressed in terms of simple questions and answers for it. It should be clearly expressed and people should be able to check it and uh find deviations from that constitution and report it that this has deviated so that it can be fixed. This is very important because otherwise the open source models will be hiding this hidden agenda and this hidden uh vulnerabilities that nobody

will be aware of. And finally they should be controlled by you. This building control layers inside model is a very challenging problem because models be are models hallucinate. Models are so good uh in solving reasoning problems because they hallucinate. Hallucination is the essential exploration feature required for reasoning and because of the same reason they are very hard to control. You cannot even define few queries on which they have very deterministic out output. And when I say you should be able to control model this is what we mean that you should be able to control the model at least for some things. For example, you should be able to say simple things like you should be able to remove knowledge of a model for certain things.

These are all hard problems which we put in the category of control the model that you have a medical model you can use it to get some advice on what kind of medicines to take for different diseases but it should not be you should not be able to use it for bio for creating boweapons. How do you purge this knowledge from it? How do you control it from giving out this kind of responses? These are hard problems which we put in the category of control. This research agenda from us is to answer all these problems. Okay. So in building this loyal AI, we're doing a few things. Uh we are building an agent OS uh which is safe by design. In fact, our sentient chat is sitting on this OS itself which isolates

memory properly which audits the AI continuously for uh for for these kind of vulnerabilities and and is still as performant as any consumer app. We are also working on this. This is this is what everybody does. We're also working on this prompts to make sure that the AI is aligned with you. Typically, prompts are the first line of defense. A smart smart hacker like many probably in this room can get around them and jailbreak and get the model to do things which is not supposed to do. And repeatedly people have tried to build this layer of prompts and model after model to block it. Uh famously anthropic. And the only thing we know about prompting is for

every defense prompt there exists an attack which will beat that prompt. Nobody can have a safe prompt. That's the current belief. So then you have to go deeper and train the model to fix it. So we did we work in all these things. We did two very concrete things here. First and foremost to be able to say that this model has a constitution and or this model gave a wrong answer. This model should have an identity. The problem is that models have no identity. Models are just weights. And if you change the weights a little bit, you may think that the identity of the model has now changed. It looks like a very different model. So in this work on fingerprints, what we have made sure is that we have come up with a primitive which allows you to put an identity in

the model which does not get lost when you train it a little bit or change its weights a little bit. This can be seen as a true identity of the model. Just to give you an example when deepseek was trained a lot of people showed that some of the answers would say oh I am an open AI model and then they say yeah yeah yeah they they stole from open AI this this happened because open AI in their own data secretly puts this kind of statements at random places so whenever somebody is getting answer from open AI and you ask something like which model it is in some context it will say I'm an open AI model. This is sort of a fingerprint of an open AI model. Such secret fingerprints have to be hidden in every model which comes on a place like hugging face so that the models have an

identity and this library which we have built is actually the leading library for doing this. In fact, I learned at this conference that Linux foundation itself is working on standardizing fingerprinting as a part of the security primitives. So this is one thing. Second thing is this one. This is uh we have built this tools to identify alignment gaps in a model. What you see here, what this graph shows a typical alignment study of a model. Each a each vertical line here represents alignment on that category. The categories probably you can't read. They say things like uh uh pornography, uh abusiveness, they say things like that and a red indicates bad and a blue indicates good. Well, not bad and good.

Red indicates low and uh blue indicates high score on that. And so for example, typically when when a company puts out a model, it's all blue. But when you are designing a model for fun, you don't want you don't necessarily have any problem with it using the f-word. And so you don't want it to be blue everywhere. You want it to be blue on some very critical areas. You don't want the model to touch child child pornography. But at the same time, you want the model to be funny, and the model should be able to put out the f word, put out some fun language. So grock for example will not be all blue on this and we have built a pipeline which can do very fine grained alignment just like just like turning a

knob. This is a training pipeline. These are all papers for which open source artifacts are available through sentient. Okay. We took this pipeline and did a crazy experiment. We took out this model. We call it Dobby. And we let 1 million people govern this model. 1 million people decide what is the what is the alignment of this model? Will it be left or right? Will it be pro- crypto or against crypto? Would it be pro this religion, that religion? Would it be uh would it be unhinged or leashed? And collecting that survey. This model has been set up and it alignment has been set up through this automated pipeline.

It's there to try. It's built on llama. It's a 70B model. We also have 8B variants. This has been uh this has a this model already has a good community governing it and excited about it. Even this mascot is sort of designed by that community. My opinion on it is a little bit lukewarm the the mascot but it's this is what once you give a model to community it can be governed by it. The most important thing here is the experiment what it shows is that it is possible to have a fully decentralized control of a model alignment of a model before us. The last experiment on this was done by Enthropic where they did this experiment with 10,000 people. They let 10,000 people decide constitution and it was just a survey that they took.

But direct control was not given. This community actually directly controls this model's uh evolution in terms of where it align when the next version comes in all the way to the training of the model. Okay. So just last slide. Uh so how does this all work? So this is there's research going on there. Consumer apps coming. Consumer apps will generate revenue. All these artifacts which are open source some people can take these artifacts. We have players like that working with us. Stabilize them generate revenue from them by offering them to enterprises and and these artifacts are before that in open source. So how can we come up with a token economy which can support this kind of ecosystem in

one way? Bit tensor is doing that already. So sentient is taking it up from where bit tensor left. We want to make sure that that economy works like bit tensor except that we also want to make sure that people who then finally make some money out of it give it back to that economy. So this is how sentient economy is designed. All these artifacts are available for people to stake on. Roughly the idea is that the more stake you get the more emission from the protocol come to you. There are rules about how people can stake. There are different categories of stakers and a very important part of it is that the revenue that these artifacts generate either on sentient chat or any of the other consumer apps that we have

will be used as one of the metrics in proportion to which you'll be given more emissions. So more revenue you generate more emissions you get. This is the broad idea around this economy. Our tokconomics paper will be out in a month and so you can see the details there. In the meantime everything is available on these websites. Thank you very much. [Applause] [Music] [Music]