Civic AI for Self-Governing Communities

Transcript

[music] Uh good local time. Uh very happy to be here. Uh and I see that people are still trickling in. So maybe uh I speak slightly slower. Um so yeah, this morning uh I said that this is a crowdsourced talk and you posted 36 questions by the last count uh on the share the mircat board. Uh and so this is entirely crowd sourced. Um I saw a couple questions uh that basically said uh it's for Vitalik and not me. So I will not answer those two questions. Uh but I will uh try to get through all of the others uh as well. So um the two questions uh that are most top voted and you can check it uh the receipt on the

DevCon app uh are privacy uh that is to say where the privacy ends where the openness begins uh when should we think about the privacy transparency trade-off or is there no trade-off at all and also um people care a lot about agents that manage treasury um meaning that if a agent runs a community's money and takes risk that nobody agreed to. Uh how should we re in the agent uh together? And then people also ask about what if the agent is not loyal only to one person. What if it's loyal to a relationship? If it's a agent that's not loyal to me and not loyal to you, but to the health of our relationship, what

kind of agent is this kind of relational agent? It's a very good question. And also when we have a breakup uh when we don't want to have a relationship anymore uh what should happen to that agent? Again a very interesting question. Um and also uh people uh wanted some conversations around the Ethereum Foundation's idea of detoalization and what that would mean in the world of agents and models. What would a detoalizing agentic stack uh actually mean? And uh finally like who the agents and the public system should answer to. And so um I think all these are very good questions uh on the civic AI ideas. So I will just uh go through the themes uh one by one. I will not uh

go into a lot of technical depth. Um there's a lot of technical links and sources on the civic.ai website. Uh there's literally a page called sources that can let you check all the mathematics and so on. Uh so this talk is mostly about the broad ideas but I look forward to uh more in-depth inquiries and questions over email. Okay. So the first question um where do we need more privacy and where do we need more transparency? Um there's this very um popular meme about equality versus equity. I think many of you have seen before, right? Like if you uh make everyone grow 10 more levels, it's actually not making the world more

equal. Actually, maybe it's making the strongest uh even stronger because they reach a level that can dominate the others. And so I think of uh the power of privacy which is of course a very strong technology um is that the more power you hold over other people then more transparency you owe to those people and if you don't hold any power over any other people uh then of course uh you should not be made transparent uh to other people. uh if a state uh makes its citizens transparent to the state but the state itself is not even equally transparent to the citizens then we have a power concentration problem and uh the AI systems uh that can only access

celebrate this problem is one of the defining challenges of our era and so I think uh for anyone who is on the receiving end of those asymmetric power uh relationships our messages our health our vote and so on should stay uh private and uh the public rules that governs um us all. Of course, those stay public. And so there's a very simple heristic for any piece of data. Just ask uh one question. Does it record who someone is uh allowing somebody to dox somebody else? Uh or whether it just records the fact that someone hold power over others. So the first must be protected and the second must be made transparent. It's a very simple rule of thumb. Elections uh already work that

way. The secret ballot is still one of the most important social technologies invented. Uh basically says the count is public. Everybody should be able to validate and verify the count. Uh but whomever individually vote uh that choice need to stay private and we already in the Ethereum ecosystem have many voting system that work the same way. A proof only shows the aggregate. the count is correct and a voter can quietly even change their vote uh after a briber watches and so the briber could never be sure there's many mechanisms that protect the individual privacy now the question also asked uh who draws a line uh again the people affected must write the rule together uh so that rule

set is public in AI systems uh those rules are sometime called constitutions or model specifications or the Microsoft call it code of conduct uh muse called a charter metamu. So many many different words. Uh but it's basically a shared rule on what counts as good for that particular model or agentic um interaction. And so uh since a couple years ago, I've been advocating uh for all the frontier labs to publish their specs or constitutions and also to uh declare them in public domain. meaning that if you uh fork you change it to a different direction that your community needs, you're not violating the intellectual property or copyright uh of

the frontier model companies and uh that also allows us to fork the constitutions. So for example when clot three was being trained um we took the clot 2 constitution and sent uh to a hund no actually a thousand random uh US people and the people in the US collectively look and have a conversation using polus uh on the claw 2 constitution to find bugs find loopholes find vulnerabilities in those uh constitutions so it's like a citizen jury audit and people very quickly found loopholes for example in claw two it assumed that everybody walks and forgets some people are in wheelchairs because the researchers in anthropic at the time probably did not have uh people with

disabilities um and so uh once people um highlight the delta then people can uh write the testing suite the so-called eval and so then people can prove that claw 3 now respect disability rights more without sacrificing u the capabilities in other regard and so as more and more agentic systems u govern our collective decision I At every level the constitution and constitution like documents need to be made public in this way so that people can not just check the delta but also make sure that the behavioral changes respond not to just the artificial analysis or other benchmarks uh but rather to the people's benchmarks that we write together. Now a related question asked uh what could we

prove without exposing the private details behind the proof. Uh most systems today ask for a whole document uh when they only need uh one proof. Uh in Taiwan, we put in action a couple years ago the wallet.g uh.gov thetw system, wallet government system, uh that basically embeds selective disclosure into the national identity system. So that if I want to, I don't know, prove that I'm of a legal age, uh, to, I don't know, collect some alcohol or something from the local grocery store, um, I just mint a selective disclosure showing that I'm over 18 years old, uh, without revealing anything like my birthday or place I live and so on. And the most important

thing of this kind of system is that it needs to be the default. We need to make this kind of systems more convenient to use. Um it's a little bit like switching from passwords to pass key. If you use a pass key system that's harder to use than password system, then nobody makes the switch. But if you make it so that the biometric the passcase system uh shrinks even just a couple seconds uh from the password authentication, then everybody flips the default and use the easier way. Um the Ethereum ecosystem also have uh anonymous adha uh that uses the reuses the India adhar system uh and built this kind of solid disclosure on top of it and this has influenced the equivalent in Japan the my number system

anonymous anonymous my number and as well as many other systems. So I think um in in defcon uh we're kind of in this small proof of concept community. I remember when I was in Bangkok uh I collected those tickets using Zupass again a uh system that does not need a full disclosure. I think our um incoming challenge uh is to look at the current vulnerabilities that are you know a Gentic system review a vulnerability literally every day now if not every hour. Um and so uh people as vitilix uh said will quickly see any personal information [snorts] uh collected in mass in aggregate as toxic waste and so we'll look for ways to uh go beyond the

collection and uh in this vacuum uh if we can then say oh by the way we now have a selective disclosure system even zero knowledge systems uh that does not um endanger you with more toxic waste I think there will be a uh real window of opportunity in the next few months uh to basically look at all the different uh ministers in the world, digital ministers that has to answer to the parliament and they will have to say never again and any technical solution that will allow them to actually mean it when they say never again uh will probably win the political overtone window. So I think uh we need to be ready to communicate that this storm is coming. We're all weathering the storm but somehow we in this community have

anticipated the storm and uh now we have some umbrellas uh for other ministers. So this is uh the answer to that question. Now the other question asked if a agent manage a community treasury uh so every transaction is valid on chain but then it takes risks that nobody agreed to. How do we collectively check uh that it did what the committee wanted? And again um I remember a few years ago uh two years ago there was a agent called Frasa uh that is a kind of bug bounty. It guarded a prize pool and the instruction basically said never send this money and never change this rule and anyone can pay to send it one message and kind of try their luck and

so the first 481 messages uh fizzled out. uh it didn't break it but then the next message broke it because it convinced Frasa that this tool for sending money is actually not about sending money but really about receiving money uh and then over a small donation and then Frasa used the tool and send a whole pool uh about $47,000 and so of course it's a bug bounty is a game proof of concept uh build to be beaten which makes a very clean experiment and I think the the rule here is very simple is that never treat uh this auto reggressive agentic generative system that outputs tokens uh as actually following any hard rule. Any rule that

it claims to follow are actually soft rules. And so whatever soft rules guard rules that we put around it, we need additional hard rules um to safeguard it. And indeed when we developed the civic AI ideas in Oxford, uh we basically said um we need to separate the instructions to the models which can only ask and advice and have thousands of ways of being broken uh versus the guarantees the invariants uh that we heard about in the keynote. uh that you can translate some of those uh expectations and instructions into variants that are verified either uh by a special purpose model that is just safeguarding the larger model or even

better uh by softwares uh for example I use bend bendd uh bend 2 uh that basically translates those invariants into code that is parallelizable uh on GPUs like on CUDA but also it's a proof carrying so it carries its own uh lean team verification. So it's not a trade-off between like a faster to verify versus uh more formal to verify. Nowadays uh you can have this kind of tool that have your cake and ED2 uh so that you do not uh leave much to the larger model uh in the middle uh to uh determine its own soft rules. And so I think the uh agentic system that we now use are more or less all of that sort in

that if you ask it to check to auto review to check its own flaws, vulnerability and so on, it can generate very convincing essays. Uh but now research have also shown uh that uh in their activation space uh basically in their in their mind they can also reason that this is important to not let a user worry about it or this is important to hide this from our chain of thought. So we are soon going into a uh place where not everything that can be read from the chain of thought uh transcripts uh can be verified uh in such an automated way.

So again people are exploring various ways like uh reading those neurons activations directly and so on. But for now I think the most practical way is simply to determine the action space that it can take and then write as I mentioned invariants uh that basically determine uh the hard limits uh that it must obey. And so um I think many people here um also knows that there are smart contracts that already work this way. Uh for example on uh Leo the easy track a proposal waits for three days. If enough token holders object then it stops. But most proposals have a fasttrack that passes quietly and anyone who disagrees also has a clear window and also a low bar to challenge and some uh DA

organizations go one step further. So members who disagree can then port the leaf with their share before the decision takes effect. Uh for example, housing society's committee can work the same way if proposes spend and members get a few days to object before the money moves. And so for the question uh about the treasury, I think that is basically the answer. Inside the previously agreed limits, the agent acts uh with this invariance and then every action leaves a receipt um preferably on chain that anybody can match and check.

And so outside limits are if you detect something that is um questionable uh and then the agent must ask the humans uh and then wait for the answer. And the risk nobody agreed to now has a plain meaning. anything the rule leaves out and anything above their capsu that is for humans and so I compare this design with the usual design of what people call human in the loop uh usually if you are brought as a individual in the loop of agentic system uh I don't know about you but I more and more feel if I am just a enterpressor uh in a humane the loop system uh I feel that I'm a hamster called in a hamster wheel uh and the wheel just rotates ever faster and faster at at some points. It just asks

me for an enter for NDS or something. Uh but I don't really feel that I'm steering the hamster wheel. It's not really navigating the way that I uh want it to be. On the other hand, if there's a existing system of communities of two or more people and the agents actions take place um on the shared uh space like a Google doc, a slack, a discord and so on, then it's not individual humans pulled into the loop of AI but rather individual agents pulled into existing loops of humanity and the good thing about this is of course if it's in the interpersonal space then community already have a code of conduct already have a prefers speed on which to uh function and we don't feel that uh the

artificial synthetic intimacy you know the agent always saying oh you're 100% right uh your ideas are so loadbearing let's delve uh into this uh wonderful insight uh those cannot hijack you anymore because the more it tried to hijack any individual the more uh cringey it would look like to the other people in the same group channel so I think bringing agents into the loop of communities of humanity and having very clear set uh date based or u time based rules. I think that is one simple way to counter against the synthetic intimacy of uh humans so-called in the loop of AI.

Now uh another question also asked okay so now we all know how to draw the line together but should the line be enforced by code or by the state law or by the norms of the community and uh if you look at all the functioning systems today um it's all three uh because each of them are doing a slightly different job uh code can hold the limits that can be specified by code but then always we see exceptions so we need people holding the norms to judge exceptions and uh if the people do not uphold the norm or just goes missing or do not answer then we have the law that should uh basically summons the responsible person and name who must answer kind of as a last uh

backs stop. Uh indeed on X uh I got a question um on my opening keynote uh today saying that okay the uh anti-fraud law in Taiwan which was crowdsourced uh basically says people uh can flag every incoming fraud and if it's uh not um taken down in 24 days after a authority basically said that this people's trained bot really flagged a real scam uh then Facebook or YouTube or whomever uh showing those unsigned ad and did not take it down must then pay the same damage. If somebody gets counted 5 million, then uh they must pay 5 million. Um and so the Twitter uh X.com question asked, but who is actually enforcing it? And the answer here, it

really is everybody because fraudbuster is uh public. It's a almost like a public ledger. Everybody can see that the state must forward to the health authority if you like a scam related to uh drugs and so on. uh and uh that authority again is on a timetable uh to make a judgment and once it's made a judgment then it loops back to whomever flag a similar scan. So you actually know that your flag uh or your uh comments and so on um actually changed the receipts of what the platform gets.

So this is again the power being radically transparent to the people but the individuals doing this flagging does not need to dox uh themselves again protecting the privacy to the people on the receiving end of power but demanding transparency for the big tech and for the state that is uh on the giving end of the power. So I think that is how um nowadays the working systems need to coordinate between the law and the norm and the code. And um um another question is about this relational agents uh which I really think it's very good. So now you're bringing this agents into existing communities but existing communities uh the relational health um may or may not be uh satisfaced uh forever uh people uh

break apart the communities fork uh and so on. So what uh happens to the agents right so I think uh on the smallest scale if two people share a model for their relationship and then they agree to leave uh what should happen uh one idea is that we can agree to the ending uh before the model even started acrewing memory so each person's private note can just stay on their own device and what they share uh lives in the notes both of them can read and so that is kind of common pool knowledge but otherwise it just keeps um like anchors uh hyperlinks so not actually the private details but rather just the

links like UU ids and so on that uh this private store can disconnect and uh stop to validate at any given time. Technically this is called uh attribution based control and this is now offered as one of the better ways to solve the problem of overagregating. Basically, it's extremely difficult to exercise your right to be forgotten if your um ideas and your story is in the pre-training of those frontier models. Uh to force them to forget is extremely difficult, if not impossible. Um and so uh one uh very good uh research direction uh and now it's being proven on a small scale is for each person and each small community to train their own

attribution based control pod uh and only share the links um as I was mentioning the UIDS and so basically the model in the middle is no longer a pre-trained model that kind of drill the data oil uh and to basically do all the pre-training by itself. It's more like a coordinator model uh that is kind of small uh if you use open router or use sakana you already know what I'm talking about basically looking at every prompt and say oh these uh pots seems to be related to this query and maybe I would just forward your query to those various different pots and so um I think this is much more feasible than machine unlearning then it also solves the problem of like what people want

copyright attribution to solve basically paying uh for new commissioned work works that um help the perplexity make the model more powerful because once it's already aggregated it's very difficult to figure out uh what kind of input really contributed to this particular query but if you don't aggregate in the first place but always just do attribution based control then on every query it's very easy to uh calculate uh if you go without or with this kind of pod how much did they contribute and so if we all uh practice very similarly is attribution based control then breaking uh up uh from communities uh can entitle someone to uh come out with their fair share of

contributions and so nobody will feel that uh their digital twin is still being captured and enslaved uh in this uh community model. So that's the answer to this question. Now um people are also uh wondering about detoalization as you can see in the Ethereum Foundation mandate. uh detailization means a world in which no organization, no system, no moral order has total dominance over any individual's life. Um I think this is uh a very reliable good aim is like a northstar for me. Um because building toward this kind of the world uh opens up the canvas. Um, I like to think of myself as a good enough ancestor and

it's not a perfect ancestor that designs out all the bugs. Uh, so the future generation can only follow the code we write in our generation but rather to leave the future generations with more canvas with a wider canvas with more materials uh for re reuse uh because they will face a very different world uh compared to the one our generation lives in. Um and so if we have total dominance over them, it's actually not the best idea because the world itself will change. So I think that is for me a very good uh northstar. And so in practice though uh when we uh talk about agentic models um this means a few practices that we can all do today. Uh first of all uh I I never actually use just one

chat model. Um I used to um ask the chatbots uh to never use the pronoun I. Uh so they never pretend to be my best friend. Um like every turn I use the system pronoun saying uh always give me answer as a one pager as a HTML. Uh and that HTML doesn't have any pronoun in it. So I don't have a chance to get flattered uh by the model. And now it's getting much easier uh because uh there's a lot of open harnesses. I use uh OM uh previously I used OM uh that basically arranged this uh on your behalf. Uh you would have a coordinator model, a planner model, verifiers, multiple workers and for each to

basically reach you uh they have to convince the other adversarially uh trained models that they're actually fulfilling um the initial ask instead of just reward hacking to convince you that they have worked on the the ask. And so these kind of internal debate uh between the models I've also found that if uh you assign all those different roles to the models trained by the same uh pre-training company uh they collude uh very easily. But if you uh go to like really different pre-training models uh for example Gemini Argon uh versus uh Claude um then they actually uh are very adversarial against each other and do not tend to repeat um each other's um blind spots. And of course the best way

is still like after using this kind of harness for a while then you also train your own local uh post-train model is now getting especially if you're using a Mac um faster than most of the cloud models now if [music] you know what you're doing and so I think that is the most important thing which is we have many helpers and [music] then each one we can add we can switch to others so now we're switching uh to the second part uh of the talk uh where I think there are questions meant for vitilik about uh the mandate holding the censorship resistance or capture resistance or collision resistance and open source and privacy and [music]