Autonomous agents and artificial artists / Gene Kogan

Transcript

okay hi everybody uh how does this work okay so my name is Gene kogan and this talk is about autonomous agents and artificial artists so I'll just quickly tell you about my background so you know where I'm coming from uh I've been basically spent most of my career in um in machine learning in one form or another I uh for I studied machine learning in college and then became kind of a research assistant working on applications of machine learning to music technology working on things like recommender systems collaborative filtering stuff

like that also sort of pulling out interesting information from audio signals for applications to creative applications you pull out the beat or the tempo of a piece of music or the genre or the mood characteristics things like that in order to drive playlisting algorithms stuff like that at the same time I had a hobby as a generative artist I've been using for a long time Frameworks like processing open Frameworks Max MSP if any of you are familiar with those things and um and then these two worlds kind of started to collide around 10 years ago and when generative AI started to take off as a

field or what's now called generative AI it wasn't called that back then so I started using techniques like Gans Style Style transfer deep dream and and in order to make you know interesting kinds of art so I've had a front row seat to watching this field really really take off over the last eight years that image on the left is a face if you could believe it trained on a date made from a Model A variation autoencoder this kind of neural network that was trained on a data set of celebrity faces this was the state of the art in 2015 and as everybody can

probably see it's it's it's really it's grown up very very quickly very fast so we're now making realistic imagery also video text of you know uh that well that's very very capable probably everyone here has interacted with chat gbt at some point so the fields really exploded in popularity and interest and so I've been kind of reacting to that lately these used to be videos so but I'll just uh some some work of mine from uh collections that I've made of artworks these are actually videos that are looping but um not for today I'm most well known for my educational

works I've been making these resources kind of tutorials code and instructional materials book chapters things like that for teaching people in in different creative professions including artists and interaction designers and so on basically how to adopt these materials and to use them in their own craft and so this is a project called ml foray and I've also been running workshops around these resources for most of the last seven seven eight years including for a couple of years is basically a full-time job so this topic of an autonomous artificial artist I really love it's

kind of been my um my obsession for the last few years it sits at the Confluence of multiple completely different fields in my opinion so there's a computer art aspect to it of course and how that overlaps with artificial intelligence and machine learning and then there's this whole decentralization element to it blockchains and so on that has become you know intertwined and then there's also this kind of philosophy of Mind component to it which which I won't really get into much very much today but is no less interesting so you know questions of what is agency in the computer network you know what is um what is it what does it mean to be autonomous and those are

all things that have really driven this topic so from my vantage point I've had a chance to to anticipate some of the questions that are are today becoming more and more prominent so um you know if you're a digital artist you you're basically responsible for every pixel normally but in AI art this is really challenging and so it's a little bit different because in with AI generated images you're the last step in a very very long and complicated sequence that starts with you know many millions or even billions of people contributing data to this large data set

compute resources that are stretched all across the globe and um you know hundreds or even thousands of people writing the software on that underpins your own software and so this is such a collective Enterprise that it really kind of um at least to me it really challenges conventional pre-existing Notions of office and attribution copyright of course is a big issue now you know copyright as far as I can tell really wants a single person or at best an organization to be responsible for or attributable to one of these models or an image that came out of it but but really in in AI generated art it's

there's so much more obviously and so I've been really interested in in questions that arise from that so around 2016 I became acquainted with the art Dao idea I mean maybe some of you have seen this um basically a couple people who are now friends of mine Trent McConaughey and Simon delaruvier started theorizing about this idea of a of a smart contract that would um create generative art autonomously put it on the blockchain and sell it and then you know use the proceeds to pay for itself to pay for its own computation and I was really attracted to really inspired by this idea it really completed I think an idea that

had already kind of come from artificial intelligence circles lots of artists had been playing around with this idea of person personhood or liveness to their to their Creations thinking like can we make programs that would really seem to people to be completely you know have their own agency in their own autonomy and this was um and and I and but they were in my opinion kind of lacking this autonomy character and I think the art idea really completed it and I thought for my part I would try to kind of merge these two Traditions together into a single you know cohesive idea and I wrote an article about this in 2019 artist in the cloud were introduced the Abraham

project which was basically the idea of taking an art worked out at its core and placing in the center a generative model so an AI generative model and decentralizing all of the aspects of its training and operation in order to to really make it you know kind of have this autonomy I should mention that around this time Dows were thought of a little bit differently I think than they are today these days I feel like Dao's are kind of just like social organizations around the blockchain by the time the autonomous character of them was more emphasized and um and so you know Dows are this you know thing that interacts with people at the edges and this was um kind of you know trying trying to make a system that would

collectively to you know convert everyone's intelligence everyone's imagination into into unique and original art and I published a paper and um a workshop poster at the creativity Workshop in neurops which is the big machine learning conference where now in the first couple of years I really emphasis size the decentralized compute besides for blockchains there's a lot of interesting research kind of adjacent to machine learning in decentralizing aspects of the machine learning process from Federated learning which kind of decentralizes data collection to even things that are like a multi-party computation homomorphic encryption I think some people here probably seen the

zkml talks which is also kind of a an adjacent topic and those things are are more for decentralizing the the actual compute or you know creating certain kinds of privacy and security guarantees and I emphasize that in the first couple of years while I was studying this now in the meantime the idea of autonomous agents has become much more mainstream in just the last six months I would say in the aftermath of you know Chachi PT gpt4 maybe some of you have seen these projects like Auto GPT and baby AGI and those are open source projects in which people are trying to take these language Which models and place them into larger systems which have some kind of a more

autonomous character to them so you know a language model is something that you would normally think of like you go to Chachi petite and you prompt it but these are goal directed agents which are given the goal and then just kind of continue thinking in a loop and then trying to complete certain tasks that it sets for itself in response to these goals so these are agents that are capable of that they use language models at every step to kind of Reason about a goal divide it into tasks plan out those tasks execute them with access to certain kinds of tools so tools like searching the internet with Google access to data feeds and apis being able to make images and connect to

the internet if essentially and I would say that right now from having played with them it's still very raw they're not going to do anything you know really really worthwhile for you right right now but I would say that as the base models continue to improve and this idea of how to kind of compose with language models becomes more mature I think that this will become a really really big area I think everyone's going to have their own autonomous agents personal assistants companies will have Bots that are answer questions effectively and it's something to to really look out for there's also a paper a few weeks ago maybe a month ago called generative agents which was really interesting the researchers placed a bunch of these I think 20 of these

autonomous agents into like a 2d game map and then simulated a a little society and so each of the agents had kind of a heartbeat and an internal monologue and some memory of itself and would go on interacting with each other making new memories and and so on and the researchers even reported certain kinds of emergent behaviors how the they organize the a birthday party for one of the one of the age and you know had all sorts of like social interactions it's really really interesting stuff now the generative AI world has been maturing a lot there's a really big open source community that I'm plugged into and um and so the the

the tools just keep getting better I think and there's kind of this there's interesting notion that these um all of these tools are are in a single you know with that many of the participants in this world we share tools you know there and not only share tools but resources like like the actual models themselves the data sets the compute resources it's just this kind of large pool and so in the sense it it almost looks like public resources although people don't really think of them that way but they effectively function as public goods and for that reason I think there's a lot of uh yeah a lot of interesting aspects so so actually these slides I supposed to get rid of but anyway models and data

sets are also getting larger so um and the the models are improving very fast so there's a lot of uh interesting problems I I would say open open questions and opportunities in generative AI I already mentioned the copyright issues and attribution there's also this data sovereignty obviously lots of people's data is involved in in this a lot of artists in particular have questions about it and you know data sovereignty is has always been a big issue in the blockchain world and in generative AI it's becoming much more of an issue because you know we we acknowledge that these models have a lot of intrinsic value to them and so there's really no answers to to you know how that maybe how that incentivizes

people to even contribute in the first place of course machine learning is intrinsically centralizing because the compute resources are so intensive and so that's an issue that's presented there's been an influx of new participants artists and application developers and so on who for the most part uh you know are using all of these same resources but um for now there's there's really a lack of incentives for them to to Really coordinate with each other because despite the fact that they're all using kind of the same models I wrote an article a few weeks ago about an idea for creating some kind of an on-chain

provenance for generative AI so if you can imagine all of these different projects using the same flavors of stable diffusion same compute resources to be able to publish on-chain all of the Creations that they make with a reference to a particular checkpoint and the input spec that that made it a prompt and the random seed and so on and the idea for this is to not just to create a standard among them but to encourage some degree of interoperability and composability and so on and then the the second aspect of this is to create something that's a little bit more web free oriented where um you know that web 3 oriented in the sense of a decentralized platform that

they share uh because you know the the first thing that we'd like to do and this kind of brings me to Eden is uh to be able to create these generative models as a service um and and many such Services exist apis and so on that provide you know uplinks to these um these powerful models but all of them have the the sort of generic web 2 problems you know platform risk of course um you know and so so the idea with Eden is to is to try to create a web 3 Centric version of this which uses first of all a decentralized identity you know your your wallet is your identity you're not an entry in our database and and also the um this provenance layer as

well so and and the first they asked first important aspect of that is that all of the data is shared and and open in public so if Eden for example ever goes offline or stops offering these models you can basically take take all of that fork it off and continue running your operations you can all the models are open source you can host them elsewhere and so really trying to create a way for these otherwise independent parties to to have some kind of a a Commons between them you can go to eden.art and we have a couple of applications that allow you to log in with your wallet and make make Creations we're working on the on-chain

component part of this we have on the test net basically a provenance layer that's still kind of get in tuned and very much encourage people to to get involved so that's all I have I have a couple minutes for questions so thank you very much over there technology obviously makes it easier to fake events things like that do you have an idea of how that problem is going to be solved or if it's solvable at all yeah so the provenance aspect is very much adjacent to for example if you maybe caught some of the zkml talks you know trying to create a ways of verifying the the authenticity

of images and the origin so if you're referring to you know being able to trust that a piece of media came from you know a particular place of origin I think those are things that that really get solved by by a provenance layer provenance doesn't have to be on chain lots of companies of course have their own different versions of this watermarking you know um you know watermarking the the media and so on for for my part I think the on-chain aspect of this really helps to also Preserve decentralization very quick question as well how long does the job title prompt engineer last uh I don't know actually it's it feels like um most of these terms are very

ephemeral you know I I think a lot of the terms will change you know it's like how long is the term blockchain lasted I feel like people you know we're more into that term five years ago but it's not as though blockchains went away I think you know that that there's still there will always be a skill set to learn around how to interact with computers whether you call it programming or prompt engineering or whatever comes after that doesn't really change the the character of this you you know the the actual technical skills are always evolving but there's still a lot to learn and prompt engineering for its part there's still a lot of very interesting you know it's almost like archeology you know we're still learning

about the models that we've made and you know what's kind of unique about them thanks a lot yeah yeah over there foreign correctly Eden is a platform that's that's decentralized in order to have everything related to MLB you know open sourced and and then available right yeah and and decentralization is a multi-faceted uh multifaceted here so not everything is decentralized we use Amazon you know we have uh we have a bunch of gpus that that by necessity are

sort of in a cloud somewhere and so the idea is to um is to first of all decentralize as much of that as possible and to create kind of some kind of a web 3 overlay on top of it I when I was doing Abraham I was really concerned specifically about GPU sort of decent or trying to make the computation happen over many computers now I've sort of de-emphasized that because I actually don't think it's necessarily there's a lot of problems you can solve without doing that first and so um there's and so for me right now the focus is on decentralizing the sort of the data trail and and the access kind of kind of web3 access layer on top of web 2 resources and to make it so that all of

this is reproducible so open source is it's everything is open source and forkable and so on so yeah I was wondering uh what do you think about the centralization of the models themselves because those models are usually a large couple of gigabytes right even fine-tuned ones so do you see any potential future we can decentralize that or hosting of those models well a lot of those models are are open source and and and and and licensed permissively and so there's also there's already an aspect in which they're just out in the open I think the the question is how they're orchestrated I I think that's at least the first question I would like to see also kind of more decentralized networks for for training those models and then perhaps even creating kind of economic models around

them that incentivize creating them so there's been a lot of ideas at that space you know creating interesting kind of royalty streams encouraging remixing and and so on and I think that's a really promising area thanks I'll get one more page that imagery is currently at it seems like it's relatively close actually video so like so there are already a number of video tools so there's um there's a few companies that are that have uh very good text to video tools they're certainly not as good per

frame as as the text to image tools and there's also open source versions of them which are still you know kind of flimsy but um the if you just look at where where everything was two years ago um it really feels like the uh the uh pace of of improvement is extremely rapid and so I would say the the text to video tools I think you'll start to see them in this capacity in less than a year I mean you'll see that you can see them even now if you dig hard enough um to the point where they're more mature I think it'll it'll take a little bit longer yeah the name of one or two of those just so I can Runway has a has a good uh very good text Tim uh text to video tool I say that's kind of the the

most prominent one and also you know Google has demonstrated it I don't think they have any like kind of services around it but from a scientific standpoint it exists there's also incredibly um good audio generation as well happening so you'll start to see this come from music as well thanks yeah okay I think I think I'm out of time so thanks a lot for coming thank you [Music]