How to embed real-time AI video workflows into your product

Transcript

Okay, fantastic. Thank you everyone for joining. My my English teacher always said show don't tell. So here we are at the beginning. This is uh this is real time video AI. My name is Richard Agradi and I work with the Lie Foundation and I'm going to spend the next 12 minutes talking to you about what is real time video AI, why it's important to the world and how you can ultimately get involved. Now, real-time video AI is incredibly difficult to do. Why? Because usually when you have batch-based jobs, when you have usual gener generative AI, it can be done with um high latency, right? It can be take 20 25 seconds to

to process everything uh and and deliver the output. Whereas with real-time video AI which as you can see as I move my head there is a a slight slight delay but requires very very low latency and high compute on a continuous basis. This is a very heavy data stream that comes through but as you can see there is a real output and it responds to my body. So we can go into different ones but I will let everyone explore uh in their own time.

So I'm going to talk today about real time um video AI pipelines and why the future of live video uh is uh interactive, open- source and creative. Um so as you can see there are uh a a new ways of creating and manipulating live video and we really believe at Live Pear that the future of video is interactive. The future of video is open source and the future of video is creative. Um as I've already said my name is Rich and I spend my days thinking about the the future of live video with Live Pier.

And to begin with a a brief history. So obviously generative AI models have ushered in this new era of creativity. Starting with Google uh Google DeepDream and the Hallucinations back in 2015. Uh then obviously in 2022, Midjourney hits the scene and has 1 million users within 6 months. runway comes on the scene and opens up new video creativity uh and anyone can suddenly generate their own videos and uh and come up uh and put them together with high resolution. And then finally the much awaited o openai Sora appeared on the scene in 2024 and has fast become uh the leading platform

for generative video. But there is a new wave of creators and developers who are now exploring uh how to apply these AI models to video real time. Now what does that mean? Real time AI video combines multiple models into one pipeline and then applies that pipeline directly to a live stream. So you have the incoming video. The video then gets decoded. The AI models are then applied real time and all of this is uh the aim is to have this subsecond within a matter of milliseconds. The new the models are then applied to the video.

The video is encoded and the outgoing video is then that manipulated stream along with all all of the metadata that uh that that comes with it. Now like I said this is computationally very very hard and requires a huge amount of infrastructure distributed infrastructure around the world to be able to do this because the whenever you're thinking live video you're thinking content delivery networks you're thinking uh globally distributed uh compute which requires uh a very very high bandwidth and low latency.

And the cool thing is though that obviously with hugging face and the the the birth of open well not the birth but the uh the summary the aggregation of open-source models out there. There are loads of open-source media models that uh that we can play with and combine into a a given pipeline. So you may have seen video background removal, lip syncer um or stable diffusion which have obviously uh been developed by all amazing open source developers around the world and can be wrapped up into a a single pipeline and applied directly to live streams.

So what does this mean? Well, ultimately it means that video can now become creative, interactive and intelligent. Right? So this is this is an ingest of Eric our founder on the on the right and the output of uh him being transformed into a character from Shrek. So it's again this is another video that is uh ingested real time and you can see that the image is manipulated and uh the video is man manipulated and then transformed into something else real time. So why does that mean that it's interactive? Well, we all know that the the future of video is interactive. We

know that uh we will have virtual assistants, virtual teachers. uh we know that uh we're going to to have we already have image recognition at a a large scale. Computer vision is moving forward. So we need to ensure that videos are not just something that we passively consume but something that we also interact with. And this is how they can become intelligent and also lead to a new um a new wave of creativity that comes through not just creating the video but inter interacting and manipulating it. And Daydream is Live Pier's first real time video AI application. For those of you who uh who

missed it, we can quickly go back and have a have a tour. Um so this is uh this is Daydream as it currently is and what we can see is there are already loads of preackaged u pipelines that we can apply to a given live stream. So for instance I can turn myself into into Bender robot. If I click on here you will see that it automatically manipulates it. Now obviously this is great for you know a bit of uh a bit of fun whether that's on on on social or also creating um uh virtual agents that uh can be uh can be used by any sort of vtuper or or otherwise. But this is all

happening real time and you can imagine the next stage is not just uh an agent doing this uh doing this real time but also when I for instance am a uh I'm a content creator and I want to go to sleep for for 12 hours but I want to be on a 247 live stream people can still interact with me if I have uh a virtual equivalent of myself. Um there are plenty of others that are are here that have been created both internally by the LIE team and uh and various other uh other contributors that we have in our in our ecosystem. And there is a really um exciting new wave of creative technologists that are combining Oh,

I've gone double. Um there we go. Uh it's part of the fun. uh that are that are exploring new ways of uh of interacting. Now there's a long way to go. Um we know that uh particularly open source models do lag behind uh closed source models. But this is where open-source AI um and the and the creativity that comes through open source is already leading to a massive um a massive um growth in the number of pipelines available.

So please do try it out. Uh I I will say that we are in beta. So uh do expect bugs. Uh please come up with some amazing prompts. We do regularly run prompt competitions. So, uh, feel free to check it out. But I think the main thing that I want to, uh, it's all well and good having an avatar and and having a little bit bit of fun on social, but what are the real world applications here when it comes to real-time video AI because we can obviously get uh get lost in a sort of uh, closed circle in uh, in in web 3. We know this. So, what are the real applications of real-time video AI um across the board? And we're we're

looking at just the beginning. So there's everything from accessibilities around disability, sign language, there's healthcare and disease detection, real-time sports analysis, public safe safety with AI um uh AI moderated uh AI um uh regulated uh surveillance uh of of uh private and public spaces. There's film and entertainment. There's education and virtual classrooms. The number of possibilities with video are endless. that that's already 80% of uh of internet bandwidth. Um and in the future with the the the birth of uh new interactive video, we see this going up even higher.

So how so where does Lifepair fit into this? Well, Lifepair is really leading uh is bit building the the the leading real time uh AI video community and platform. um we really are trying to onboard developers and researchers at scale to make sure that we can fund um uh community members to to have further experiments and also we're launching an API soon so that we can start seeing some of those use cases beyond Daydream that we've uh we've put forward uh already and that is obviously going to be a super exciting time when um when developers can start having a play themselves.

And a quick history. So, Liper has been going since 2018. Um, and as we've said, video is hard and we've been working on this um since 2018 and the the launch of the the the token. Well, actually since 2016 when the company was founded, but we launched a token in 2018. We bootstrapped a network of GPUs before anyone knew what a GPU was, focused on transcoding. Um, we launched Lipis Studio which was an easytouse API on top uh of of the network and uh and have now entered this cascade vision where we're really positioning Live as the the the leading force in real time AI video. And on my side, uh we've recently set up the

the Liper Foundation, which is a very small lightweight team, uh looking to really fund and accelerate the uh new research experiments and applications with real-time video AI. So if you are interested, please do reach out to me uh either uh here if you're in front of me or um or find me on uh on on Telegram or on Discord and I'll be more than happy to to uh give you some pathways of how you can contribute. I think that's time. Thank you very much. Over to you.