Building Open Infrastructure for Real-Time Multi-User Control

Transcript

Now we are welcoming Miros for building onchain infrastructure for real time multi-user control. Um thank you for uh here being here and please continue with your thought. Thank you. Thank you very much for being here. So I'm going to talk about the open infrastructure for realtime multi-user control. Uh something about me. Who am I professionally? I'm working with video streaming engineering and I do it apparently also in my free time. I'm an open source developer and I'm going to uh show my open source project now for you.

Um so what is Neco? Uh Neco is basically a self-hosted virtual browser and it runs remotely. Um you can think about it as um having a virtual browser on um computer and multiple people connect to it and watch the same content at the same time. Uh everything is working through WebRTC. That means you don't need any um client software to access it and it has a really low latency. It's about multiple using multiple users joining the same session. So you can watch together synchronized the same content with multiple people. Uh there is a shared video and audio and the

interaction is also shared. And I'm going to talk about how the interaction part is managed for this project. Maybe you're asking yourself why does this even exist and what is it? There used to be a service called Rabbit five years ago but it completely disappeared. People used to watch movies together, browse internet together remotely. Um and since it disappeared there was no any alternative but people still wanted to do this thing. People still wanted shared browsing and people still wanted the shared interaction.

the problem never went away and there was no solution for this. So let's take a look at look at existing solutions what might be there. Uh you might think of screen sharing like zoom or meet or teams. Uh those solutions don't solve the problem that we currently have because uh they are only there passively. So multiple people can watch. There is only one presenter. There is no interaction. There is also no isolated environment. So everything that you are sharing is your screen. You can share your um tab or window. You also need the bandwidth to stream it um to the internet which requires another

uh upload. It's a simple setup. You don't need anything. It's widely available but it's not solving the collaboration. Another use case could be remote desktop. Uh there exist uh multiple protocols like uh VNC, RDP, we have team viewer or anyesk. Uh there is also no VNC or apach guacamole available on the web without installing any client software. They give users the full desktop access. It works with any application but again it's only meant for one user controlling and there is no support for multiple interactions. there cannot uh multiple people cannot connect there and if it would also allow it

there would be a chaos if they if everybody wanted to control at the same time. Another solution could be collaborative editors like Google Docs or Figma. Uh they offer native collaboration. They are really efficient on synchronization. They are built for massive uh editing but they are application specific. So there is no real-time media. You can only edit document that is available. There is no possibility to plug in any other application that has been built specifically for this kind of collaborative editor. So there is no support for arbitrary context.

The last solution could be cloud browser platforms. There exist some online like Hyperbeam or Berste. They allow isolated environment browser as a service online. Um they offer remote workspaces but they are managed infrastructure. Who owns the cookies of the browsers? Who owns the interaction? It's everything on cloud. It's not something that we can self-host. So that's their biggest problem. This is exactly where Neco is positioned uh and fills in the gap where we want money viewers. We need a shared interaction also um real-time media that is self-hosted

and offers explicit ownership. There is nothing that would be available. That's why Neco was created. So what is Neco again? Um it's a browser that runs in browser um it can be accessed through webtc and now we establish that it can it allows multiple users to control at the same time. Uh but there is catch. How can we create this uh without the chaos?

Because if we have multiple people controlling at the same time, it's a problem that needs to be solved and none of the previous solutions solve that for any application that is available. People use it also for not only for the total the interaction. There is like this watch party. So people watch videos together on Neco. Um they can do interactive presentations where people can actually um join the presentation and uh play themselves with the um application that is being shared. It's used as a collaborative tool. Um it can also support by teaching. So if somebody is teaching through Neco, it can give

the control to the other users and they can try it by themselves. Uh it can also be used for embedding anything on the web page. So if somebody wants to embed their application that is um vendor specific. It's not available for web. It can be streamed using WebRTC to the web page and the logic stays at the server and is never exposed to the client. So it's also protection for the theft. And it also can be used as automated browsers for AI tools that can as well interact with it the same way as people are interacting with it. And they it allows uh possibility to always have um user to join the session and do some tasks manually if it's not done well by the AI. So intervene

we still have the real problem even after all these use cases. So um let's say we have 20 people that uh want to use this tool. Everyone wants to click, everyone wants to um interact with it. What are we going to do? We have only one browser. Um even though we have many users, there is still only one state. We cannot have multiple cursors. We cannot have multiple actions happening at the same time. If somebody clicks uh on different things at the same time, who is going to win? which click is going to be then used. Uh there are can be a lot of conflicting actions and the collaboration would be then just pure chaos.

The wrong solution would be of course to accept all of these clicks to basically merge all actions and allow everyone to just click on the web page. We hope to resolve conflicts by uh accessing who was the first one to click and just hope for the best. But then we don't know who is actually controlling it. We don't know what's happening. It's it's the collaboration becomes just a noise. So that's where Neko comes in and introduces its ownership model. Um, Neco has one active controller that defines an explicit ownership and it uses transferable control thus creating deterministic behavior.

Means there is there are many users. There is one host token that can be passed between the users and who has this one token can actually control the browser. So we don't merge the intent, we serialize it. The architecture of Neco is as following. Um we use the capture layer to get the actual um stream from the uh desktop sharing. Uh then we distribute it to multiple people using WebRTC. Then we use the arbitration to decide who gets to control and after we have the specific user is controlling the host.

Then the one gets the execution on top of the layer and gets to control the browser. So let's look at the architecture. The capture distribution it's basically captures the desktop output. So we run the browser in um X11 environment with pool audio uh that is just grabbing the screen. GTMr is used as a frame buffer capture to get all the uh data from the screen. Uh the web RTC then is uh distributing this to all the users. Um we encode the media in H.264 or VP8 format because they are available on the browsers and users don't need to install

anything and can just open a web page and browse our browser. The web RTC was used because it's uh low latency with video and audio. I will get to that later so that everybody sees the same thing at the same time. But we could do it also using ALS or um uh FMPhere uh over um websocket. But why did we choose WebRTC? The WebRTC is a standard in um current browsers for video for uh ultra low latency video and audio.

Um all the solutions before also mentioned they might like the audio support but WebRTC gives us this ability to have also audio support and it supports adaptive streaming. So we can decide uh we have 20 users um connected and they have different bandwidth uh constraints and we can decide which one gets what kind of uh video that we encode on the server. uh WebRTC was built for the real time interaction. So it's a perfect fit for this project. The next level is arbitration which is the most interesting part. Uh it defines who owns who is the host the host ownership and it defines the whole

permission model. Um in this layer the control is being transferred from one user to another. Um there are multiple users and they can request who is going to get the uh they who is going to control and server decides who gets the control based on their um levels of um administrator levels or if there is a possibility to uh who is going to basically control the control flow is as follows. So the user can request control WebRTC data channel and uh server can validate if they get to control or not. So if there

is already a current user hosting they need to get approval and the host needs to give up their ownership and the other person can then take the ownership. If there is no other user and the uh current user is allowed to control um the control is granted immediately and the user can grab the control but on approval there will be a popup for the administrator that is currently hosting the session and they want to allow this user to own uh currently the ownership they get to have it but since they are not the administrator the administrator can take the control at any time they want. All viewers are notified of this change. That means everyone knows who is

the who is the host at the at every time. There is a log of events and it's absolutely uh it's audited who gets the control at which time. So we can uh trace in the back when there is some uh actions uh done on the server that were not intended. Uh there is a full log of who had control at which time. The last layer is execution. It's currently X11 server that is using the X test uh which is a uh specific uh library for the Xorg server that allows faking user input.

Uh there is also um another custom driver for touch inputs. So users with uh touch screens can as well access this um their events. P audio is used for virtual microphones and speakers and V4L2 loop back is for webcam pass through. So we can um actually uh connect the users microphone and uh webcam and the users can have the native experience as if they would have their browsers uh locally.

But the interesting part is that this execution layer is independent and is um actually just a interface that can be implemented by anything. So this leaves us uh with possibilities in the future to control anything as the system doesn't know that it's controlling a browser or another application or anything that is underlying it. That means it must not be only browsers what's supported. There can be full applications, it can be full desktops, some virtual machines, even like a robotics or I can also imagine drones that can be just controlled and um like pit cameras. There is literally no u

limitation if we set up the architecture correctly. This is an example of what already is implemented. Um most of the applications were are already coming from the um open source users by themselves what they saw that they want to use it for and um already implemented nice applications for Neco that fulfills their needs. Um that's actually um the main takeaway that I want to uh give you from this talk is that um this uh opensource project is there so that you can own the infrastructure not only the infrastructure and code but

also own the interaction which was not yet addressed in many of the opensource project that I looked through. um with replacing the execution targets, it can be forked and used for literally anything that the imaginations uh allows and the architecture is reusable. Um even though currently there is only a browser and applications, the architecture allows for anything in the future. Fork the browser, fork the desktop, fork the future. This fits the conference team.

So wrapping up, um, shared control is really hard. We went through existing tools that do not solve the problem. Um, I've shown how Neco is solving this problem explicitly. And the architecture that was created just because we wanted to watch videos together five years ago now extends beyond browser and allows for shared interaction that was not dreamed of before. when creating this uh project. Um thank you very much.

>> Very interesting session. I wanted to explore more as you said it's more than a browser. So >> yes. >> Yeah. And um any questions in the room? If not we can also continue after you can ask alo questions after the talk. Okay. Uh thank you so much again and uh thanks for u giving such a talk. Thank you. >> Thank you.