Watch your LLM live with Visdom: Release v0.3.1

Transcript

Hello everyone. I'm Vidan Sheni. I'm an open-source developer and a contributor with Fors Asia. And I've been contributing with Forasia for around 2 years now. For the past few months, I've been working on Wisdom. And that's what we're going to look at for the next 15 minutes. Has anyone here trained machine learning models? Oh, that's a lot of hands. Then wisdom, this talk is going to be very helpful for you. Now, what is wisdom? Wisdom has been around for many years. It is a visualization tool. Basically, we run our Python programs from our computer and then we can see them live on the browser using our own local server.

Wisdom is a popular project now. It has over 10,000 stars on GitHub. For some time the development had slowed down. This was initially started by Meta but then a team of contributors from Fishia community and beyond showed up. And after that a lot has happened. In September we released version 0.3.0 Z bringing together more than 3 years of development, 432 commits in which 24 contributors were credited and 23 of them were new. Hundreds of changes across the client, server, front end tests and documentation. And we did not stop after that. Of course, we have been

building since then. Additional 256 commits, 229 new PRs. Let me show you what we have been building. First, we have introduced experiment tracking, new visualizations like Sanki, Violin, uh, ROC and his 2D histograms, tables, image heat maps, a plug, pluggable storage layer. Oh, that was a huge piece of work that we really pulled off. then an open API spec and stronger password hashing and Python 3.12 as the minimum.

Moving on, while training your uh models or running your experiments, has anyone like opened a plot to uh two months later maybe and then they see oh why did this exactly skip here? No. Anyways, that is a problem that we have solved on your journey in machine learning and data. You will essentially get to that stage in maybe 6 weeks or so. Traditionally, wisdom was about plots.

You created an environment, sent loss curves, looked at them, but when you run many experiments, plots are just not enough. Which learning rate did I use? Which model? And why this run failed? Why did one specific run do better? So we added native experiment tracking parameters, metrics, tags and status all stored in the experiment. Now wisdom remembers context in which something happened. Once experiments carry structure, we can ask questions. Instead of opening environment after environment, we narrow hundreds of runs down to a few. Accuracy above 0.90, learning rate below 0.01.

And I want only the experiments that has the status finished. Anyone here used the hyperparameters dashboard? Maybe like the one in tensorboard. Oh, this will feel familiar. Now inside wisdom, this is one of the largest additions in the new release. The H params dashboard has many experiments together in one place. Sort and filter runs. Color them by a parameter or a result. Look for relationships with a scatter matrix. Explore many dimensions with parallel coordinates. Select runs and compare them field by field. If you need data elsewhere, just export it as CSV or JSON. Now, this changes wisdom

from something where you watch one experiment into something where you understand many experiments together. This is a video that I prepared for you guys. Now this is a sweep of 13 Lura finetuning runs. 12 of them are finished. One is still running. Parameters, metrics, tags. I sort by accuracy. Now the filters accuracy at least 0.90. Learning rate below 0.01. Only finished runs. 13 runs become five. Color the table by accuracy. Then the skater matrix. Every parameter against every metric for the survivors. Parallel coordinates. one axis per parameter and metric.

Brush the accuracy axis and the best run stays highlighted. Take three runs and open compare. Differing fields on top, identical fields below. Metrics shaded and the run that is still going is still going. Its loss curves keep extending one point per second. No reloads. Is anyone here familiar with Optuna? Oh, then you know how hard it is to read trial logs.

Here is how it looks in wisdom on your So once wisdom understand many experiments, the obvious next step is hyperparameter optimization. We added an optuna integration where every trial becomes a wisdom experiment and one wisdom builds an Optuna study dashboard. optimization history, parameter importance, timeline intermediate values, prune trials, contour plots, and a parto front for two objectives. We also want you to spend less time writing visualization plumbing. Wisdom already supported PyTorch and scikitlearn. Now there is a kas callback, a lightning logger and xibboost autologging plus optuna studies. Use the framework you already

use and let wisdom handle the visualization. The traditional Python client is synchronous. So each request can block the caller. The new release adds async wisdom. The same API used with async.io. Several visualizations sent concurrently. So training loops and other async code do not wait for visualization. We also removed a round trip. So that was a huge optimization that we got and a huge speed up. Before an append could only first ask does this window exist and then send the update. Now it sends the request and if the window doesn't happen, it is created and then it is updated.

Now this is the interesting part. Has anyone of you looked at a plot 2 3 weeks later and then see why did the learning rate or why did the experiment run jumped here? Yeah. So if not you will soon do this like in 3 to 4 weeks in this to this fire you will see why oh I'm now just scratching my head and uh I cannot figure out why why is this jumping suddenly out of here. So here is the fix a graph that's showing showing something has happened but a few weeks later nobody remembers why. So we click on a point and then add a note there. We switch the learning

rate and the explanation stays with wisdom attached to the data. It is saved on the server so it survives a reload. Visualization is not the end of workflow. A plot can be exported to PDF. Image exports as PNG, JPG, SVG or PDF with DPI metadata. And the table pane exports structured result as CSV or JSON from experiment to report without screenshots. A plain pyos training loop with wisdom's logger. No plotting code, train loss and learning rate appear and fills in as it trains. Above them, a scikit learn model autolog its parameters and curves. Now an async program one async io. Uh and

three windows appear together. Loss accuracy a sample batch annotations. A pencil click the spike switch learning rate. Reload the page. The note is still there. exports the plot as PNG at 300 dpi and as PDF and the run table as CSV and JSON now wisdom also looks different. Paints share a design system. The top navigation is reorganized. The environment looks clearer. The

connection state is visible. Uh you see the green pin online and the environment selector is cleaned up. Same wisdom way easier to read. Now not every improvement and every work that we have done deserves a beautiful screenshot, right? I mean that's just not possible. So, wisdom now save uh changed environments automatically instead of waiting until shutdown. Storage work move off the server's main input output loop. The server is modernized around async io with graceful shutdown. The front end moves to react 19. Cypress was replaced by playright and we run code security analysis and

there are more than 2,800 tests. A project like wisdom cannot move forward by adding features alone. We also need confidence that the existing workflows are still working. None of this was created by one developer. Wisdom is developed by an open-source community. Front end, Python APIs, ML integrations, testing, performance, documentation, bug fixes, reviews, and endless nights of work. New contributors keep joining and we have regular developer meetings where we discuss what is being built, review problems and decide what actually comes next for wisdom. A project that had slowed down is moving again because

people decided to work on it together. Now what would you want wisdom to support? According to us, developers should not decide alone what is we're going to put in a research tool. We want to meet the people who actually uses it. AI researchers, ML engineers, data scientists, students, people working with computer vision, people training large models, people running hundreds of thousands of experiments on their local systems.

Which workflows are still difficult? Which visualizations are important for you and we're missing those? Which integrations do you really need? While we add all of these capabilities. Our sole goal of Wisdom is it should be lightweight and it should be simple. So it is really simple. Now it's just three commands. PIP install Wisdom. No proprietary backend, no external account and your experiment data stays on your machine on your infrastructure.

Has anyone here ever created a server just to share their experiments and runs with their colleagues or teammates? Well, that is a problem that a lot of people go through. Here is the fix for that. Sometimes even running your own server is too much for your experiments. So there is one more project wisdom dev. You sign up, make a workspace for a project, invite your team as admins, members, viewers and give your training scripts and API key. Plots go into that workspace and nobody outside can see them. If you want to show a result or someone outside the team, you create a readonly link and for a limited time. It is hosted today at wisdom.dev. You can try it at the QR above.

So how how does it actually work? Your script talks to wisdom.dev with an API key. A small gateway checks the key, your role in the workspace and your plan and then hands the request down to wisdom. The changes inside wisdom are small. So each workspace gets its own space and none of it is switched on when you run wisdom. So wisdom says exactly the same. Just pip install wisdom. Trying it just takes a minute. Sign up at wisdom.dev. Make a workspace. Create an API key and point the client at it.

It is still early for wisdom dev. So please try it and tell us what you need. So everything I showed you today is going in the next release. It's almost 229 pull requests, a lot of commits, all these exciting features of native experiment tagging, H params dashboard, Optuna hype optimization and all of it. Now we have a surprise for you guys.

But for that I would need the wisdom team to come on stage. So I'd welcome Norad training Manika Jurya Rajnish Tari and Jant Parasher on the stage and of course Mario Bailing. But for this gift, we would be needing your help. Are you guys ready? Let me first explain what the task is.

Our team is going to countdown from 10 to 0 and after that we're going to make the next wisdom release right here on this stage. We need your help for this. We would want you to countdown with us so that we can make a successful release. Are you guys ready? the team. Are you guys ready? >> Yes. >> Is the video team ready?

>> Yeah. So >> 9 8 7 6 5 4 3 2 1 zero. Wa! Free. Welcome.

Now we'll just switch back to the last slide. Oopsies. This is just new for West. Now try the new release. Break it. Tell us what you need and we will make it happen. That is all that I have for you guys today. But we have a workshop at 11:00 and 4:45 just outside this builder's old stage. Now I would like to invite a dear friend of mine whom I met the at the foria summit. This is watching AI to hearing it live. I would like to invite Anav

Angar.