Are AI models running directly in the browser finally good enough to use in production? Let's find out! 😉
It's been a week since my talk at FrontKon in Prague. As I mentioned before, it was fantastic! I'm adding a few photos below, just to prove that I actually speak at conferences and don't just make these things up. 😀
My talk was titled "Your Browser Is a Compute Platform (And You're Using It for Forms)", and I showed how we can use WebAssembly and WebGPU to perform advanced computations directly in the browser. I was a little concerned to discover that, out of everyone in the audience, maybe three or four people had even heard of WebGPU! And that's a shame, because WebGPU can accelerate certain types of parallel computations by tens of times compared to CPU-based implementations.
But okay, at least the people at FrontKon know what WebGPU is now. 😁 And if you'd like a talk about this at your conference, feel free to reach out! Especially if your conference happens to take place in a particularly attractive tourist destination. 😀
After a conference, I traditionally prepare a blog post related to my talk. However, since I've already written about WebAssembly and WebGPU many times, today I'd like to approach the topic from a different angle. Namely, let's take a look at the current state of AI models running directly in the browser, which, of course, make extensive use of these technologies. 🙂
Browser-based AI models are a fascinating concept. No inference API calls. No network round trips to an AI backend. No worrying that your users will consume too many tokens, because you're not paying for them! But of course, local inference still takes time, and downloading the model in the first place can take a while.
Another advantage is that these models can work offline once they've been downloaded and cached by the browser, provided the application itself supports offline operation.
And that also brings a significant privacy benefit: inference happens directly on the user's device, so their data doesn't need to be sent to an external AI service.
However, as you can probably imagine, these are relatively small models. We're not going to run a massive frontier-scale model in a typical browser tab. 
Some of you may remember that I experimented with browser-based AI models about a year ago. Back then, we were mostly laughing that they were a fun curiosity, but not much more than that.
But I've been following people from Hugging Face who work on Transformers.js on LinkedIn, and every now and then, their demos would appear in my feed. At first glance, some of them looked genuinely impressive. You know, practically spaceships running inside a browser. 😉
So before FrontKon, I decided to take another look at what they had to offer. And it turned out that after upgrading Transformers.js to version 4.3.1, which I used for these experiments, and trying some newer models, the results were genuinely impressive!
Of course, Transformers.js isn't the only option for running AI models in the browser. There are other libraries and runtimes, but I really like Transformers.js, and I think it's a pretty representative example of what this technology can achieve today.
At the same time, I don't work for Hugging Face, and nobody is paying me to sell you a spaceship in your browser. So I'll tell you what browser-based AI models can actually do today, what works well, and what still doesn't.
If you like the project, please give it a star on GitHub. I'm collecting them! 😀
For these demos, I wanted to focus on practical applications. More specifically, I wanted to find out how we could use browser-based AI today, right now, in real production applications.
I probably wouldn't build a mission-critical system that depends entirely on these models working perfectly. 😉 Instead, I'd treat them as interesting, optional enhancements to existing application features, especially in areas like e-commerce, accessibility, and customer support.
The application itself is nothing particularly exotic, by the way. It's a regular React application written in TypeScript, using Web Workers to run model inference without blocking the main UI thread.