Using AI to streamline browser performance testing.
AI coding tools have made it easier for developers to experiment with software.
However, REDspace Principal Developer Jamie Stackhouse had become frustrated with how often those experiments ended. “So much of the AI code people make ends up in a repo, and nobody uses it,” Stackhouse says. He wanted to try something different. Rather than use AI to build another piece of software that might never make it into production, Stackhouse was interested in using it to improve engineering processes that already needed to happen.
In particular, he wanted to automate some of the tedious work around performance optimization so developers could spend more time testing ideas. That led him to build bperf in his spare time. Bperf is a browser-performance benchmarking tool designed to make rigorous performance testing faster and easier. It runs benchmarks across Chromium, Firefox and WebKit while checking that changes to the code continue producing the expected result.
Stackhouse initially built the tool for himself, then shared it with other engineers at REDspace. The project quickly prompted conversations about other places where the same approach could be useful. For REDspace, that makes bperf an interesting example of the kind of technical experimentation our engineers bring back into the company.
Performance improvements you can trust.
Coming up with an idea that might make a piece of code faster can take a few minutes. Proving that it actually worked can take much longer. Imagine an engineer changes a function and a benchmark shows a 5% improvement. That difference might be real. It could also come from background activity on the computer or normal variation between benchmark runs. The engineer needs enough repeated measurements to determine whether the improvement falls outside that noise. They also need to confirm that the code still produces the correct result.
Browser differences add another complication. An optimization can behave differently in Chromium than it does in Firefox or WebKit. “You should start it up on three different browsers,” Stackhouse says. “You should run enough trials that you know what the median is. There are all these statistical things that you should do. But to do it manually takes hours.” That can turn a quick performance experiment into a much larger piece of work.
Before running the target benchmark, bperf calibrates itself against an anchor test with a known target execution time. Based on how quickly that test runs in the current environment, it determines how many trials should be collected for the benchmark. That gives developers more confidence that a small performance improvement represents an actual change rather than normal variation on the machine.
Building AI into the optimization loop.
Stackhouse designed bperf so an AI agent could work inside the loop. The agent can inspect previous results and form a specific performance hypothesis. It can change the code being tested and run the benchmark again. With a measured result while checking that the expected output hasn't changed. If the optimization works, the change can be kept. If it doesn't, the agent can try something else.
That keeps the feedback loop relatively short while preserving the testing discipline required to evaluate each idea. For Stackhouse, this was where AI became particularly useful. It could help remove some of the repetitive work surrounding A/B experimentation while leaving engineers to think about the underlying performance problem. The same approach also makes it easier to explore grouping small optimizations that might otherwise seem too small to justify hours of manual benchmarking.
Performance matters on connected devices.
Performance optimization is familiar territory for engineers at REDspace. Our teams work extensively with connected-TV applications running on televisions and set-top boxes. Many of those devices have considerably less processing power than the computers developers use to build and test software. That changes how engineers think about seemingly small inefficiencies.
An application running at 60 frames per second has roughly 16 milliseconds to complete the work required for each frame. Media processing has to fit within that same time budget as the rest of the application. Code that executes in a fraction of a millisecond on a powerful development machine can take much longer on constrained hardware. Stackhouse estimates some operations can run 40 to 50 times slower on a set-top box.
Once enough processing accumulates, an application can start missing its frame budget. That can show up as dropped frames or an interface that feels slow to respond. Stackhouse already knew from previous work that several sections of the HLS.js media-processing pipeline had opportunities for optimization. As an open-source library used for video playback in the browser it made a useful place to put bperf through a real-world test.
With the new tooling in place, he could investigate those opportunities without turning each idea into hours of manual benchmarking. The resulting work produced four pull requests covering areas including subtitle parsing, MP4 box parsing and MPEG-TS to fragmented MP4 processing. Those changes were accepted into HLS.js and published in version 1.7.0.
That gave Stackhouse something he had been looking for when he started the project: AI-assisted work that made its way into software people actually use to make a measurable difference. It also gave other engineers at REDspace a new tool to think about. Internally, the conversation expanded to other places where the same approach could help. The underlying idea isn't specific to HLS.js. Anywhere engineers need to improve small, deterministic sections of browser code, the same benchmarking loop would make experimentation easier.
