A Real-World WebAssembly Benchmark
PSPDFKit’s mission is to provide the best way to view, annotate, and fill out forms in PDF documents on any platform. In late 2016, we released the first version of our Web SDK, which relies on a server component to render documents. Only a few months later, we released an updated version of PSPDFKit for Web — one that doesn’t require a server component and instead uses WebAssembly (Wasm, WA) to render documents directly in the browser. This was a big achievement for us as a company, and being able to provide a browser-only solution that doesn’t require any backend infrastructure lowers the barrier to entry for our customers.
Performance of the WebAssembly code has been a focus for us since day one, and it’s amazing to see the entire industry move toward a shared goal: making WebAssembly fast. This starts with optimizing WebAssembly startup time, compiler improvements, and continuous browser upgrades.
A Real-World WebAssembly Benchmark
As part of our goal to make WebAssembly faster, we created a WebAssembly benchmark, which offers browser vendors a real-world, open source benchmark for WebAssembly based on PSPDFKit for Web.
The benchmark is available on GitHub and works with both customer and trial licenses. Browser vendors can reach out to us and obtain a more permissive license key so that the benchmark can run on different machines and even on their continuous integration servers.
Before we dive right into analyzing our results, we want to share more details about our test setup. One thing we’re trying to accomplish with this benchmark is to have a score we can use to improve the performance of PSPDFKit for Web instead of creating a microbenchmark. As such, the benchmark does not call directly into WebAssembly; rather, it also tests the bridge we use to communicate with it — an important part of PSPDFKit for Web.
Our WebAssembly payload is also rather unusual. Much of the size of our WebAssembly artifact is attributed to special cases in the PDF spec that might not be executed for every document. We want our PDF viewer to deliver the best results no matter the device. This is a factor where asm.js excels, as it only needs to compile functions when they are actually run.
After collecting the first results from our benchmarks, we reached out to the individual browser vendors to discuss these results and investigate ways to further improve the numbers. At this point, we want to thank Google, Mozilla, Microsoft, and Apple — all browser vendors were exceptionally helpful along the way and provided valuable feedback to help improve our benchmark.
Runtime performance of WebAssembly has always been great when using Google Chrome. We were only a bit disappointed to find out that the v8 team abandoned the plan to implement IndexedDB caching for WebAssembly.
However, after Google invited us to a call and shared some of the company’s upcoming plans, we’re excited about what’s in store. The biggest change in the near future will be the introduction of a new baseline WebAssembly compiler. You can already try this compiler today by running the latest canary build and enabling the
enable-webassembly-baseline flag. In our internal tests, we’ve noticed significant improvements in the total startup time. We couldn’t be happier to see this compiler being enabled by default, starting with the next canary version (M69).
Additionally, the team gave us a sneak peek at the upcoming alternatives for caching the compiled WebAssembly module so that you don’t have to recompile on every browser refresh.
Update August 2018: Google released Liftoff, a new baseline compiler for WebAssembly in V8. The announcement blog post includes specific benchmarks for PSPDFKit, showing a 56%+ faster initialization time.
Firefox delivered the best results in our benchmark, no doubt due to the release of tiered compilation, along with IndexedDB caching.
They also pointed out that after a very fast initial compilation, browsers that use a baseline compiler kick off a slower, more optimizing compilation in the background which continues to execute while we run other benchmarks. For this reason, we now benchmark initialization at the end to reduce the amount of background interference.
The performance of Edge was underwhelming at first. But with recent optimizations like inlining for WebAssembly, Microsoft Edge proves that the company is fully committed to making WebAssembly faster in the future.
Internet Explorer 11 is no longer supported in our Web SDK as of version 2022.5.
Update August 2018: The WebAssembly team at Microsoft worked with us to include optimizations for our WebAssembly benchmark in Edge 44, which is now accessible via the Windows Insider build. This results in both an impressive 2.25x faster benchmark run over the previous version, and a PSPDFKit score that is now on par with Google Chrome’s new baseline compiler. We’ve updated our bar chart to include these numbers.
We’ve found that Apple’s Safari’s performance is especially bad on beta versions of macOS, and we worked with Apple’s engineers to track down a recent regression (Bug 187196) where trace points turned out to be the expensive factor. Apple has also been exceptionally awesome in reacting to this bug, and the fix has already landed in master.
With this blog post, we want to say thank you ❤️ to all the browser vendors for their efforts to make WebAssembly fast and successful. We also want to communicate that our door is always open and we are here to collaborate with you to make our product and the web platform better!
If you like this article, be sure to also check out our others: WebAssembly: A New Hope and Optimizing WebAssembly Startup Time.
Regardless of whether or not you’re a browser vendor, please try out the WebAssembly Benchmark and feel free to reach out in case you have any feedback. We have an official Twitter account, and you can also find us on our personal accounts (@giuseppegurgone/@PhilippSpiess).
ALLOW_MEMORY_GROWTH=1. Because of the dynamic nature of PDFs, some of them might need a lot more memory than others. Our internal tests have shown that this option works best in our use case.