Not all bytes are equal

Popcorn is a library that lets you run Elixir in the browser. It uses OTP/BEAM configured to work in that environment.
We recently released 0.4. I encourage you to read the release blog post as well.

First rule of optimization: do less.

I want to talk a bit about optimizations I made for Popcorn, which is a library for running OTP/BEAM in the browser.
We started with 25.9 MB of uncompressed assets, of which the VM itself took 3.25 MB.
After implementing them, we got down to 10.7 MB (with a 2.44 MB VM).

I tried to include explanations for people who aren't really accustomed to low-level BEAM and C work. Hopefully, you'll get more out of this than just reading about -X kB removed from the VM binary.

With that said, let's start.

Prelude: structure

Popcorn consists of a few things:

• beam.wasm, which is the actual VM running Elixir code.
• beam.mjs, which loads beam.wasm and sets up the things that Wasm needs (a filesystem, console output, ...). Hereafter referred to as "glue JS code."
• Popcorn JavaScript sources (JS API, Web Workers entry point).
• Tarballs with .beam files, including the standard library (stdlib) and user code.

We're mostly interested in the Wasm binary and stdlib tarballs.

All benchmarking was done on my M3 Mac, running in Playwright's Chrome browser. Time benchmarks were run 30 times to account for noise, with both cold and warm runs.

Standing on the shoulders of giants

There's a blog post from the Grisp folks on optimizing OTP for microcontrollers. I took their advice and stole their config for building OTP :)

It gives us three things:

• We don't waste time building things that aren't needed in the browser.
• We don't need to adapt these things to work there.
• It makes the build itself smaller.

We dropped the WX bindings, FTP, Observer, etc. from the start. They're controlled through switches such as --without-wx.

This was the first step toward having a buildable VM, so I don't count it as an optimization but rather as a starting point.
The build at this point was 3.3 MB. Good for a proof of concept and ripe for picking low-hanging fruit to make it smaller.

Compilation switches

These 3.3 MB were with us for quite some time. I was busy moving features and ensuring that the TypeScript part of Popcorn was working correctly. When taking a break from that, I looked into Emscripten optimizations. Maybe there were some things we had missed?

One of the first, almost free, things to check out is the compiler flags. OTP/BEAM (or rather, ERTS) is a C project [1]. C compilers have flags to control how they should behave when compiling C files and how to combine them (i.e. linking).

I present to you: the -Oz switch.

In the case of C compilers, it tells them to optimize for size (you may know the -Os switch; -Oz is more aggressive). In the case of Emscripten, it enables Wasm-specific optimizations that also affect Emscripten's glue JS code.
We enabled link-time optimization (LTO), reduced the number of Emscripten APIs used, and minified the JS glue source too.

Together, those optimizations reduced the Wasm and JS glue size by 435 kB. The JS glue itself dropped from 307 kB to 72 kB.

Brief, simplified explanation of LTO: C compilers go through all .c files (compilation units) independently and optimize within those units. When a function from one unit is used in another, it often makes optimization impossible:

• We only have the signature of the function being used, without its implementation.
• We don't know all the call sites.

The linker is where units are assembled into a single binary, so that's where LTO happens. It hands the full code back to the compiler, which then performs final optimization passes.
This process is quite slow and costly in terms of CPU and memory.

It's similar to having a single .c file, but not quite (visibility rules, etc.).
That's why some projects combine all their files into one big compilation unit to enable better optimizations (e.g. SQLite's amalgamation), and some programming languages use a single compilation unit by design (e.g. Zig).

--without-*

You can't waste time and space on things that aren't there in the first place. OTP/BEAM has some features that aren't really needed in browsers or don't work in them. In particular:

We mostly added some #ifdefs to the C code and stubbed out implementations.
The patches themselves are mostly only a few lines of code, yet they save significant space—collectively, this was around 500 KB, with the biggest savings coming from Zstd (340 KB).

Undefined behavior or "how I found an OTP bug"

Background

Wasm as an architecture is very thoroughly designed. It has very compact bytecode, few data types, and a pretty strict calling convention (i.e. where you put a function's arguments when calling it).

On x86 and ARM, when you call a function in C with the wrong number of arguments (or arguments of the wrong type), in most cases nothing happens: the caller puts the arguments in a few registers and cleans up after the call. If you use too many or too few registers but the function doesn't use them, it's fine.

C compilers complain when you do that with direct calls (i.e. calling f() when f expects an argument), but not if you call it through a function pointer. You can add a flag to the compiler to detect that, but it's unset by default.

In Wasm's case, a mismatched number of arguments when calling a function means a runtime crash.

Amusingly, the wrong argument types are typically fine, as most things become an i32 value anyway.

Emscripten developers were kind enough to provide us with a flag to work around mismatches: EMULATE_FUNCTION_POINTER_CASTS. It creates wrappers for all indirect calls, avoiding this issue entirely.

All is well unless you measure the size of the .wasm file: it added 95 KB in our case.

Foreground

As you may suspect, OTP/BEAM had a few places with this mismatch, although all but one [2] were related to incorrect data types. This is easily detectable, so I created an issue with a reproduction, and it's already fixed in the OTP repository.

That's 95 KB saved, and we have one less bug in OTP.

This was the last optimization related to the size of the binary, which now stood at 2.44 MB.
One nice side effect we got from it: OTP now boots 130 ms faster (from 408 ms to 283 ms). This likely comes from browsers having less to do when parsing and executing Wasm.

Now, let's focus on the standard library files (.beam).

Standard bignary

Elixir and Erlang stdlibs are pretty extensive, which in my Popcorn-optimization language translates to "do we need all those bytes?"

Naturally, we'll try to trim them down.

Storage

Typically, an application (e.g. elixir or kernel from Erlang) consists of multiple .beam files and a manifest/metadata file that tells us what other applications it needs to work, what files it contains, its name, and so on.

We decided to keep all applications as they are, packing them all into tarballs for reasons that will be explained shortly.

Bytecode format

Let me introduce you to the .beam bytecode format:

Spoilers!

It's an IFF file that consists of multiple chunks, each of which has some metadata (e.g. size) and content.
For example, the atom table contains UTF-8-encoded strings [3] for all atoms used in the module, with metadata telling us how big the chunk is and how many atoms it stores.

.beam files have some chunks that are required (exported functions, functions we call from other modules, strings, module code itself) and some optional chunks.
We'll be focusing on optional ones.

Most of them are removed by calling :beam_lib.strip/1; we additionally remove the Type chunk.

Dbgi

This chunk contains language-specific debug information, which can include the Erlang Abstract Format and information used to reconstruct the source. It is absolutely not needed in the typical case, so we remove it.

LocT

Local function table. Erlang has a concept of "local functions," which are basically non-exported functions, and "external functions," which are typically functions called by their module-function-arity (MFA) form (e.g. {Enum, :map, 2}).
The difference matters for hot code loading: external calls always go to the newest version of a module.

This chunk is informative only and used for introspection; it isn't needed at runtime.

Attr, CInf, ...

• Module attributes: behaviour, on_load, and user-defined attributes. Not needed at runtime.
• Compile info. What compiler version was used, source paths, etc.

There are a number of custom chunks that are added by tools or languages like Elixir (e.g. the ExCk chunk). They are not needed at runtime.

Some of these chunks are used when calling module_info/1. This unfortunately breaks some calls to it, but it's an acceptable trade-off for what we're trying to do.
In Popcorn, stripping is controllable by the user, so if you want to trade some bytes for this functionality, strip: false away!

Type

The OTP/BEAM you run on your computer or server most likely uses JIT. While Erlang and Elixir are dynamically typed languages, JIT benefits from type knowledge—if a variable is always a small integer, it can emit faster machine code.

Popcorn doesn't use JIT but rather an interpreter, so it has no use for that information. Removing it saved 165 kB.

Stripping

Erlang provides :beam_lib.strip/1, which keeps only the chunks the VM needs. We additionally remove Type. Stripping takes stdlib .beam files from 21.4 MB to 7.7 MB, and removing Type gets it to 7.5 MB (-65% overall).

Tarballs

As mentioned, we pack applications into tarballs. After doing that, they can be thought of as concatenated files. This approach makes them an excellent target for compression. Repeated atoms, strings, and parts of instructions can be deduplicated—we just use an off-the-shelf compression algorithm.

We observed that Brotli-11 performs the best, and it's widely supported by browsers. We also offer more traditional gzip compression.

While implementing this, we got bitten by :beam_lib.strip/1: it compresses the module afterward, which is undocumented. Already-compressed data makes compression algorithms very unhappy, and fixing that one mistake gave us a 30% improvement (from 3.93 MB to 2.66 MB).

To recap, we're at a 2.44 MB VM and 2.66 MB of standard library files. It was finally time.

Boot time

The last thing I was interested in improving was boot time. We started with a baseline of 460 ms, which is quite slow.
The size optimization did save us some time (down to 340 ms), but there was more to squeeze out.

Threads

After running benchmarks, it turned out that creating Web Workers on the fly is slow. We can create a pool beforehand to save some time.

Emscripten already does that for us (though there's no pool by default), so it's just a matter of finding the optimal number of workers.

BTW: we need Web Workers to emulate the OS threads that OTP uses for schedulers—they run your Elixir processes.

Popcorn needs eight threads:

1. The main thread, running OTP's main(). We don't want its execution to block the browser, so it runs in a dedicated Web Worker.
2. The bridge between JS and the VM.
3. Schedulers for running your Elixir code (take a process, run it for some instructions, run another one). We control how many we want to run; by default, it's just one.
4. A "dirty" scheduler for CPU work (native code doesn't give control back as often as Elixir processes, so it runs separately).
5. A "dirty" scheduler for I/O.
6. The drivers' async work thread.
7. An auxiliary thread (runs GC, timers, ...).
8. I/O polling (standard input and such).

...so we run exactly eight Web Workers, which was much better for VM startup time.

Checksums

When loading stdlib modules at startup, OTP also calculates their checksums. The checksum is mainly used to identify anonymous functions: OTP can have two versions of a module loaded (the module-swapping feature), and you want to know which version you're running.

Base OTP uses MD5, which is a defensible choice since nodes are typically booted once, MD5 has worked historically, and it's deeply embedded in the term serialization format. For Popcorn, this meant 20 ms spent on checksumming.

We replaced the algorithm with XXH3, which is a modern hash algorithm. It decreased the entire checksumming time for all standard library modules to less than one millisecond!

This breaks distribution (if one node runs MD5 and another runs XXH3), but fortunately, we don't have distribution enabled anymore.

Always measure the fish

That's basically it for today. Behind the scenes, I explored more optimizations than I mentioned in the article, but they either didn't improve things much or even made the size or speed worse.

The final table looks like this:

That's quite good if you ask me.

BEAM has more to offer when it comes to improving size and speed, but time is finite and you need to pick your battles.

If you want to check out more articles about Popcorn, here's a link.
Take care!

PS: packing light

This article was sitting in my drafts for a while and we did some changes to bundling in that time.

For example, we started including /priv that, in deps, sometimes contains native deps (which we can't use) or Dialyzer cache (faster static analysis after installing the dependency). Mateusz found and fixed that.

We also removed -export-dynamic flag which is connected to dynamically loaded NIFs which we don't include. This saves ~9% from VM size and 200kB from JS glue.
Compressing all the assets and removing one unnecessary copy of .js source also helped.

1. C++ if you count JIT implementation.↩︎
2. The problem was in driver [4] callbacks: internal drivers' callbacks take an additional argument that was ignored when calling them.↩︎
3. It can also be Latin-1 encoded (with an "Atom" or "AtU8" chunk name, depending on the encoding).↩︎
4. Drivers are extensions to the VM that allow you to react to events and work with them at a low level (I can add a footnote to a footnote?!).↩︎