Continuous Integration for Audio Plugins. Tips, tricks, gotchas.

Continuous Integration (CI) is a concept with origins in the 1990s: every time you add or change code, the project is built and tested.

CI in the 2020s usually means you'll use a hosted CI service (GitHub Actions, GitLab CI, Circle) or you have some dedicated hardware on premises and use something like Jenkins. It builds and tests on every change (e.g. git push) to the codebase on all relevant platforms.

There's also Continuous Delivery (CD). In addition to building and testing, this "ships" everything that lands on your main git branch to customers. CD can help reduce the risk and friction that one accumulates holding on to big "atomic" releases (which take months to prepare and QA because there are so many changes going out).

CI/CD is has been the default in most modern web development and other software businesses for a couple decades. Audio, of course, has been a bit laggy with best practices. But it's catching up! Larger audio businesses like Moog use CI and many indie devs like myself consider it an ideal way to streamline releases.

In audio development, one tries very hard to not break customer projects or ship breaking changes — getting closer to this ideal is why CI/CD exists.

For plugin devs, running things like pluginval as a part of your build process is extremely valuable.

Why bother automating builds?

Here's what I personally like about running CI/CD for an audio product:

• Creating a new distributable release across multiple platforms is as easy as git push*. This reduced friction means it's trivial to cut and publish a new release. 5 times a day if I like! In other words, releasing is low friction and doesn't chew up any of my day to day time.
• I don't need to physically jump between Windows and macOS machines to create builds. I have some baseline confidence that if it built and my tests pass that I haven't broken my Windows builds, for example.
• Production release builds happen automatically in a stable, reproducible environment. Builds going out to users are never made by my local user on my developer machine. That means my ability to ship does not depend on my daily driver's current state. This also means my local user can be used for testing installation, code signing (which means I can test and debug installation issues).
• As I build, I've slowly built up a test suite, increasing confidence that my framework and dsp are behaving correctly. Running it through pluginval gives me confidence that DAWs won't choke due to a recent change.

* Most of the time. Sometimes tests fail or there's a problem with the build pipeline. Being notified about this immediately is a feature!

Who should be using CI?

I'm a solo developer (with a single product in beta), but audio companies big and small use CI. If you are on the fence, I'd recommend it if:

• You ship a lot of plugins with shared functionality that needs to be well tested.
• You are iteratively building a complex dsp framework over time and you want to make sure things don't break.
• You are publishing open source (increases confidence and adoption).
• You have the goal to release more confidently and/or more often.

One plugin team of 2 I interviewed said:

We put a lot of effort upfront into our CI. We were able to fix things quickly and put out updates. This is part of thinking about the user experience. User experience isn't all about making nice buttons that are easy to use. It's thinking about how quickly you can react to a frustration of the user.

Geert from Moog told me:

Everything that my team works on is continuously integrated. We have as many tests as we can. We try to separate unit tests, functional tests, integration tests, system tests. I do approach all of it as if it's an enterprise web project, where we have to be thinking about releasing every day. Even if we don't release every day, everything always has to be stable.

What are the costs?

Maintaining your CI pipeline is definitely an investment. It has some real costs, in particular the setup and "getting things green" to begin with.

You'll need to learn git and be comfortable using branches. You may have to learn how to do some bash scripting or brush up on your CMake.

If you are working with a platform like GitHub Actions, you should definitely expect to spend some hours wrestling with pushing builds, waiting for them to fail, reading logs, wash, rinse, repeat. It can feel... unideal at times.

The plugin team of 2 I interviewed recommended allocating 10% of one's time on the CI pipeline.

That probably sounds like a lot. The bigger truth is that in reality, shipping confidently eats a lot of time no matter what. That's true whether you run CI or just have a bunch of scripts locally, or even a build.txt that you are copying and pasting commands out of for each release.

The difference with CI/CD is that you are investing that effort up front, out of band, to enable you to ship efficiently in your day-to-day. You gain consistency, risk-reduction and can easily reuse the pipeline for many projects. Removing the human element removes a common source of error and time, especially when stressfully trying to ship a last minute fix. When that critical fix is needed, it can be a git push away from the customers' DAW.

Building in CI is a gateway drug to testing

If you don't already have any tests, that's fine. Testing can be hard to prioritize:

• Shipping cross-platform audio applications is already too much work for one or two humans. Testing feels like "yet another thing" that would be ideal, but is tough to prioritize in the face of reality.
• JUCE, as a framework, runs tests and builds against many versions platforms in their CI. However, the framework is vast, coverage is light and much of it wasn't built with application testing in mind. This can complicate testing JUCE applications (looking at you, apvts). For example, I only know one person successfully doing view testing with JUCE (though I do know a few people who have tried, and I do know people doing view testing with WebViews).
• Audio development generally makes heavy use of multiple threads (real-time thread, message thread, etc) which are classically difficult boundaries to test interactions between. You should never really need to do this (unless you are writing your own framework.)

Once you are building in CI, it's trivial to toss in a few basic tests using a framework like Catch2 that will run on every git push.

The benefits are:

• Writing tests clarifies intent. It separates understanding the problem and writing the code. Once you can correctly describe correct behavior, writing or fixing behavior is often the easy part.
• Preemptively catching bugs. I can't tell you how many edge cases I've caught simply because the tests isolates one particular function.
• Regressions stay fixed. If a bug in the dsp is identified, you can write a test for the correct behavior, then fix it. From here on out, that test being green ensures your dsp is behaving.
• You can build up a suite slowly over time, which builds your confidence to ship new code over time.
• Refactor safely. Once you have tests, it behaves as not only documentation, but a test harness for ensuring your refactored code isn't breaking things elsewhere across your app.

I wrote more about testing dsp code on my blog. Or watch this great talk from Jatin from ADC:

CMake or Projucer?

As the maintainer of a CMake template, it's probably obvious that I think CMake is the ideal route to get started with in CI. I wrote an intro to JUCE and CMake here, if you are completely new to CMake.

With CMake, throwing something like this into GitHub Actions is a good start:

cmake -B builds . -DCMAKE_BUILD_TYPE=Release
cmake --build builds

However, many companies use Projucer (JUCE's custom build tool app) and also run a CI/CD pipeline (including JUCE themselves).

The Projucer is actively maintained and kept at parity as a build system with CMake. It has a command line interface which you can explore with the --help argument.

./Projucer.app/Contents/MacOS/Projucer --help # macOS
./Projucer.exe --help # Windows

You can do things like call --resave to export a JUCE project to a platform specific project such as an Xcode project (which you can then build via xcodebuild.) So this route is usually a bunch of scripting of Projucer and native build tools.

What happens in CI?

You can do a lot of automated things in your build pipeline:

• Build on all the platforms you're shipping to.
• Test on the platforms you care about. For example, you might want to run some basic sanity tests on macOS going back a few years.
• If you are a plugin, run pluginval on each platform you care about.
• Create installers, codesign and notarize.
• Automate product releases. For example, it's easy to ship to Moonbase from GitHub Actions.

GitHub Actions Gotchas

I've worked with many CI services. GitHub is the one I currently use for Pamplejuce (my open-source project template), my other open source projects as well as private projects.

The good:

• It's free to get started, 2000 free CI minutes a month
• You can run "self-hosted" runners on your own local machines (for free).
• There's lots of examples and resources available. See the "Templates" section of my directory of open-source JUCE projects for a taste.

The bad:

• Building on macOS eats 10x the "minutes" from the budget they give you. So one actual macOS minute is equal to 10 "minutes". Given that a typical build, test, installer creation and notarization might take up to 15 minutes, that's only going to give you about 10 builds a month before you start paying money.
• Iterating against the platform and waiting for builds is.... undeniably awkward and annoying. You push, wait 5 minutes, and then something either works or doesn't and you have to push to try again. This can feel frustrating, but once things are setup, you usually only need to tweak things when environments change.
• GitHub maintains "runner images" for the last couple releases on each platform, but they might only be a subset of the OS versions you support. Changes to these runner images can sometimes break your builds (especially on Linux for some reason).
• Self-hosted feels awkward for multiple projects — you'll need to install a whole separate "runner" per project.

The AAX gotcha

As mentioned in my Code Signing roundup article, to sign AAX binaries you'll either need physical access to the machines or you'll need to sign up for PACE's cloud signing.

Devs sometimes see this and think "that means I can't use hosted CI like GitHub Actions" — personally, I don't find that an acceptable trade-off. A single plugin format (which for many, will not constitute a majority of revenue) should not dictate your entire shipping strategy and developer experience.

Here's the options as I see them:

• Pay for PACE cloud signing. Obviously, your expected revenue from AAX has to clearly outweigh the costs. It's at least worth getting a quote. This is probably smartest if you are an existing business selling AAX, moving to CI.
• Use self-hosted runners on GitHub. I do this for macOS (and have decided not to ship Windows AAX until there is significant demand).
• Setup Jenkins locally to build AAX on Windows and Mac.
• Do you need to ship AAX at all? The answer is obviously a resounding yes if you are targeting professional mixing and mastering engineers and expect ProTools users to be a big segment of your purchasers. If not, you can recommend your buyers use a plugin wrapper like Modalics' excellent Plugin Buddy if they want a free AAX wrapper.

Other tips

• Don't go insane: It's easiest to get one platform "green" in CI at a time. For example, if you are trying to get notarization working on macOS, disable Windows builds until macOS is happy. This feels clunky, but is pragmatic. You don't want to be waiting for builds you don't care about (I wish GitHub provided an easier way to just select one built at a time!).
• Don't go insane: Building C++ applications can take 5-15 minutes (depending on the state of the cache). Iterating against CI (changing one thing, pushing it) can take time, so while setting up, it's best to have some other lightweight task to chew on to wait for your verdict.
• BTW, JUCE 8 recently got better support for Mozilla's sccache which will shave minutes off your build times.
• VS Code comes with GitHub Actions yaml validation out of the box. Save roundtrips to hosted CI by validating syntax, etc.
• If you ever build and run Debug in CI, you'll probably see hangs on assertions.
• On paid platforms like GitHub Actions, I recommend being defensive and adding timeouts to steps — you don't want an API (e.g. codesigning, notarization) stalling out and costing you minutes. On GitHub Actions, you would use timeout-minutes.
• There's more tips for JUCE CMake scattered around my manual for Pamplejuce.