Building with Bazel

Episode 17 - Understand Distributed Builds

One of the coolest aspects of working with Bazel is that you don’t even need to build your app on the same machine. You can configure Bazel to do all of its building on an entire different machine.

In fact, you can use an entire data center to do your builds. Remember, this is where you can realize all those gains with Bazel.

By being meticulous with your dependencies and organizing your targets, your singular build can spread over multiple machines working unison.

Bazel also allows us to perform remote caching as well. Remember, when we declare a build, we have inputs and outputs. We can configure a server to store the outputs of builds. This means, if nothing has changed from a build target, Bazel can be configured to use that output. This can also drastically increase the speeds of the build. Being open source, there are lots of different from setting up your own server caching network or even using Google’s Cloud Storage.

For more information on remote caching, see Bazel’s article on remote caching (https://bazel.build/docs/remote-caching).

In this demo, we’re going to be using a service called Build Buddy. We’ll use it to configure our Joke Generator to perform remote building. We’ll use the Joke Generator since it’s an easy project so we can focus on the actual remote build.

Demo

To get started, open up a browser and head on over Build Buddy.

https://www.buildbuddy.io

Build Buddy allows a free account with limited build options. For example, this is only allowed for three users for open source or personal projects.

Create a new account with Build Buddy. Once you are up and running, we need to update our workspace. We need add some specific build buddy rules. Head over to the following:

https://www.buildbuddy.io/docs/rbe-setup

Scroll down to the BuildBuddy toolchain. Copy the code. Then paste it into your Workspace.

Okay, so we added some rules but we also need update our bazelrc file. Head back to Build Buddy and click their setup link. Copy the following:

build --bes_results_url=https://app.buildbuddy.io/invocation/
build --bes_backend=grpcs://remote.buildbuddy.io
build --remote_header=x-buildbuddy-api-key=hg9JT7rp1xBG8zj1QsWj

Next, we’ll add the following per the documentation.

build --remote_executor=grpcs://remote.buildbuddy.io
build --remote_timeout=3600
build:remote --jobs=1000
build:remote --host_cpu=k8
build:remote --cpu=k8
build:remote --crosstool_top=@buildbuddy_toolchain//:ubuntu1604_cc_toolchain_suite
build:remote --host_platform=@buildbuddy_toolchain//:platform_linux
build:remote --extra_toolchains=@buildbuddy_toolchain//:ubuntu1604_cc_toolchain
build:remote --nojava_header_compilation
build:remote --define=EXECUTOR=remote

Now back in the command line we’ll kick off the build.

bazel run //:knock_knock --config=remote

After some time, the build is compiled over at build buddy and now we get our knock knock joke. And of course, heading over to Build Buddy, we can see the breakdown of the entire build process. And of course, if we ever need the binary produced from the build, we can download from the artifacts section. As you can see, this is quite a powerful tool.