Building and maintaining a complex software system over time like Android (AOSP) can be a daunting task, especially when the system has many dependencies. Perhaps you can identify with one or more these maintenance challenges.
You need to fix a bug in an Android release you built months or years ago, but the backtraces are useless because you no longer have the symbols laying around anywhere.
Maybe you did have a copy of the symbols, but when you went to build and test it, the build broke because of incompatibilities with some host tool, or an important patch was missing.
Or you upgraded your build host’s OS, and now your Android build breaks.
Or someone broke the build recently, but you didn’t notice it for weeks because you haven’t been doing full repo syncs and builds, or you weren’t building that particular product.
Perhaps you’re unable to reproduce a build you rolled way back when, or maybe you just feel guilty for not having a well-controlled release process, and you finally decided that now’s the time to automate clean, reproducible builds for your release process with continuous integration.
A proper release process with continuous integration can solve all these problems, and containers are an important part of the solution.
Why build Android in a container?
There are several good reasons for building Android in a container:
consolidate all the build dependencies in one place
easily build on other machines, even those running different versions of the host OS (e.g., Android versions older than 2.3 need to be built on a 32-bit Ubuntu 10.04 host OS, which has reached its end of support)
expose hidden dependencies, like uncommitted patches in a developer’s source tree
version-controlled, reproducible build environment (e.g. via Dockerfile, etc.), which is necessary for e.g. continuous integration builds
it’s easy
How to build Android in a Docker container
Kyle McKenna’s AOSP docker container makes this easy:
$ mkdir -p ~/docker/lollipop
$ export AOSP_VOL=~/docker/lollipop
$ cd ~/docker
$ git clone https://github.com/kylemanna/docker-aosp.git
$ cd docker-aosp/tests
$ bash ./build-lollipop.sh
When it’s done, you’ll find the build output in the usual place, aosp/out/target/product.
This is an excellent starting point, but you probably want to build AOSP for a different product than the default specified in the script. Choose a different repo and branch by adding the following to Dockerfile:
ENV TEST_URL ssh://joe@gerrit.example.com:29418/platform/manifest
ENV TEST_BRANCH android-5.0.1_r1
Pitfalls and Tips
The example above works well when pulling the sources right from Google, but it’s a bit painful if you’re pulling from a repo that requires authentication.
ssh auth issues
Kyle’s ssh_config turns off StrictHostKeyChecking, which reduces security. If your container is only going to be connecting to known set of machines, you can avoid reducing security while still allowing the container to connect to other machines by pre-populating ~/.ssh/known_hosts with the right host keys:
Copy the known_hosts file into the directory containing the Dockerfile.Remove the following line from Dockerfile:
ADD ssh_config /home/aosp/.ssh/config
Add the following lines:
ADD known_hosts /home/aosp/.ssh/known_hosts
RUN chown aosp:aosp /home/aosp/.ssh/known_hosts
RUN chmod 0600 /home/aosp/.ssh/known_hosts
ssh agent forwarding problems
If the aosp script detects that you’re using ssh agent forwarding, it will helpfully bind-mount the ssh agent-forwarding socket inside the container. But if the aosp user inside the container doesn’t have the same uid as the user owning the ssh agent forwarding socket, authentication inside the container will fail.
The safest way to fix this problem is probably to make the aosp uid inside the container match the uid of the user’s ssh agent forwarding socket. You can do this by changing the line in the Dockerfile that creates the aosp user; just add --uid myuserid, as indicated below:
RUN useradd --create-home --uid 1234 aosp
However, this breaks the general principle of not making containers dependent on host configuration. If you know of a cleaner way to solve this problem, let us know!
Mirror AOSP locally
Sometimes network outages or rate-limiting will cause repo sync to fail. If that’s the case, or if you’re running repo sync frequently and want to be a good citizen by not hammering the upstream, create a local mirror of all the AOSP repos. Syncing against a local copy is very fast (if the machine holding the mirror is reasonably capable) and well worth the effort to get rid of repo sync failures.
Make the build clean
To guarantee that you really are building from scratch, make sure no build objects are being written to the source tree. AOSP builds cleanly, but often third-party ports sprinkle .o files hither and yon. Imagine .o files built with one toolchain hiding under the bushes from a previous build, and later getting linked against by a different toolchain or for a different product--not a pleasant thought, is it? Problems like that are not fun to find and fix, and won’t win you Employee of the Year.
Plus not only will you gain assurance that you’re getting a clean build, you’ll also be able to use OUT_DIR_COMMON_BASE, which is very useful if you have multiple Android source trees, or if you share source trees.
Often in third-party ports it’s kernel or u-boot builds that write to the source tree. Fix that by using O= to specify where build objects should be written; e.g.:
make -C u-boot O=$(abspath $(TARGET_OUT_INTERMEDIATES)/uboot)
If it’s too painful to prevent build objects from being written to the source tree, an alternative is to make sure that all those build objects get deleted before a build starts.
Summary
So building Android in a container can be a helpful part of a robust release process, enabling building to be easily migrated to other machines, fixing build dependency problems by consolidating the dependencies in one place, and version-controlled, reproducible build environments. Give it a try, and share your favorite tweaks with us in the comments!