Verification Builder¶

Reproducible Builds (RBs) mean to confirm that an APK was really built from the source it claims to be built from. As we cover only open-source apps, this source is publicly available. So if we build an APK from the very same source, and apply the signing blocks from the authors APK, we should end up with a byte-identical APK. For this purpose, at IzzyOnDroid we use rbtlog, which is described in this document. Hints for developers on how to best help their app achieving RB can be found on this page.

Verification builders¶

While IzzyOnDroid runs multiple „verification builders“ themselves, those are not bound to our project, nor do they interact with it from their end. Basically „everyone“ can run such a builder. Each maintainer of such a builder is free to choose which apps their builder covers. Those builders then provide build recipes as well as complete logs of the builds performed, plus an index summing them up. These indexes then can be integrated with the IzzyOnDroid repo browser (which covers multiple F-Droid compatible repos) to show the results. Support for client apps is planned, as is visualisation (for the latter, also see rbtlui below).

the scripts in rbuilder_setup help you set up your builder by automating installation steps. Adjust the config, run the script – have your builder ready within 5 minutes!
Preparing Recipes: describes how to set up your build recipes.
Debug failed RBs: gives you some hints on how to analyze (and hopefully fix) failed RBs with the tools provided by rbhelper.

Setting up a verification builder¶

It is highly recommended to set up your builder in an isolated environment, such as a VM (preferably an ephemeral one) provides. Sure it would work on your own local machine without a VM, but for security reasons you then should only try rebuilding apps from developers you trust. Of course, signing your commits should happen outside the VM (do not give the VM access to your private keys). Also see the security section in the rbtlog readme.

If you run a Linux system based on either Debian, RPM, or Arch, we recommend you to take a look at our rbuilder_setup repository. The scripts provided there require you only to adjust a config file and run the setup script – which will establish your builder in usually less than 5 minutes. For other systems, you'll need to adapt the preconditions from the 01_sudo_requirements_*.sh scripts there; the remaining scripts should be system agnostic (as long as it is a Linux/Unix system and the requirements from the first script are fulfilled).

So far we've got success reports from builders running on Debian, Linux Mint, Fedora, and Arch Linux.

Workflow¶

The following just describes the basic steps as performed with rbtlog directly (rbhelper provides wrappers and more to ease those tasks), and based on the example config provided by rbuilder_setup. For more details, please refer to the README.md file in your rbtlog root.

Note that whenever you want to operate rbtlog, you will have to activate your venv (see above) using ${HOME}/venvs/repro-apk/bin/activate – and deactivate it when done (this is one of the things rbhelper automates).

The following examples reference the example setup as described in rbuilder_setup, with e.g. branch names prefixed by the builder name (here: fraggle).

Testing builds (without writing logs)¶

While adding new recipes (or fixing those that broke), you will want to just build and investigate the output, without generating the verification logs. You can do so utilizing scripts/build.py for this:

cd ${HOME}/builder/rbtlog
python3 scripts/build.py --keep-apks ${HOME}/builder/rbtlog/apk -v podman org.fossify.clock:1.0.0 |tee z.log

This would look for recipes/org.fossify.clock.yml and expect a tag 1.0.0 to be defined there. It will then build the APK based on the recipe specified there. You ideally do that either in a separate testing branch, or in a copy of your rbtlog tree. Whichever way you choose, make sure to not accidentally check things in that weren't supposed to.

The --keep-apks parameter is optional, but you will want that for debugging of failed RBs. Just delete the APK files once you're done. With the above command, a log of the entire process (not to be confused with a „verification log“) will be saved to z.log for your investigations.

Using rbhelper simplifies this to:

rblog build org.fossify.clock:1.0.0 |tee z.log (to build the unmodified recipe), or
testbuild org.fossify.log 1.0.0 |tee z.log (to have rbtlog also check the upstream APK for the embedded commit hash and adjust the recipe accordingly)

As described above, rbhelper takes care to activate/deactivate the Python VEnv automatically. And of course you can always skip the |tee z.log part if you don't want a log file to be created.

Updating recipes¶

By recommendation given by Fay, your recipes start their existence in your master branch (fraggle/master), which should also be the only place where you modify them. Modification of course includes updates, so:

cd ${HOME}/builder/rbtlog
git checkout fraggle/master
python3 scripts/update-recipes.py --continue-on-errors --update-hashes -v recipes/*.yml

Updated recipes are then committed to your repo (always sign your commits!!!), e.g. git commit -S -m 'update recipes'.

The corresponding command with rbhelper: rblog update-recipes [packageName].

Updating logs¶

This is the main task: build everything to see if it's RB. This happens in your log branch, so:

cd ${HOME}/builder/rbtlog
git checkout fraggle/log
git merge -S fraggle/master     # merge in your new/updated recipes
python3 scripts/update-log.py --keep-apks "${HOME}/builder/rbtlog/apk" podman recipes/*.yml
python3 scripts/make-index.py logs/*.json > index.json

All done and all fine? Don't forget to commit (signed): git commit -S -m 'update recipes' – ideally after you've fixed any „failed RBs“.

Corresponding rbhelper commands: rblog update-log [packageName] and rblog writeIndex – no matter what your current working directory might be.

Signing commits¶

We consider it essential that all commits to a builder's repo are signed, to ensure they have really be confirmed by you. Integration into the IzzyOnDroid repo browser will check for this – so if you plan to have your builder integrated there, this is mandatory: when not properly signed, your index will be rejected. For a proper signing setup, ensuring your commits and merges will always be signed, see our wiki on Sign your commits.

Tools¶

rbhelper¶

Our rbhelper repo holds some tools built around rbtlog, to ease your work with the builds and the builder. This includes:

rblog: a wrapper around rbtlog automating the activating/deactivating of the venv whenever needed and enhancing tasks, also sending notifications via NTFY if you like
scripts and shell functions to compare APKs if they're reproducible, diff components, extract hashes and more
easy linting of recipes/logs
extracting build logs
and some more.

Find the details in the project's Readme. rbhelper evolved out of daily needs when running the IzzyOnDroid builder.

rbtlui¶

rbtlui helps you visualize builders:

which apps are covered by which builder
what's the RB status of an app in which builder
show the logs of a specific build

More to come. The Android app of course is RB itself, and can be found at IzzyOnDroid.

Repo Browser¶

The IzzyOnDroid repo browser also visualizes RBs. In its app lists, a green shield shows up for apps whose most recent release was confirmed to be reproducible. And the „Packages“ section of each app's detail page shows you if a given APK was confirmed to be reproducible (green shield), failed RB (yellow shield with exclamation mark), or was not yet tested (gray shield); the green and the yellow shield link to the corresponding results, and their titles show which builder had processed them.

F-Droid Clients¶

Visualizations are also available in Droid-ify and Neo Store, and more is on the road map there.

Alternative setups¶

Take a look at the original rbtlog repo for a setup using Github CI, handling recipe updates as well as log updates automatically in their own branches and sending you PRs to check (and to sign-commit). Should you establish a similar configuration with other CI (especially with Woodpecker at Codeberg), we'd very much welcome your setup instructions!

Verification Builder¶