Debug failed RBs¶
This document gives you some hints on how to analyze (and hopefully fix) failed RBs with the tools provided by rbhelper. Wherever build recipes are mentioned/quoted, they reference what we use for our Verification Builder, i.e. rbtlog.
For an analysis, we recommend you copy the two APK files (*upstream.apk and *unsigned.apk) into an empty directory – this makes many things much easier (e.g. most of the rbhelper commands can be run without any parameter, as they'll then find the corresponding APK files automatically). In this document, we assume you have done so.
We cannot address all potential issues here – we probably didn't even encounter all of them ourselves yet. But we can look into the most common cases, which we will do.
First step: Get the overall diff¶
In the directory with just the two APKs, issue the rbtest command. If it just says „RB confirmed“, there's no more to do 😜 – otherwise you now should find a third file in your directory: <packageName>.diff. This file shows which files differ between your two APKs. Some differences are easy to fix, others are harder. Sometimes there are multiple things wrong. We'll take a look at some examples here now.
When in the following we say „files differ“, it does not just mean there's an entry in the diff – but that their file hashes differ. Let's look at an example entry:
perms ver FSys Size tt CompSize Algo Timestamp CRC FileName
----------+----+----+-------+--+--------+----+-------------------+--------+-----------
- -rw-r--r-- 0.0 unx 120 b- 118 defN 1981-01-01 01:01:02 21cde471 META-INF/version-control-info.textproto
+ -rw-r--r-- 0.0 unx 120 b- 117 defN 1981-01-01 01:01:02 21cde471 META-INF/version-control-info.textproto
These two lines are shown as differing. But if you take a look at the CRC column, you see the file checksums are identical – so the files themselves do not differ. Which is also reflected by the fact that Size (the uncompressed file size) is the same. The difference here is in CompSize, so the files are compressed differently. This is a special case we'll deal with in the compression section. All else is about files which do really differ, i.e. also have different checksums (CRC).
As this already shows, the diff output produced by rbtest has a helpful header, to make it easier for you to correlate things. Abbreviated column names are also explained there.
Important
if multiple issues apply, before you try to fix one, check all the others. There might be one you cannot fix locally, but need the upstream developer to perform adjustments first. Or there are things that simply cannot be fixed, and you'd just waste your time.
No diff but the certificates¶
If nothing but the certificate files (usually 3 files at the very end of the diff) differ, that's normally caused by zip alignment or signing algorithms. To see the details on this, use the zipalignment command. With just your 2 APKs in your working directory: zipalignment *.apk will output something like this:
file='9701284772fe24a72e1ee380d9ba154bd042ae90685254701fafda7e1d2a5230-com.orgzlyrevived-v1.8.37-upstream.apk'
zipaligned (4-byte alignment) : yes
files with apksigner padding : 202
apksigner alignments from extra fields : 4
most likely uncompressed .so page alignment : none
file='b9b945d117df2a0c63f0543492513169be437d4f81b46a01db73c38d41e34f8f-com.orgzlyrevived-v1.8.37-unsigned.apk'
zipaligned (4-byte alignment) : yes
files with apksigner padding : 0
apksigner alignments from extra fields : none
most likely uncompressed .so page alignment : none
Here the developer used a more recent version of apksigner, which added extra padding. So you need to adjust that after building your APK, i.e. at the end of the recipe. Luckily, reproducible-apk-tools come to the rescue here:
- OUT="app/build/outputs/apk/release/app-release-unsigned.apk"
- git clone -b v0.3.0 https://github.com/obfusk/reproducible-apk-tools.git
- reproducible-apk-tools/zipalign.py --zipalign --pad-like-apksigner --replace "$OUT" /outputs/unsigned.apk
The name of the $OUT file you find in your build logs (it's the one you usually move to /outputs/unsigned.apk at the end of your build). As there is no .so page alignment in the upstream APK, we just use --zipalign with the --pad-like-apksigner. But if the zipalignment output looks like this:
zipaligned (4-byte alignment) : yes
files with apksigner padding : 52
apksigner alignments from extra fields : 4 16384
most likely uncompressed .so page alignment : 16KiB
for the upstream APK instead, we will also have to adjust page alignment:
- OUT=app/build/outputs/apk/release/app-release-unsigned.apk
- git clone -b v0.3.0 https://github.com/obfusk/reproducible-apk-tools.git
- reproducible-apk-tools/zipalign.py --page-size 16 --pad-like-apksigner --replace $OUT /outputs/unsigned.apk
Note that in both cases, you will need to have the python3 package specified in the extra_packages: section of your build recipe, as that's a requirement of reproducible-apk-tools. Also note that in these two examples, you no longer need to move the APK to the output directory, as the zipalign.py command already takes care for that.
META-INF/version-control-info.textproto¶
This file holds details from the versioning tool, usually Git. If these two files differ, the APKs have been built from different commits. This can have multiple causes:
- the developer tagged the wrong commit. In this case, a
git checkout <commithash>to the hash provided in the*upstream.apkadded as first step to your recipe should cure this. - the developer built from a „dirty tree“ with local changes, and then after that build succeeded immediately committed those changes and tagged the release based on the new commit: use
git reset --soft <commithash>instead. But this will only help if there are no other differences. - the developer built from a „dirty tree“ with local changes, then made more changes, committed, tagged the new commit – but used the APK built „in the middle“: have fun. You might be able to find out which changes to apply – but it's more likely to be wasted time. Report your findings upstream, and wait for the next release with a hopefully cleanly built APK.
Manifest¶
Just the AndroidManifest.xml differs? Well, the upstream APK was not built from a „clean tree“ at the tagged commit then. Most likely, after the indicated commit was written, the developer updated something in the manifest (usually versionCode and versionName), and then built the APK. Seeing that went fine, they tagged that commit, created the release, attached the APK – and only then committed the changes. Bad idea: APKs should always be built from a clean tree at the commit the tag points to – no local changes, no artifacts remaining from previous builds. But let's take a look at what differs in the AndroidManifest.xml. For that, simply invoke manidiff in the directory holding no other APKs than the two we want to compare. You should then find a file named manifest.diff in the same directory, which could e.g. look like this:
XML ELEM START [lineno=2, name='manifest', #attributes=7]
- ATTR: http://schemas.android.com/apk/res/android:versionCode=147
- ATTR: http://schemas.android.com/apk/res/android:versionName='2.34.5'
+ ATTR: http://schemas.android.com/apk/res/android:versionCode=146
+ ATTR: http://schemas.android.com/apk/res/android:versionName='2.34.4'
ATTR: http://schemas.android.com/apk/res/android:compileSdkVersion=35
That's usually the classic case I just described: the developer's APK was not built from the indicated commit. Here you go to the upstream repository and find out what commit holds those changes, and build from that. Two variants for this:
- the commit holding the changes matches the one in the version info of the upstream APK, but not that of yours: see
META-INF/version-control-info.textprotoabove and fix that first. - the commits do not match: use
git checkout <commithash>to the commit holding the change, followed bygit reset --soft <commithash>to the one ofMETA-INF/version-control-info.textproto(soMETA-INF/version-control-info.textprotopoints to this one, to match the upstream APK). As you then build from a different commit, this might solve other differences at the same time.
If that diff is rather large and confusing, you can also give xmanidiff a try, which diffs the original XML as extracted by androguard.
Differing res/*.xml or resources.arsc¶
These usually mean that either
- the APK was not built from a clean tree at the indicated commit (local changes, artifacts remaining from previous builds)
- different build methods/options where used
Most likely, that will be something that needs fixing upstream – not something that can be fixed just on the rebuilder side. Use the arscdiff command from rbhelper to view and investigate the differences of resources.asc, or manidiff if the differing file is some res/*.xml.
M7.json¶
This file is generated by the AboutLibraries library – and when it differs, usually contains a build timestamp at its very beginning. While for a quick temporary fix one could adjust that using fix-files:
- git clone -b v0.3.0 https://github.com/obfusk/reproducible-apk-tools.git
- reproducible-apk-tools/inplace-fix.py --internal --zipalign fix-files app/build/outputs/apk/release/app-release-unsigned.apk 'sed s/"generated":"[^"]*"/"generated":"2024-06-17T15:38:17.319Z"/' 'res/M7.json'
(using the timestamp from the upstream APK). That would mean having to adjust the recipe on reach new release. So better ask upstream to fix it properly – which can be done in the build.gradle:
aboutLibraries {
// Remove the "generated" timestamp to allow for reproducible builds
excludeFields = ["generated"]
}
or, for Kotlin, in the build.gradle.kts:
aboutLibraries {
// Remove the "generated" timestamp to allow for reproducible builds
excludeFields = arrayOf("generated")
}
Note that with AboutLibraries 11.5.0 and higher, this generated date field is disabled by default – so the corresponding developers should be encouraged to update.
Windows builds and newlines¶
Newline differences between building on Windows vs Linux make builds not reproducible. A very common place for this to show up is META-INF/services/*. Until that bug is fixed (which might take a while), this can be worked around using inplace-fix.py:
- reproducible-apk-tools/inplace-fix.py --internal --zipalign fix-newlines "$OUT" 'META-INF/services/*'
The --zipaling might need to be replaced with the --page-size argument, which you can find out by running the zipalignment command on the *upstream.apk (see above).
If there are multiple places in need of this adjustments, you can just add them to the end of the above command – each one quoted separately. For example: … 'META-INF/services/*' 'res/pages/*.html' 'res/foobar/*.json'. No need for separate calls.
baseline.prof¶
This file holds the compiled baseline profile, and will usually always differ if one of the classes*.dex file differs, as it a.o. contains the file hash of it. In such a case, you should fix the DEX problem first. If only the baseline differs, this can have various reasons. In our experience, we've found two cases where this happened: a developer switched back from JDK 21 to JDK 17 (this is a rather unusual result for this) – or a non-deterministic build (which we also call „flaky build“). The latter then usually shows as an „off-by-one“ in the diff (which can be generated using the baseline command from rbhelper), like this:
- num_hot_method_ids=8444
+ num_hot_method_ids=8445
As non-deterministic means a different outcome on multiple runs, this can often be worked around by playing the lottery: running multiple builds in a row until we succeed. For this, we first determine the checksum of the baseline.diff from the upstream APK, and then build in circles until we can either match it, or the build times out. That part of a recipe looks like this:
- PROF_FILE=app/build/intermediates/binary_art_profile/release/compileReleaseArtProfile/baseline.prof
- PROF_SHA1=6a7083fcd6c3371d9a32288ee213825d6a381bd5
- for _ in {1..10}; do
- ./gradlew clean assembleRelease --no-build-cache --no-configuration-cache --no-daemon
- test -f "$PROF_FILE"
- if [ "$( sha1sum "$PROF_FILE" | cut -d' ' -f1 )" = "$PROF_SHA1" ]; then
- break
- fi
- done
(copy-pasting that snippet, the PROF_SHA1 can be adjusted automatically running scripts/update-hashes.py recipes/<packageName>.yml <tagName> from the rbtlog root). If you don't know where to find the corresponding baseline.prof in the build tree, simply insert a - find . -name '*aseline.prof' line before the test line. That will list all candidates and, if your guess was wrong, the test will abort the build immediately afterwards, as the specified baseline.prof was not found.
If that does not help, an option might be to disable baselines. That should be tested for potential performance impacts then, though. For build.gradle, add:
tasks.whenTaskAdded { task ->
if (task.name.contains("ArtProfile")) {
task.enabled = false
}
}
For build.gradle.kts, instead add:
tasks.whenTaskAdded {
if (name.contains("ArtProfile")) {
enabled = false
}
}
Compression¶
Files show differences, but at a closer look that's just the compressed size while the CRC is identical – like here:
perms ver FSys Size tt CompSize Algo Timestamp CRC FileName
----------+----+----+-------+--+--------+----+-------------------+--------+-----------
- -rw-r--r-- 0.0 unx 120 b- 118 defN 1981-01-01 01:01:02 21cde471 META-INF/version-control-info.textproto
+ -rw-r--r-- 0.0 unx 120 b- 117 defN 1981-01-01 01:01:02 21cde471 META-INF/version-control-info.textproto
This could mean that different compression levels have been used – which can be found out using list-compresslevel.py from the reproducible-apk-tools:
$ list-compression-levels *upstream.apk META-INF/version-control-info.textproto
filename='META-INF/version-control-info.textproto' compresslevel=6
Should the value differ for the *unsigned.apk, it can be adjusted for the latter in the build recipe:
reproducible-apk-tools/fix-compresslevel.py unsigned.apk fixed.apk 6 META-INF/version-control-info.textproto
(which might need to be followed by a reproducible-apk-tools/zipalign.py call then). But if the indicated compression level is identical, or the call fails on the upstream APK stating it could not determine the compression level, that often rather means the APK has been built on a system using zlib-ng instead of zlib. A potential work around for this is building zlib-ng in the app's recipe, and add it to the LD_LIBRARY_PATH – before building the APK itself of course:
- git clone -b 2.2.2 https://github.com/zlib-ng/zlib-ng.git
- pushd zlib-ng
- _options=( -G Ninja -DCMAKE_BUILD_TYPE=None -DCMAKE_INSTALL_PREFIX=/opt -DCMAKE_INSTALL_LIBDIR=lib -Wno-dev -DWITH_GTEST=OFF -DWITH_UNALIGNED=OFF )
- cmake -B build "${_options[@]}"
- cmake --build build
- cmake -B build-compat "${_options[@]}" -DZLIB_COMPAT=ON
- cmake --build build-compat
- cmake --install build
- cmake --install build-compat
- popd
- export LD_LIBRARY_PATH=/opt/lib
- ./gradlew assembleRelease
For this, some extra packages are needed:
extra_packages:
- build-essential
- cmake
- ninja-build
A special case is compression of *.ttf files. If that proves impossible to fix, it can be simply switched off in build.gradle – by the upstream developers, of course:
aaptOptions {
noCompress 'ttf', '.ttf'
}
Which can of course also be done for other files added to the noCompress line, like noCompress 'ttf', '.ttf', 'foo', '.foo'.
Differing *.so files¶
These are in most cases embedded build paths (especially with Flutter and Rust), Build-IDs, or both. Helpful in identifying them is Diffoscope, which rbhelper covers e.g. via the dscope command. With only a single *upstream.apk and *unsigned.apk (as they result from a build run of rbtlog) in the working directory, this command can be run without any parameters, and will result in a diffoscope.html file that can be opened in a web browser. The output can be overwhelming, and often contains a lot of binary blocks. But one quickly learns what to look for.
Different build paths¶
If there are differences in embedded paths, this is what you should start with. Having used the defaults of our recipes, you most likely used build_repo_dir: /build/repo – which you will see for the *unsigned.apk. This is what then usually needs to be adjusted in the recipe. For apps built via CI by e.g. Github Actions this mostly means:
build_home_dir: /home/runner
build_repo_dir: /home/runner/work/<repoName>/<repoName>
build_user: runner
provisioning:
android_home: /usr/local/lib/android/sdk
For other variables used in GH images, see e.g. here. Also check the .github/workflows/* in the app's repo for details, where you usually also find which image and JDK version was being used.
linsui points out that the embedded android_home path can also be „mapped“ via build.gradle, which seems to be what F-Droid.org prefers:
externalNativeBuild {
cmake {
cFlags "-ffile-prefix-map=${rootDir}= -ffile-prefix-map=${System.getenv("ANDROID_HOME")}= -Wl,--hash-style=gnu,--build-id=none"
cppFlags "-ffile-prefix-map=${rootDir}= -ffile-prefix-map=${System.getenv("ANDROID_HOME")}= -Wl,--hash-style=gnu,--build-id=none"
}
}
or, directly in the CMakeLists.txt file:
add_compile_options("-ffile-prefix-map=${CMAKE_CURRENT_SOURCE_DIR}=.")
This uses the -ffile-prefix-map=OLD=NEW compiler options to remove parts of the embedded paths, and thus works in general (not depending on the build environment) – but must be implemented with the source code by the app's author(s). Also see Build path at reproducible-builds.org for details on this.
For other specific languages and their flags concerning embedded paths as well as (see next section) build-ids, instructions can be found e.g. in the Reproducible Builds documentation at F-Droid, for example on Golang and Rust.
Build IDs¶
For several frameworks, differences in build paths also means differences in build ids – so with some luck, those were solved alongside. If not, this has to be solved upstream by disabling them. To achieve this, for cmake ≥ 3.13, add this to CMakeLists.txt:
add_link_options("-Wl,--build-id=none")
or, if that fails:
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,--build-id=none")
target_link_options(my_target PUBLIC "-Wl,--build-id=none")
For older cmake, add this to gradle files:
android {
defaultConfig {
externalNativeBuild.cmake {
cFlags "-DCMAKE_SHARED_LINKER_FLAGS=-Wl,<linker args>"
}
}
}
(where for this case, <linker args> would be --build-id=none)
For NDK builds:
android {
defaultConfig {
externalNativeBuild {
ndkBuild {
arguments "LOCAL_LDFLAGS+=-Wl,<linker args>"
}
}
}
}
Further, LLVM passes different defaults to linkers on different platforms; --hash-style=gnu is used on Debian by default. To change the hash style to this, --hash-style=gnu can be passed to the linker.
Native Library Stripping¶
Another cause for differences here can be stripping. Potential fixes at upstream in build.gradle:
android {
packagingOptions {
doNotStrip '**/*.so'
}
}
new way:
android {
packaging {
jniLibs.keepDebugSymbols.add("**/*.so")
}
}
(source)
clang¶
One more potential reason: clang version string harms reproducibility with meta-compile information. And a potential solution for this by linsui:
add_custom_command(TARGET ${CMAKE_PROJECT_NAME}
POST_BUILD
COMMAND ${CMAKE_OBJCOPY} --remove-section .comment --remove-section .note.gnu.build-id $<TARGET_FILE:${CMAKE_PROJECT_NAME}>
VERBATIM
)
Other possible reasons¶
There are several other possible causes, see e.g.
- embedded timestamps, as we've already seen with
M7.jsonabove. Also seeSOURCE_DATE_EPOCHfor cases timestamps are „needed“.
Differing *.png files¶
Differences here usually happen if PNG files are generated or manipulated at build time. The most frequent cases are PNG crunching, and PNGs generated from vector drawables. Both needs to be fixed upstream. For build.gradle this means:
android {
// disable PNG crunching:
aaptOptions {
cruncherEnabled = false
}
// disable generating PNGs from vector drawables:
defaultConfig {
vectorDrawables.generatedDensities = []
}
}
Similarly, there's shrinkResources false for contents of res/:
android {
buildTypes {
release {
// disable resource shrinking:
shrinkResources false
}
}
}
For build.gradle.kts, the latter would be written as isShrinkResources = false.
More details can be found in Shrink, obfuscate, and optimize your app.
Differing classes*.dex files¶
These can be very tricky, and often mean some guessing, „gut-feeling“ (if you've tried enough recipes, you get ideas you could not always put in words: „this looks like XYZ, but don't ask me why…“) as well as try-and-error. Still, there are some cues. So use the dexdiff command from rbhelper to obtain the differences (you can also use Diffoscope lateron if you're stuck here), and take a look:
- differences only in annotations? This could mean different JDKs are used. Newer JDKs often have more/longer annotations than older ones. So if you e.g. built with JDK 17, and the upstream APK has more/longer annotations, try building with JDK 21. Worst case upstream did use a non-LTS JDK (e.g. 18, 23); those should better be avoided.
- one side has (a lot of) additional „stuff“ – or it simply „looks weird“: upstream possibly didn't build from this commit, or built from a „dirty tree“ (local changes, artifacts remaining from previous builds as they forgot to run
gradle clean) - Over multiple versions, it's always the same classes showing up here with the same differences: that could be a non-determinism in the R8 Optimizer
- another (though rare) cause for differences e.g. in ordering could be concurrency: reproducibility can depend on the number of CPUs/cores, see e.g. the bug report on unneeded DEX code differences based on number of CPUs used in build process. For this, you can use the
build_cpus:setting in your YAML file to match the number of cores upstream uses – if you know that (for example, if they built using Github actions on a Linux VM, that would mean 4 cores).
ZIP ordering differences¶
One cause of different ordering in ZIP files cound be some Android Studio builds having non-deterministic ZIP ordering, which in the observed cases might have been caused by the Android Gradle Plugin (AGP) used by those installations. This should be solved since AGP 7.1.x, though. To confirm or rule out this reason, upstream developers could be asked to provide a build created at the command line, using ./gradlew assembleRelease.