58 city Android Qigsaw upgrade-v1.0 4.1 multi ABI construction
1. Background
Qigsaw is an open source framework used by Wafers' dynamic capability, which carries the operation of 58App, arbitrary gate and Mocha. The group started from qigsaw v1.0 in 2020 / 03 3.2.2 fork access
https://github.com/iqiyi/Qigsaw
v_ Version 1.3.2.2 only supports single ABI dimension construction. As more and more domestic app stores need to upload 64 bit apks, there is a growing demand for Qigsaw to support Base APK and Split APKs to split based on ABI dimension
Qigsaw v1.4.0 supports APK splitting based on ABI dimension:
2. Scheme investigation and design
Summarized as:
- Align the latest version code of Qigsaw
- Retain the old custom business logic and bug fix
- Transform 58App packaging script, ftp backup, AVM release process, any door background
- Wireless basic capability test, arbitrary gate, Mocha service line test
Original contracting process:
Target contracting process:
3. Upgrade process
3.1 Qigsaw
(1) Get qigsaw v1 4.1 latest code
Based on a new branch, pull qigsaw v1 4.1 latest code (this branch is completely the latest code without any old modification code)
(2) Manually add previous modification points
| # | changelog |
|---|---|
| 1 | maven publishing script |
| 2 | Implement split apk upload logic |
| 3 | SplitConfiguration adds interface support for setting fakeactivity / service / receiver |
| 4 | Remove the group call restriction of AABExtension class |
| 5 | Handle the resource loading problem when the current basic component does not jump to the scene |
| 6 | Add multi dex to install Activity and start Activity filtering configuration |
| 7 | Add external processing options for mobile network download pop-up |
| 8 | Breakpoint download, parallel downloader repair |
| 9 | Add task to upload split config file |
| 10 | Add the option of Activity list property of plug-in filter injected resources |
| 11 | Fix 5 and 6 system adaptation problems |
| 12 | Fix exception thrown by resource loading |
| 13 | fix downloader nullpointer |
| 14 | Adapt agp4 0 (the latest version has been officially repaired) |
| 15 | Support R8 compilation |
(3) Problems encountered
Resguard adaptation problem
AndroResGuard is a resource obfuscation tool that can be used to reduce the size of APK. People in the wechat team open source the project, which directly modifies the APK file
Resguard principle:
(1) Confusing resources
When calling resources, Android uses int values instead of res name directly. This corresponding relationship will be stored in resource after packaging In the ArsC file. AndResGuard reduces the file size by changing res name to a, b and c
(2) 7z compression
AndResGuard uses the 7z limit to compress images and resource files (including resource.arsc mentioned above) to reduce the size of apk
58App AndResGuard packaging process (package with v7a output):
Packaging process after upgrading Qigsaw:
Obviously, the process is incorrect, which will increase the size of the subcontracted abi package. The basic package should be resguard before subcontracting. Therefore, modify the Qigsaw packaging script as follows:
// source: buildSrc/com.iqiyi.qigsaw.buildtool.gradle/QigsawAppBasePlugin:
if (QigsawSplitExtensionHelper.isMultipleApkForABIs(project)) {
SplitBaseApkForABIsTask splitBaseApkForABIs = project.tasks.create("split${baseVariant.name.capitalize()}BaseApkForABIs", SplitBaseApkForABIsTask)
splitBaseApkForABIs.baseVariant = baseVariant
splitBaseApkForABIs.apkSigner = apkSigner
splitBaseApkForABIs.dynamicFeaturesNames = dynamicFeaturesNames
splitBaseApkForABIs.supportABIs = QigsawSplitExtensionHelper.getSupportABIs(project)
splitBaseApkForABIs.baseAppCpuAbiListFile = baseAppCpuAbiListFile
splitBaseApkForABIs.baseApkFiles = baseApkFiles
splitBaseApkForABIs.packageAppDir = packageAppDir
splitBaseApkForABIs.baseApksDir = baseApksDir
Task resguardTask = AGPCompat.getResguardTask(project, "${baseVariant.name.capitalize()}")
// apply plugin: 'AndResGuard' must be applied after the qigsaw application plug-in, otherwise the resguardTask will not be found
if (resguardTask != null) {
SplitLogger.w("found resguardTask")
resguardTask.dependsOn baseAssemble
baseAssemble.finalizedBy resguardTask
splitBaseApkForABIs.dependsOn resguardTask
resguardTask.finalizedBy splitBaseApkForABIs
} else {
SplitLogger.w("not found resguardTask")
baseAssemble.dependsOn splitBaseApkForABIs
packageApp.finalizedBy splitBaseApkForABIs
}
}
Here is a knowledge point:
Host App apply plug-ins A and B. if Apply A first and then Apply B, the task in plug-in B cannot be obtained in plug-in a Therefore, you need to adjust the application order of 58App Qigsaw plug-in and resguard plug-in
// apply plugin: 'AndResGuard' must be applied after the qigsaw application plug-in, otherwise the resguardTask will not be found
if (Boolean.parseBoolean(ON_JENKINS)) {
apply from: 'andResguard.gradle'
}
if (Boolean.parseBoolean(AAB_SWITCH)) {
apply from: '../DynamicFeatures/qigsaw-application-apply.gradle'
}
Qigsaw 7z depth compression problem
58when using the AndroResGuard plug-in, 7z compression is turned on, so 7z compression also needs to be turned on in the Qigsaw plug-in to keep the package size consistent
// 58App AndResGuard configuration
apply plugin: 'AndResGuard'
andResGuard {
enableResGuard = Boolean.parseBoolean(rootProject.enableResGuard)
mappingFile = file("./resource_mapping.txt")
use7zip = true // Turn on 7z deep compression
useSign = true
// ....
}
The Qigsaw packaging plug-in also has 7z compression options:
// buildSrc/com.iqiyi.qigsaw.buildtool.gradle.task.SplitBaseApkForABIsTask
// Recompress the subcontracted apk packets
if (use7z) {
run7zCmd("7za", "a", "-tzip", unsignedBaseApk.absolutePath, unzipBaseApkDirForAbi.absolutePath + File.separator + "*", "-mx9")
} else {
ZipUtils.zipFiles(Arrays.asList(unzipBaseApkDirForAbi.listFiles()), unzipBaseApkDirForAbi, unsignedBaseApk, compress)
}
After opening, it is found that the 58App v7a package is reduced to 113M, and the exception of resource not found occurs during operation.
Let's start with the following 7z compression:
7z is a mainstream and efficient compression format, which has a very high compression ratio
eg: 7z -tZip a test.zip ./test/* -mx0
Compress all files in the test folder into ZIP files in the storage compression mode. The compressed file is test zip
a add options for
-tZip is in compressed format (-t7z...)
The specific parameters of compression ratio are as follows:
-mx0 only stores uncompressed, the fastest, uncompressed, and the zip size is the same as the original folder
-mx1 extreme compression
-mx3 fast compression
-mx5 standard compression
-mx7 maximum compression
-mx9 ultimate compression
Let's first look at Qigsaw subcontracting logic:
Therefore, the reasons for the above problems are:
The AndResGuard plug-in has conducted a mx9 deep compression of the 58App basic package APK, because the Qigsaw plug-in needs to decompress the original package when subcontracting,
Process LIBS and ABI configuration files, and then compress them. Whether 7z or zip is used for re compression, the original basic package will be destroyed
Solution:
The APK is not decompressed during subcontracting, and the zip operation is directly used to modify and re sign. While solving this problem, it can also speed up the packaging speed
// buildSrc/com.iqiyi.qigsaw.buildtool.gradle.task.SplitBaseApkForABIsTask
File baseApk = baseApkFiles[0]
List<String> abiList = supportABIs != null ? supportABIs.split(",") : null
if (abiList == null || abiList.isEmpty()) {
SplitLogger.e("Base apk ${baseApk.absolutePath} has no native-library abi folder, multiple apks don't need.")
return
}
if (abiList.size() == 1) {
SplitLogger.e("Base apk ${baseApk.absolutePath} has only one native-library abi folder, multiple apks don't need.")
return
}
abiList.each { String abi ->
if (SUPPORTED_ABIS.contains(abi)) {
// You cannot directly use 7z depth compression, which will affect resources The use of ArsC causes the resource not to be found, so the zip command is used here
// Copy base apk
File copyBaseApk = new File(baseApksDir, "${project.name}-${baseVariant.name.uncapitalize()}-${abi}${SdkConstants.DOT_ANDROID_PACKAGE}")
if (!copyBaseApk.parentFile.exists()) {
copyBaseApk.parentFile.mkdirs()
}
if (copyBaseApk.exists()) {
copyBaseApk.delete()
}
FileUtils.copyFile(baseApk, copyBaseApk)
String copyBaseApkPath = copyBaseApk.getAbsolutePath()
// Delete signature related files
runCmd("zip", "-d", copyBaseApkPath, "META-INF/CERT.RSA")
runCmd("zip", "-d", copyBaseApkPath, "META-INF/CERT.SF")
runCmd("zip", "-d", copyBaseApkPath, "META-INF/MANIFEST.MF")
Set<String> masterSplitHandleFlags= new HashSet<>()
abiList.each { String ABI ->
if (abi != ABI) {
// Delete other ABI's lib
runCmd("zip", "-d", copyBaseApkPath, "lib/${ABI}/**")
// Delete other ABI's built-in splits (include master)
dynamicFeaturesNames.each { String splitName ->
if (!masterSplitHandleFlags.contains(splitName)) {
runCmd("zip", "-d", copyBaseApkPath, "assets/qigsaw/${splitName}-master**.zip")
masterSplitHandleFlags.add(splitName)
}
runCmd("zip", "-d", copyBaseApkPath, "assets/qigsaw/${splitName}-${ABI}**.zip")
}
}
}
// Update base apk cpu abi list file
File baseAppCpuAbiListFileForAbi = new File(baseApksDir,"assets/${baseAppCpuAbiListFile.name}")
if (!baseAppCpuAbiListFileForAbi.parentFile.exists()) {
baseAppCpuAbiListFileForAbi.parentFile.mkdirs()
}
if (baseAppCpuAbiListFileForAbi.exists()) {
baseAppCpuAbiListFileForAbi.delete()
}
baseAppCpuAbiListFileForAbi.write("abiList=${abi}")
// ProcessBuilder execute multi commands
File baseAppCpuAbiScript = new File(baseApksDir,"baseAppCpuAbiScript")
if (baseAppCpuAbiScript.exists()) {
baseAppCpuAbiScript.delete()
}
baseAppCpuAbiScript.write("#!/usr/bin/env bash\ncd \$1\nzip -d \$2 \$3\nzip -m \$2 \$3")
runCmd("chmod", "755", baseAppCpuAbiScript.getAbsolutePath())
runCmd(baseAppCpuAbiScript.getAbsolutePath(), copyBaseApk.getParent(), copyBaseApk.getName(), "assets/${baseAppCpuAbiListFile.name}")
// Resign apk if need
SigningConfig signingConfig = null
try {
signingConfig = apkSigner.getSigningConfig()
} catch (Throwable ignored) {
}
boolean isSigningNeed = signingConfig != null && signingConfig.isSigningReady()
if (isSigningNeed) {
File signedBaseApk = new File(baseApksDir, "${project.name}-${baseVariant.name.uncapitalize()}-${abi}-signed${SdkConstants.DOT_ANDROID_PACKAGE}")
if (signedBaseApk.exists()) {
signedBaseApk.delete()
}
apkSigner.signApkIfNeed(copyBaseApk, signedBaseApk)
File destBaseApk = new File(packageAppDir, signedBaseApk.name)
if (destBaseApk.exists()) {
destBaseApk.delete()
}
FileUtils.copyFile(signedBaseApk, destBaseApk)
} else {
File destBaseApk = new File(packageAppDir, copyBaseApk.name)
if (destBaseApk.exists()) {
destBaseApk.delete()
}
FileUtils.copyFile(copyBaseApk, destBaseApk)
}
}
}
Here is also a knowledge point:
When you execute a command using ProcessBuilder, you can only execute a single command at a time. If you want to execute multiple commands at a time, you can wrap them into a script file and then use ProcessBuilder to execute them
The above two questions have submitted merge request s to Qigsaw:
Fix the problem of increasing the size of the subcontracting ABI APK package after the resguard plug-in is applied to the host app & 7z deep compression causes the resource not to be found #57
https://github.com/iqiyi/Qigsaw/pull/57
3.2 58App
(1) Remove SPLITS_APK,UNIVERSAL_APK related configuration
// After removing the splits configuration, Qigsaw has implemented a set of subcontracting mechanism, which conflicts with the system subcontracting mechanism
splits {
abi {
enable !Boolean.parseBoolean(AAB_SWITCH) && Boolean.parseBoolean(rootProject.SPLITS_APK)
reset()
universalApk Boolean.parseBoolean(rootProject.UNIVERSAL_APK) // If true, also generate a universal APK
rootProject.CPU_ARCH.split(",").each { value ->
include value
}
}
}
(2) Modify WubaPackage jenkins packaging configuration
- v7a is the default package, 58Client_vxxx_release.apk
- Backup all packages, v7a, v8a packages to FTP
- Any gate dynamically updates the download address of the basic package to the full package address
(3) Qigsaw API change
// WubaQigsawLib/QigsawManager
public static void applicationOnCreate() {
Qigsaw.onApplicationCreated();
Qigsaw.registerSplitActivityLifecycleCallbacks(new QigsawSplitActivityLifecycleCallbacks());
// Qigsaw needs to be called manually Preloadinstalledsplits, otherwise the Application with splits installed will not be called at startup
try {
Set<String> splitNames = AABExtension.getInstance().getSplitNames();
Qigsaw.preloadInstalledSplits(splitNames);
} catch (Exception e) {
LOGGER.e(e);
}
}
3.3 AVM
The following changes are involved:
- Distribution management - package configuration and add CPU_ARCHES=armeabi-v7a,arm64-v8a
- Version distribution management - basic package changed to v7a package
- Any gate - the base package is changed to the full package address
- Any door - basic package download address shows full package, v7a, v8a