Android performance - RocketX

1, Background description

With the increasing project volume, the compilation speed also increases. Sometimes a modification needs to wait for several minutes. Based on this general situation, RocketX is introduced to improve the speed of full compilation by dynamically replacing the module with aar in the compilation process.

2, Effect display

2.1 introduction to test items

There are 3W + class and resource files in the target project, which are fully compiled for about 4min (18-year-old MBP generation 8 i7 16g)
Effect after full volume growth through RocketX (average value of 3 times for each operation)

The project dependencies are shown in the figure below. app depends on bm business module, and bm business module depends on top-level base/comm module

rx(RocketX) compilation - you can see that the compilation speed of rx(RocketX) in any module is basically controlled at about 30s, because only app and modified modules are compiled, and other modules are aar packages and do not participate in compilation.
Native compilation - when the base/comm module is changed, all modules at the bottom must participate in the compilation. Because the app/bmxxx module may use the interfaces or variables in the base module, and I don't know whether there is any change to. (then the speed is very slow)
Native compilation - when bmDiscover is changed, only app module and bmDiscover module are required to compile (faster)
For rx(RocketX), the speed of compiling top-level modules is increased by 300%+

3, Thinking problem analysis and module construction:

3.1 analysis of ideas and problems

The unchanged module dependency needs to be dynamically modified in the form of gradle plugin to the corresponding aar dependency. If the module is changed, it will degenerate into project dependency, so that only the changed module and app modules can be compiled each time.
You need to change the implementation / API module B to implementation / API AARB, and you need to know how to add aar dependencies and eliminate the original dependencies in the plug-in
You need to build a local maven to store the aar corresponding to the unmodified module (you can also replace it with flatDir, which is faster)
When the compilation process starts, you need to find which module has been modified
It is necessary to traverse the dependency relationship of each module for replacement. How to obtain the module dependency? Can I get all module dependencies at one time, or can I call back each module? Modifying one of the module dependencies will block the subsequent module dependency callbacks?
After each module is changed into aar, the child dependency (network dependency, aar) of its own dependency is given to the parent module (how to find all parent modules)? Or directly to app module? Is there any risk of APP module dependency breaking? A technical solution is needed here.
The hook compilation process is required to replace the modified aar in loacal maven after completion
Provide AS status bar button to realize the function of opening and closing, accelerate compilation, or let developers use the habitual triangular run button

3.2 module construction

According to the above analysis, although there are many problems, the whole project can be roughly divided into the following parts:

4, Problem solving and Implementation:

4.1. How to add aar dependency manually, analyze the implementation source code and implement the tryInvokeMethod method of the entry in DynamicAddDependencyMethods. It is the methodMissing function of a dynamic language

tryInvokeMethod code analysis

 public DynamicInvokeResult tryInvokeMethod(String name, Object... arguments) {
       //Omit some code
       return DynamicInvokeResult.found(this.dependencyAdder.add(configuration, normalizedArgs.get(0), (Closure)null));
 }

The dependencyAdder implementation is a DirectDependencyAdder

private class DirectDependencyAdder implements DependencyAdder<Dependency> {
        private DirectDependencyAdder() {
        }
        public Dependency add(Configuration configuration, Object dependencyNotation, @Nullable Closure configureAction) {
            return DefaultDependencyHandler.this.doAdd(configuration, dependencyNotation, configureAction);
        }
    }

Finally, in DefaultDependencyHandler this. Doadd is added, and the DefaultDependencyHandler can be obtained in the project

public interface Project extends Comparable<Project>, ExtensionAware, PluginAware {
     ...
     DependencyHandler getDependencies(); 
     ...
}

The three parameters of the doAdd method are found through the debug source code. configuration is the object generated by the three strings of "implementation", "API" and "compileonly". dependencyNotation is a LinkHashMap with two key value pairs, namely Name: aarname and ext: AAR. The last configureAction can be null and call project.com dependencies. Add will eventually be called to the doAdd method, that is, you can call add directly.

 public Dependency add(String configurationName, Object dependencyNotation) {
        return this.add(configurationName, dependencyNotation, (Closure)null);
    }

    public Dependency add(String configurationName, Object dependencyNotation, Closure configureClosure) {
       //Here, we call doAdd directly 
        return this.doAdd(this.configurationContainer.getByName(configurationName), dependencyNotation, configureClosure);
    }

Then, add the implementation code of aar/jar according to the gourd and gourd: configName is the configName in childProject, that is, the three strings of "implementation", "API" and "compileonly". Take it intact:

    fun addAarDependencyToProject(aarName: String, configName: String, project: Project) {
        //Add aar dependency. The following code is equivalent to api/implementation/xxx (name: 'libaccount-2.0.0', ext: 'aar'). The source code uses linkedMap
        val map = linkedMapOf<String, String>()
        map.put("name", aarName)
        map.put("ext", "aar")
        project.dependencies.add(configName, map)
    }

4.2. localMaven gives priority to the implementation of flatDir. By specifying a cache directory getlocalmaven cachedir, throw in the generated aar/jar package. When modifying the dependency, you can add the corresponding aar through 4.1 above:

  fun flatDirs() {
        val map = mutableMapOf<String, File>()
        map.put("dirs", File(getLocalMavenCacheDir()))
        appProject.rootProject.allprojects {
            it.repositories.flatDir(map)
        }
    }

4.3. When the compilation process starts, you need to find which module has been modified

Use lastModifyTime to traverse the files of the whole project
Each module has a granularity. Recursively traverse the files of the current module, integrate the lastModifyTime of each file, and calculate a unique ID countTime
By comparing the countTime with the last one, the same description is unchanged, and the difference is changed The calculated countTime needs to be synchronized to the local cache
The overall 3W files take 1.2s, which is acceptable. At present, it is in the class changemoduleutils KT

4.4. module dependency acquisition

Through the following code, you can find the time to generate the dependency graph of the whole project and generate the dependency graph parser here. Before the actual compilation, ensure that the substitution can take effect after the dependency relationship is obtained, and after the global module dependency diagram has been generated, it can be met through the following listening:

  public interface DependencyResolutionListener {
    void beforeResolve(ResolvableDependencies var1);

    void afterResolve(ResolvableDependencies var1);
}

   project.gradle.addListener(DependencyResolutionListener listener)

How to obtain the dependency of each module? The dependency is hidden in Configuration Dependencies, then through project configurations. Maybecreate (configname) finds all Configuration objects and gets the dependencies of each module

4.5 replace module dependency project with aar technical scheme

The traversal order of each module dependency replacement is unordered, so the technical solution needs to support unordered replacement
The current scheme is: if the current module A is not changed, you need to replace A with A.aar through localMaven, and give A.aar and A's child dependency to the parent module of the first layer. (you may wonder what to do if the parent module is also AAR. In fact, there is no problem with this one. I won't talk about it here. It's too long)
Why should we give the parent to the app instead of directly giving it to the app? The following figure is A simple example. If B.aar does not give module A, the interface of module B used by A is missing, which will lead to compilation failure

Give the technical scheme demonstration of overall project replacement:

The overall implementation is in dependencieshelper KT this class, because it is too long to talk about, if you are interested, you can refer to the open source library code

4.5. hook compilation process. After completion, replace the modified aar in loacal maven

Click the triangle run to execute app: assemblydebug. You need to add an uploadLocalMavenTask after assemblydebug. Run our task through finalizedBy to synchronize the modified aar:

val localMavenTask = childProject.tasks.maybeCreate("uploadLocalMaven"+buildType.capitalize(),LocalMavenTask::class.java)
localMavenTask.localMaven = this@AarFlatLocalMaven
bundleTask?.finalizedBy(localMavenTask)

4.6. Provide AS status bar buttons. One small rocket button emits fire and the other does not, which represents enable/disable. A broom clean rockectx cache needs to be written by intellij idea plugin, that is, there are two plug-ins, one gradle plug-in and one AS plug-in:

5, One little surprise a day (more bug s)

5.1. It is found that by clicking the run button, the command executed is app: assemblydebug, and each sub module is not packaged with aar in the output

Solution: by studying the gradle source code, it is found that the package is executed by the task bundle${Flavor}${BuildType}Aar. Then you only need to find and inject the tasks corresponding to each module into app: assemblydebug and run it:

        android.applicationVariants.forEach {
            getAppAssembleTask(ASSEMBLE + it.flavorName.capitalize() + it.buildType.name.capitalize())?.let { task ->
                    hookBundleAarTask(task, it.buildType.name)
                }
        }

5.2. It is found that there are multiple duplicate jar packages after running

Solution: the implementation fileTree(dir: "libs", include: ["*.jar"]) jar dependency cannot be handed over to the parent module, and the jar package will enter the lib in the aar, which can be directly eliminated. It can be judged by the following code:

// The dependencies here are the following two: there is no need to add it to the parent, because the jar package directly enters the libs folder in its aar
if (childDepency is DefaultSelfResolvingDependency && (childDepency.files is DefaultConfigurableFileCollection || childDepency.files is DefaultConfigurableFileTree)) {
// The dependencies here are the following two: there is no need to add it to the parent, because the jar package directly enters the libs folder in its aar
//    implementation rootProject.files("libs/tingyun-ea-agent-android-2.15.4.jar")
//    implementation fileTree(dir: "libs", include: ["*.jar"])
} else { 
    parentProject.key.dependencies.add(childConfig.name, childDepency)
}

5.3. It is found that aar/jar has multiple dependencies

 implementation (name: 'libXXX', ext: 'aar') 
 implementation files("libXXX.aar")

Solution: if the first one is used, the second one will be merged into aar, resulting in class duplication

5.4. Finding aar new posture dependence

configurations.maybeCreate("default")
artifacts.add("default", file('lib-xx.aar'))

The above code makes aar a separate module for other modules to rely on. Default config is actually the holder of the module's final output aar. Default config can hold a list of AARS, so adding aar manually to default config is also equivalent to the product packaged from the current module.

Solution: through childproject configurations. maybeCreate("default"). Artifacts finds all AARS added and publishes localmaven separately

   fun getAarByArtifacts(childProject: Project): MutableList<String> {
        //Find all current artifacts Add ("default", file ('xxx. aar ')) depends on the incoming aar
        var listArtifact = mutableListOf<DefaultPublishArtifact>()
        var aarList = mutableListOf<String>()
        childProject.configurations.maybeCreate("default").artifacts?.forEach {
            if (it is DefaultPublishArtifact && "aar".equals(it.type)) {
                listArtifact.add(it)
            }
        }

        //Copy one to localMaven
        listArtifact.forEach {
            it.file.copyTo(File(FileUtil.getLocalMavenCacheDir(), it.file.name), true)
            //Cull suffix (. aar)
            aarList.add(removeExtension(it.file.name))
        }

        return aarList
    }

5.5. It is found that Android modules can be packaged as jar s

Solution: find the task named jar and inject the uploadLocalMaven task behind the jar task. The code is implemented in jarflatlocalmaven kt

5.6. It is found that there is a bug in arouter, and transform fails to pass outputprovider Deleteall() cleans up the old cache

Solution: as a result, the arouter problem is solved and the code is merged. However, no new plug-in version was released to Maven central, so I helped arouter solve it first. However, arouter did not start incremental compilation, which led to the extremely slow operation of DexArchiveBuilderTask, that is, it was very slow to pack dex. In the project, I changed the source code of arouter plug-in to support the doubling of TransForm incremental speed. The specific details will be discussed in the next section together with dex speed optimization.

6, Outlook for the next step

At present, the preliminary version has been able to run in the project, but there are still many small problems that continue to emerge and solve. It's a long way to go. I'll look up and down..

Next step plan:

dexBuild task optimization
Solve various compatibility problems

Programming VIP