1, Introduction
"Wife" and "mother" fell into the water at the same time. Who should be saved first?
As the saying goes: coding for five minutes, conflict resolution for two hours. As for Java development, the first reaction to seeing such exceptions as ClassNotFoundException and NoSuchMethodException is to arrange packages. After a routine and unconventional operation, you will often find the same Jar package and introduce multiple different versions. At this time, you can generally exclude the low version and keep the high version, because generally Jar packages are downward compatible. However, if there is version incompatibility, it will fall into the dilemma of "wife and mother fall into the water at the same time, who to save first". If this incompatibility happens between middleware dependency and business self dependency, it will be even more difficult.
As shown in the figure below, Project represents our Project, Dependency A represents our business dependency, and Dependency B represents middleware dependency. If both business dependency and middleware dependency depend on the same Jar package C, but the versions are different, they are version 0.1 and version 0.2 respectively. Unfortunately, the two versions still conflict, Some old functions only exist in the lower version of 0.1, and some new functions only exist in the higher version of 0.2. It's really "wife and mother fall into the water at the same time, and you can't save anyone first".
(picture from SOFAArk official website)
As the saying goes: the development without Jar package conflict must be a fake Java development; The development that has not solved the Jar package conflict is not a qualified Java development. In the recent project, we need to use Guava's high version Jar package, but we found that the middleware relies on the Jar package with low version and incompatible with the high version. In the face of this dilemma, we must save both "wife" and "mother", so we began to seek a solution.
2, Incompatible dependency conflict resolution
"Wife" and "mother" should be saved. How?
First of all, we think about whether we can copy the high version of Guava code that needs to be used and directly put it into our project, but doing so will bring several problems:
-
As a basic library with rich functions, Guava often has dependencies on a part of the code and many other codes, which will affect the whole body, and the workload will be much larger than expected;
-
The copied code can only be maintained manually. If the official fixes the problem, refactors the code or adds functions, and we want to upgrade, we can only do it again. Therefore, we can only think of other schemes, which can only be used as the final scheme.
Then, we think that a Java Class is loaded into the JVM virtual machine, which is different from another Class. First, they are two different classes with different full paths, but they are loaded by different Class loaders. In the JVM virtual machine, they are still considered as two different classes. Therefore, we are trying to find a solution from the Class loader. Inside Alibaba, there is a Pandora component, just as its name is like a magic box. It will install the dependencies of middleware into Pandora (internally called Sar package). In this way, the dilemma of "wife and mother fall into the water at the same time, who to save first" can be avoided in the middleware and business code.
Similarly, in similar scenarios, such as application consolidation and deployment, it can also play a powerful role. However, Pandora is only used internally by Alibaba and is not open source. Ant financial also has such a component, which is open source, It's called SOFAArk (official website. Those interested can go to the official website to learn about the principle and use of SOFAArk). We feel that we have found Mr.Right, so we began to study how to use SOFAArk. Like Pandora, SOFAArk also uses different classloaders to load different versions of tripartite dependencies, so as to isolate classes and completely solve the problem of package conflict, which requires us to Package the related dependencies into Ark Plugin (see the official SOFAArk documentation).
For the company, the benefits of such a scheme are relatively large. After being packaged into Ark Plugin, the whole company can share and the business parties can benefit. However, for our project, it is undoubtedly too heavy to adopt such a scheme. Therefore, we contacted the middleware students and asked whether there was a plan to introduce similar isolation components to solve the dependency conflict between middleware and business code. The answer was that the company's current package conflict was not a strong pain point and had no plan to introduce it for the time being. Therefore, we can only put SOFAArk aside and continue to find new solutions.
Next, we think that since Pandora/SOFAArk uses Class loading to isolate classes with the same path, if we make the groupids of the conflicting two version libraries different, even the full path of the Class with the same name is different, so there must be different classes in the JVM. If the Pandora/SOFAArk isolation method is called logical isolation, it is equivalent to physical isolation. To achieve this, it is easy to achieve this with the help of IDE refactoring function or global replacement function.
When we are ready to roll up our sleeves and do it, we can't help thinking that such pain points should have been encountered long ago, especially for basic class libraries such as Guava and Commons. Conflicts are inevitable. Predecessors should have found an elegant scratching posture. So we went to search for relevant articles. Sure enough, Maven shade plugin is the elegant scratching posture. The principle of Maven plugin is to remap the package path of the class to isolate incompatible Jar packages.
3, Maven shade plugin to resolve dependency conflicts
Finally, how to configure and use Maven shade plugin to map Guava into our own customized Jar package to realize isolation from middleware Guava. The whole process is quite clear. It is mainly to create a maven project, introduce dependencies, configure the warehouse address to be published, introduce the compilation and packaging plug-in and Maven shade plugin plug-in, configure the mapping rules (between tags), and then compile and package and distribute to the Maven warehouse. The configuration of pom.xml is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.shaded.example</groupId>
<artifactId>guava-wrapper</artifactId>
<version>${guava.wrapper.version}</version>
<name>guava-wrapper</name>
<url>https://example.com/guava-wrapper</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<!- Version and guava The version is basically the same ->
<guava.wrapper.version>27.1-jre</guava.wrapper.version>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>27.1-jre</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.3</version>
<configuration>
<source>${maven.compiler.source}</source>
<target>${maven.compiler.target}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.3.2</version>
<executions>
<execution>
<id>default-jar</id>
<goals>
<goal>jar</goal>
</goals>
<phase>package</phase>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>2.4</version>
<executions>
<execution>
<id>default-sources</id>
<goals>
<goal>jar-no-fork</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.1</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<!-- Rename rule configuration -->
<relocations>
<relocation>
<!-- Source package path -->
<pattern>com.google.guava</pattern>
<!-- Destination package path -->
<shadedPattern>com.google.guava.wrapper</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>com.google.common.wrapper</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.thirdparty</pattern>
<shadedPattern>com.google.wrapper.thirdparty</shadedPattern>
</relocation>
</relocations>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"/>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<distributionManagement>
<!- Maven Warehouse configuration, omitted ->
</distributionManagement>
</project>
After the project introduces the newly packaged guava wrapper, import selects to import the relevant classes we need from this package. As follows:
<dependency> <groupId>com.vivo.internet</groupId> <artifactId>guava-wrapper</artifactId> <version>27.1-jre</version> </dependency>
4, Conclusion
In order to use multiple versions of incompatible Jar packages in the same project, we first think of maintaining the code manually, but the workload and maintenance cost are very high, Then we think of isolation through class loaders (open source solution SOFAArk), but the related dependencies need to be packaged into Ark Plugin. The solution is undoubtedly a little too heavy. Finally, the Maven shade plugin plug-in is renamed and packaged, which gracefully solves the conflict problem of incompatible Jar packages of multiple versions in the project. From the problem, we explore the solution step by step, and the final Maven shade plugin plug-in party Although the case seems to be consistent with the essence of manual self maintenance code and seems to have returned to the origin, in fact, the elegance of the final scheme is much higher than that at the beginning. Just like the road of life, it rises spirally and advances curvilinearly.
If you encounter similar scenarios that require the coexistence of incompatible Jar packages, you can consider using the Maven shade plugin plug-in. This method is lightweight and can be used in scenarios where there are individual incompatible Jar package conflicts in the project. It is simple, effective and low cost. However, if Jar package conflicts are common and have become an obvious or common pain point, it is recommended to consider the scheme of loader isolation similar to Pandora and SOFAArk mentioned in the article.
Author: vivo Internet server team - Zhang Wei