Summary of Presto+Ranger source code compilation problems
Recently, I want to integrate prestodb and ranger, so I want to know more about presto and add Ranger plug-ins.
To this end, I compiled the source code of presto and ran 0 7.1 (online version) compiled the source code. presto is compiled on win10, and ranger is compiled on Ubuntu 16 04, the main compilation process will be explained below
1, Presto source code compilation
1, Environment
WIN10,jdk1.8,presto-0.233,maven-3.8.2
2, Steps
1. Modify presto- maven-plugin
Presto is compiled using Presto Maven plugin. Failed to compile under Window because the class cannot be found during compilation and the file path is wrong.
The version of Presto Maven plugin used by Presto here is 0.3. Download the source code from GitHub prestodb / Presto Maven plugin 0.3.
Modify ServiceDescriptorGenerator
String className = classPath.substring(0, classPath.length() - 6).replace('/', '.');
replace with
String className = classPath.substring(0, classPath.length() - 6).replace(File.separatorChar, '.');
Because the class name is by replacing the file separator of the class path with The delimiter of Windows is \, so replace '/' with file separatorChar.
Then transfer Presto Maven plugin, mvn clean install -DskipTests to the local warehouse.
2. Modify the root directory POM xml
Do you want to install Python
It may be related to the Python environment. The document is generated by sphinx and commented without installation.
<module>presto-docs</module>
Do you want to create a GIT warehouse and download it by git clone
If you have, don't worry.
No, you need to add configuration in the element
<plugin>
<groupId>pl.project13.maven</groupId>
<artifactId>git-commit-id-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
Re annotation
<Build-Time>${git.build.time}</Build-Time>
<Git-Commit-Id>${git.commit.id}</Git-Commit-Id>
<Implementation-Version>${project.version}-${git.commit.id.abbrev}</Implementation-Version>
3. Configure unix like environment
Commands in Linux will be used during Presto compilation. Git Bash has some Unix like commands.
Set the environment variable Path of Windows, plus:% GIT_HOME%\usr\bin, where GIT_HOME is the installation Path of GIT. Generally, the default installation Path is C:\Program Files\Git.
If Intellij Idea is used, you need to restart Intellij Idea to take effect after configuring the environment variables.
4. Compile the project
mvn clean install -DskipTests
After the compilation is successful, you need to enter Presto main to modify the relevant configuration
3, Modify configuration
Modify Presto main \ etc \ config properties
# http port http-server.http.port=18080 # The embedded version of Discovery is enabled in the Coordinator, so it should be the URI of the Coordinator discovery.uri=http://localhost:18080 # Plug in folder plugin.dir=../presto-server/target/presto-server-0.240/presto-server-0.240/plugin #plugin.bundles=\ # ../presto-blackhole/pom.xml,\ # ../presto-memory/pom.xml,\ # ../presto-jmx/pom.xml,\ # ../presto-raptor/pom.xml,\ # ../presto-hive-hadoop2/pom.xml,\ # ../presto-example-http/pom.xml,\ # ../presto-kafka/pom.xml, \ # ../presto-tpch/pom.xml, \ # ../presto-local-file/pom.xml, \ # ../presto-mysql/pom.xml,\ # ../presto-sqlserver/pom.xml, \ # ../presto-postgresql/pom.xml, \ # ../presto-tpcds/pom.xml, \ # ../presto-i18n-functions/pom.xml,\ # ../presto-function-namespace-managers/pom.xml,\ # ../presto-druid/pom.xml
Modify Presto main \ etc \ catalog \ hive properties
Set hive Rename properties to hive properties. bak. After startup, all connectors under the catalog directory will be loaded by default. If there is no hadoop, hive and other local environment, an error will be reported, resulting in startup failure. For other self-contained catalogs, if not used, please modify the name to avoid unnecessary trouble caused by startup.
Generally, you can verify mysql.
Modify prestosystemrequirements java
Note the code of Presto main module PrestoSystemRequirements, and find the relevant code fragments with the IDEA search function
failRequirement("Presto requires Linux or Mac OS X (found %s)", osName);
Modify the file descriptor size limit (manually changed to 10000):
private static OptionalLong getMaxFileDescriptorCount()
{
try {
MBeanServer mbeanServer = ManagementFactory.getPlatformMBeanServer();
//Object maxFileDescriptorCount = mbeanServer.getAttribute(ObjectName.getInstance(OPERATING_SYSTEM_MXBEAN_NAME), "MaxFileDescriptorCount");
Object maxFileDescriptorCount = 10000;
return OptionalLong.of(((Number) maxFileDescriptorCount).longValue());
}
catch (Exception e) {
return OptionalLong.empty();
}
}
The following a/b steps can be selected according to the purpose. Step a does not contain any plug-ins and can quickly debug the interface. Step B contains a complete plug-in with complete functions:
a. Next, comment out the PluginManager plug-in,
/*for (File file : listFiles(installedPluginsDir)) {
if (file.isDirectory()) {
loadPlugin(file.getAbsolutePath());
}
}
for (String plugin : plugins) {
loadPlugin(plugin);
}*/
Rename all the configuration files of etc/catalog to properties.bak.
b. Download presto-server-0.207.tar.gz Unzip the file to any directory. Take the plugin directory here as the plugin directory of the IDEA project. You need to open the file pluginmanagerconfig Java, modify as follows:
// private File installedPluginsDir = new File("plugin");
private File installedPluginsDir = new File("D:\\presto-server-0.207\\plugin");
The configuration file of / etc/catalog is renamed as required properties.bak.
Finally, run PrestoServer.
4, Start
First, configure the jvm parameter VM options to start
-Xmx800M -XX:+UseConcMarkSweepGC -XX:+ExplicitGCInvokesConcurrent -XX:+CMSClassUnloadingEnabled -XX:+AggressiveOpts -XX:+HeapDumpOnOutOfMemoryError -XX:ReservedCodeCacheSize=150M -Dsun.security.krb5.debug=true -Dconfig=etc/config.properties -Dlog.levels-file=etc/log.properties
Then select Presto main project, and the main class is com facebook. presto. server. PrestoServer
[the external chain image transfer fails, and the source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-ufztckkn-1643349229752) (C: \ users \ luohaizhang \ appdata \ roaming \ typora \ typora user images \ 1643106498641. PNG)]
5, Problems encountered
1,Cannot resolve symbol 'SqlBaseParser
When running the project, there is an error that the Presto parser module Cannot resolve symbol 'SqlBaseParser lacks code, for example, because the source code does not contain the generated code of anltr4. You can run the command that generates anltr4 code in the root directory.
mvn antlr4:antlr4
After executing the command, the error still does not disappear. We can see the structure of the project. File - > project structure - > modules - > Presto parser, set the target - > generated Sources - > anltr4 of Presto parser to Sources
[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-wuo7n1a5-1643349229753) (C: \ users \ luohaizhang \ appdata \ roaming \ typora \ typora user images \ 16431067431. PNG)]
Reference article:
https://blog.csdn.net/qq_36160730/article/details/109067896
https://www.cnblogs.com/ginponson/p/9500663.html
2, Compiler process of ranger source code
1. Deployment preparation
ranger: go to apach's official website to download http://ranger.apache.org/download.html , ranger0 is used this time 7.1 at https://codeload.github.com/apache/ranger/zip/refs/tags/release-ranger-0.7.1
Maven: enter the Maven official website of Apache http://maven.apache.org/download.cgi Download, this time using Maven 3 eight point two
Python2.7: Because Python 2.0 is required in compilation and trial 7 version of python, so if it is a Centos6 system, you need to manually upgrade Python to Python 2 7. Refer to historical articles for the upgrade process Python upgrade
MySQL: MySQL database is required. If MySQL is unavailable, a set of MySQL needs to be deployed. Please refer to the historical article for the deployment method MySQL deployment
MySQL connector Java: download from MySQL official website https://dev.mysql.com/downloads/connector/j/5.1.html
JDK version: the JDK version used this time is 1.8.0_ three hundred and twenty-one
The specific process of installing these components will not be repeated. Please refer to the two articles posted below.
2. Compilation
Set ranger0 7.1 after downloading, upload it to the ubuntu directory and put it in your user's home directory. Note that the current account must have permission, otherwise you will encounter many problems of insufficient permission;
Unzip and import into idea. Or you don't need the idea to decompress it directly. For the convenience of reading the code, the idea makes subsequent plug-in modifications and so on.
Modify POM. In the ranger root directory XML file
<repository>
<id>apache.snapshots.https</id>
<name>Apache Development Snapshot Repository</name>
<!-- <url>https://repository.apache.org/content/repositories/snapshots</url>-->
<url>https://repo1.maven.org/maven2/</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>apache.public.https</id>
<name>Apache Development Snapshot Repository</name>
<url>https://repository.apache.org/content/repositories/public</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
Execute mvn command and package
mvn clean compile package assembly:assembly install -Dmaven.test.skip=true;
All unit tests are omitted here, because some unit tests have problems, which will affect the compilation.
Note that python2 must be installed, otherwise you can't compile Ranger util
Results of successful compilation:
[INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for ranger 0.7.1: [INFO] [INFO] ranger ............................................. SUCCESS [ 3.591 s] [INFO] Jdbc SQL Connector ................................. SUCCESS [ 1.632 s] [INFO] Credential Support ................................. SUCCESS [ 0.479 s] [INFO] Audit Component .................................... SUCCESS [ 1.731 s] [INFO] Common library for Plugins ......................... SUCCESS [ 2.892 s] [INFO] Installer Support Component ........................ SUCCESS [ 0.265 s] [INFO] Credential Builder ................................. SUCCESS [ 0.405 s] [INFO] Embedded Web Server Invoker ........................ SUCCESS [ 0.341 s] [INFO] Key Management Service ............................. SUCCESS [ 0.891 s] [INFO] ranger-plugin-classloader .......................... SUCCESS [ 0.198 s] [INFO] HBase Security Plugin Shim ......................... SUCCESS [ 1.319 s] [INFO] HBase Security Plugin .............................. SUCCESS [ 1.280 s] [INFO] Hdfs Security Plugin ............................... SUCCESS [ 4.005 s] [INFO] Hive Security Plugin ............................... SUCCESS [ 2.247 s] [INFO] Knox Security Plugin ............................... SUCCESS [ 0.550 s] [INFO] Storm Security Plugin .............................. SUCCESS [ 0.642 s] [INFO] YARN Security Plugin ............................... SUCCESS [ 0.462 s] [INFO] Ranger Util ........................................ SUCCESS [ 5.051 s] [INFO] Unix Authentication Client ......................... SUCCESS [ 0.330 s] [INFO] Security Admin Web Application ..................... SUCCESS [ 13.658 s] [INFO] KAFKA Security Plugin .............................. SUCCESS [ 1.504 s] [INFO] SOLR Security Plugin ............................... SUCCESS [ 3.403 s] [INFO] NiFi Security Plugin ............................... SUCCESS [ 0.354 s] [INFO] Unix User Group Synchronizer ....................... SUCCESS [ 1.097 s] [INFO] Ldap Config Check Tool ............................. SUCCESS [ 2.643 s] [INFO] Unix Authentication Service ........................ SUCCESS [ 2.010 s] [INFO] KMS Security Plugin ................................ SUCCESS [ 2.111 s] [INFO] Tag Synchronizer ................................... SUCCESS [ 2.863 s] [INFO] Hdfs Security Plugin Shim .......................... SUCCESS [ 0.515 s] [INFO] Hive Security Plugin Shim .......................... SUCCESS [ 1.543 s] [INFO] Knox Security Plugin Shim .......................... SUCCESS [ 0.665 s] [INFO] YARN Security Plugin Shim .......................... SUCCESS [ 0.543 s] [INFO] Storm Security Plugin shim ......................... SUCCESS [ 0.536 s] [INFO] KAFKA Security Plugin Shim ......................... SUCCESS [ 0.572 s] [INFO] SOLR Security Plugin Shim .......................... SUCCESS [ 0.848 s] [INFO] Atlas Security Plugin Shim ......................... SUCCESS [ 0.332 s] [INFO] KMS Security Plugin Shim ........................... SUCCESS [ 0.577 s] [INFO] Ranger Hive Utils .................................. SUCCESS [ 1.189 s] [INFO] ranger-examples .................................... SUCCESS [ 0.126 s] [INFO] Ranger Examples - Conditions and ContextEnrichers .. SUCCESS [ 0.523 s] [INFO] Ranger Examples - SampleApp ........................ SUCCESS [ 0.844 s] [INFO] Ranger Examples - Ranger Plugin for SampleApp ...... SUCCESS [ 1.209 s] [INFO] Ranger Tools ....................................... SUCCESS [ 1.728 s] [INFO] Atlas Security Plugin .............................. SUCCESS [ 0.976 s] [INFO] Unix Native Authenticator .......................... SUCCESS [ 1.484 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 07:03 min [INFO] Finished at: 2022-01-28T10:38:15+08:00 [INFO] ------------------------------------------------------------------------
We can see the generated tar in the target folder of the root directory GZ package.
root@ubuntu:/home/luohaizhang/ideaProjects/ranger/ranger-release-ranger-0.7.1# ls target antrun ranger-0.7.1-hbase-plugin.tar.gz ranger-0.7.1-kms.tar.gz ranger-0.7.1-solr-plugin.tar.gz ranger-0.7.1-usersync.tar.gz archive-tmp ranger-0.7.1-hbase-plugin.zip ranger-0.7.1-kms.zip ranger-0.7.1-solr-plugin.zip ranger-0.7.1-usersync.zip maven-shared-archive-resources ranger-0.7.1-hdfs-plugin.tar.gz ranger-0.7.1-knox-plugin.tar.gz ranger-0.7.1-src.tar.gz ranger-0.7.1-yarn-plugin.tar.gz ranger-0.7.1-admin ranger-0.7.1-hdfs-plugin.zip ranger-0.7.1-knox-plugin.zip ranger-0.7.1-src.zip ranger-0.7.1-yarn-plugin.zip ranger-0.7.1-admin.tar.gz ranger-0.7.1-hive-plugin.tar.gz ranger-0.7.1-migration-util.tar.gz ranger-0.7.1-storm-plugin.tar.gz rat.txt ranger-0.7.1-admin.zip ranger-0.7.1-hive-plugin.zip ranger-0.7.1-migration-util.zip ranger-0.7.1-storm-plugin.zip version ranger-0.7.1-atlas-plugin.tar.gz ranger-0.7.1-kafka-plugin.tar.gz ranger-0.7.1-ranger-tools.tar.gz ranger-0.7.1-tagsync.tar.gz ranger-0.7.1-atlas-plugin.zip ranger-0.7.1-kafka-plugin.zip ranger-0.7.1-ranger-tools.zip ranger-0.7.1-tagsync.zip
3. Install and start Ranger admin
2.3.1 modify configuration file
# Enter the target directory cd /opt/apache-ranger-1.2.0/target/ # Unzip ranger-1.2.0-admin tar. gz tar -zxvf ranger-1.2.0-admin.tar.gz # Enter the ranger-1.2.0-admin directory cd ranger-1.2.0-admin # Modify install properties vim install.properties SQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java-8.0.17.jar // Modify the jar package and path downloaded in preparation db_root_user=root db_root_password=123456 db_host=192.168.56.105 db_name=ranger db_user=rangeradmin db_password=rangeradmin # You can log off the following #Source for Audit Store. Currently only solr is supported. # * audit_store is solr ## audit_store=solr # * audit_solr_url URL to Solr. E.g. http://<solr_host>:6083/solr/ranger_audits ## audit_solr_urls= ## audit_solr_user= ## audit_solr_password= ## audit_solr_zookeepers=
There is no need for solr component statistics here, just testing
Call script/ setup.sh installation
Note that if the following errors are reported in the middle
2019-08-20 08:54:22,460 [I] '/usr/local/java/bin/java' command found setup.sh:Line 325: bc: Command not found setup.sh: Line 325:[: -eq: Expected unary expression
That is, the bc command is missing. You can install it.
Note that mysql must have access rights and must not limit the host of the current machine.
Finally, the following results appear
2019-08-20 09:00:18,240 [I] --------- Verifying Ranger DB connection --------- 2019-08-20 09:00:18,240 [I] Checking connection.. geradmin' -p '********' -noheader -trim -c \; -query "SELECT version();" Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. 2019-08-20 09:00:18,838 [I] Checking connection passed. 2019-08-20 09:00:19,091 [I] DB FLAVOR :MYSQL 2019-08-20 09:00:19,092 [I] --------- Verifying Ranger DB connection --------- 2019-08-20 09:00:19,092 [I] Checking connection.. geradmin' -p '********' -noheader -trim -c \; -query "SELECT version();" Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. 2019-08-20 09:00:19,660 [I] Checking connection passed. Installation of Ranger PolicyManager Web Application is completed.
Indicates that the installation is complete.
4 . Start Ranger admin
# Enter ews directory cd ews # Start service ./ranger-admin-services.sh start /** Under normal conditions, the following results occur:*/ Starting Apache Ranger Admin Service Apache Ranger Admin Service with pid 236275 has started. # Check to see if it starts ps -ef|grep ranger or netstat -lntp|grep 6080
After startup, you can log in and verify with the web. The default port is 6080, and the default user name and password are admin
Subsequent self verification is enough.
If a new component is added later, such as presto's plugin, after configuration, compile and start verification.
Reference article:
https://www.cnblogs.com/gjc592/p/11383778.html
https://blog.csdn.net/mm_bit/article/details/51208500