SpringBoot+MySql+MongoDB+Spark+Redis+Neo4j+Swagger framework integration Demo
1 prepare the environment (there is already one to skip)
-
Docker (docker under Linux is recommended)
-
Docker in Windows
-
Need computer support and turn on Hyper-v (Note: Hpyer-v is incompatible with previous VMware)
-
-
Docker in Linux requires:
- VMware Workstation (recommended) or Linux server
-
-
IntelliJ IDEA
1.1 VMware Workstation (virtual machine)
1.1.1 general
- A computer installs multiple isolated operating systems as its own servers
1.1.2 introduction
- VMware Workstation (Chinese name "Weirui workstation") is a powerful desktop virtual computer software, which provides the best solution for users to run different operating systems simultaneously on a single desktop and develop, test and deploy new applications. Vmware workstation can simulate a complete network environment and portable virtual machine on a physical machine. Its better flexibility and advanced technology are better than other virtual computer software on the market. For enterprise IT developers and system administrators, VMware's characteristics in virtual network, real-time snapshot, dragging shared folder, supporting PXE and so on make IT an essential tool.
1.1.3 Getting Started
-
install
1.1.4 application
Building an Ubuntu server with VMware Workstation
-
Mirror file: Download Ubuntu Desktop
-
Vmware installation Ubuntu
[the external link image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-yay1ac6n-1626952064695) (H: \ newborn \ note \ framework \ medias\image-20210722133106943.png)]
1.2 Docker
1.2.1 general
- One click deployment and management of operating environment
1.2.2 introduction
- Docker is an open source application container engine, which allows developers to package their applications and dependency packages into a portable image, and then publish them to any popular Linux or Windows machine, or realize virtualization. Containers are completely sandboxed, and there will be no interface between them.
1.2.3 Getting Started
-
Install Docker Engine on Ubuntu
# Uninstall old version sudo apt-get remove docker docker-engine docker.io containerd runc # Installation related dependencies sudo apt-get update sudo apt-get install \ apt-transport-https \ ca-certificates \ curl \ gnupg \ lsb-release # Add GPG key curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg # Install docker sudo apt-get update sudo apt-get install docker-ce docker-ce-cli containerd.io # hello-world sudo docker run hello-world
1.2.4 application
Docker common commands
# As long as it can be distinguished, id: 88 * * * * can not be written completely
docker images
# delete mirror
docker rmi 88
# Show all containers
docker container ls -a
# Stop container
docker stop 88
# Delete container
docker rm 88
# Self starting container
docker update --restart=always 88
# Careful operation
# Start all container commands in docker
docker start $(docker ps -a | awk '{ print $1}' | tail -n +2)
# Close all containers in docker
docker stop $(docker ps -a | awk '{ print $1}' | tail -n +2)
# Delete all container commands in docker
docker rm $(docker ps -a | awk '{ print $1}' | tail -n +2)
# Delete all images in docker
docker rmi $(docker images | awk '{print $3}' |tail -n +2)
2 SpringBoot
2.1 general
- Spring is a development framework of Java, and Spring Boot is an extension of spring, which is faster and more efficient
2.2 introduction
- Create a stand-alone Spring application
- Directly embed Tomcat, Jetty or Under Trailer (no need to deploy WAR file)
- Provide opinionated "beginner" dependencies to simplify build configuration
- Configure Spring and third-party libraries as automatically as possible
- Provide production preparation functions, such as measurement, health check and externalized configuration.
- There is absolutely no need to generate code or xml configuration.
2.3 Quickstart Guide
- There are many ways, such as
- Through the online website, select the favorite modules for construction (relatively standardized, for reference only) [the external chain picture transfer fails, and the source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-co2cbfnk-1626952064696) (H: \ newborn \ note \ framework \ medias \ image-2021072211301051. PNG)]
- Create a new Maven project through IDEA and work in POM XML add related dependencies (recommended) [the external chain image transfer fails, and the source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-g5rnkhdm-1626952064700) (H: \ newborn \ note \ framework \ medias \ image-2021072212351879. PNG)]
2.4 application
Smart recommendation Demo step 1: Project Construction
- Dependency: POM xml
- Configuration: Resources / application properties
- Startup class: java/com... / xxxapplication java
- Package: java/com/
- Configuration: config
- Control: controller
- Entity (neo4j): entity
- Entity (mysql, mongodb): pojo
- Operation mapper (mysql): mapper
- Operation: godb, repository (monoj)
- Logic: service
- Logical implementation: service/impl
- Tools: utils
- Test: test/java/com... / xxxapplicationtests java
3 MySql (store FriendInfo)
3.1 general
- Easy to install relational database
3.2 introduction
- MySQL is a relational database management system developed by MySQL AB company in Sweden, which is a product of Oracle. MySQL is one of the most popular relational database management systems. In terms of WEB application, MySQL is one of the best RDBMS (Relational Database Management System) application software.
- MySQL is a relational database management system. Relational database saves data in different tables instead of putting all data in a large warehouse, which increases speed and flexibility.
- The SQL language used by MySQL is the most commonly used standardized language for accessing databases. MySQL software adopts the dual authorization policy, which is divided into community version and commercial version. Due to its small size, fast speed and low overall cost of ownership, especially the open source, MySQL is generally selected as the website database for the development of small and medium-sized websites.
3.3 Getting Started
-
Deploying Mysql container using Docker
# create directory cd /home/newborne mkdir -p mysql cd mysql mkdir -p conf logs data # Pull image sudo docker pull mysql:5.7 # Create container sudo docker run -p 3306:3306 \ -v /home/newborne/mysql/conf:/etc/mysql \ -v /home/newborne/mysql/logs:/var/log/mysql \ -v /home/newborne/mysql/data:/var/lib/mysql \ -e MYSQL_ROOT_PASSWORD=****** \ -d mysql:5.7
3.4 application
Smart recommendation Demo step 2: MySql integration
-
surface
-
friend_info
/* Navicat Premium Data Transfer Source Server : 192.168.111.133_3306 Source Server Type : MySQL Source Server Version : 50735 Source Host : 192.168.111.133:3306 Source Schema : yizhi Target Server Type : MySQL Target Server Version : 50735 File Encoding : 65001 Date: 22/07/2021 18:53:26 */ SET NAMES utf8; SET FOREIGN_KEY_CHECKS = 0; -- ---------------------------- -- Table structure for friend_info -- ---------------------------- DROP TABLE IF EXISTS `friend_info`; CREATE TABLE `friend_info` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, `mobile` varchar(50) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL COMMENT 'Telephone', `name` varchar(50) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL COMMENT 'full name', `age` int(11) NULL DEFAULT NULL COMMENT 'Age', `remark` varchar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL COMMENT 'remarks', PRIMARY KEY (`id`) USING BTREE ) ENGINE = InnoDB AUTO_INCREMENT = 11 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Dynamic; -- ---------------------------- -- Records of friend_info -- ---------------------------- INSERT INTO `friend_info` VALUES (1, '17345660628', 'Zou Zhi', 26, 'No. 72, guiyixing community, Zhizhishi Road, Luoshan County, Xinyang City, Henan Province'); INSERT INTO `friend_info` VALUES (2, '14589533442', 'Xiao Yu', 25, 'No. 79, Sheyan community, yimingyou Road, Lushi County, Sanmenxia City, Henan Province'); INSERT INTO `friend_info` VALUES (3, '18443280035', 'Fu Houxue', 20, 'No. 93, laixiaomingliu community, yidexiu Road, Shihe District, Xinyang City, Henan Province'); INSERT INTO `friend_info` VALUES (4, '13576126061', 'Xiao Tianzhi', 26, 'No. 43, zhirongyuanxiang community, houyuqi Road, Guangshan County, Xinyang City, Henan Province'); INSERT INTO `friend_info` VALUES (5, '13037652760', 'Kang Xiu', 26, 'No. 76, Weiqiu community, shenerhou Road, Jili District, Luoyang City, Henan Province'); INSERT INTO `friend_info` VALUES (6, '14765905634', 'Cao houding', 28, 'No. 10, Yijue Zhouqu community, yixianzhi Road, Potou Town, Jiyuan City, Henan Province'); INSERT INTO `friend_info` VALUES (7, '13652802885', 'Ye Qi', 18, 'No. 86, Xiuzhi road and Shiqin community, Xiangcheng City, Zhoukou City, Henan Province'); INSERT INTO `friend_info` VALUES (8, '14978566974', 'Ye Zhizhe', 29, 'No. 68, please juhuai community, gechenger Road, Puyang County, Puyang City, Henan Province'); INSERT INTO `friend_info` VALUES (9, '17017431488', 'Xue Yi', 23, 'No. 35 Yiyi community, Zhian Road, Linzhou City, Anyang City, Henan Province'); INSERT INTO `friend_info` VALUES (10, '15365926127', 'Tian Zhiqi', 17, 'No. 57, yijinxin community, Zhishen Road, Shancheng District, Hebi City, Henan Province'); SET FOREIGN_KEY_CHECKS = 1;
-
-
Dependency: POM xml
-
Configuration: Resources / application properties
# mysql database connection spring.datasource.driver-class-name=com.mysql.jdbc.Driver spring.datasource.url=jdbc:mysql://192.168.111.133:3306/yizhi?characterEncoding=utf-8&useSSL=false spring.datasource.username=root spring.datasource.password=******
-
Package: java/com/
- Control: Controller / friendinfocontroller java
- Entity: POJO / friendinfo java
- Operation mapper: mapper / friendinfomapper java
- Logic: Service / friendinfoservice java
- Logical implementation: Service / impl / friendinfoserviceimpl java
4 MongoDB (store FriendRecommend)
4.1 General
- Non relational database suitable for mass data storage
4.2 introduction
- MongoDB is a database based on distributed file storage. Written in C + +. It aims to provide scalable high-performance data storage solutions for WEB applications.
- MongoDB is a product between relational database and non relational database. It is the most functional and relational database among non relational databases. The data structure it supports is very loose. It is a json like bson format, so it can store more complex data types. Mongo's biggest feature is that the query language it supports is very powerful. Its syntax is a little similar to the object-oriented query language. It can almost realize most of the functions similar to single table query in relational database, and it also supports indexing of data.
4.3 Getting Started
-
Deploying MongoDB container using Docker
# create directory cd /home/newborne mkdir -p mongo cd mongo mkdir -p data backup conf # Pull image docker pull mongo:4.0.3 docker run -p 27017:27017 \ -v /home/newborne/mongo/data:/data/db \ -v /home/newborne/mongo/backup:/data/backup \ -v /home/newborne/mongo/conf:/data/configdb \ -d mongo:4.0.3 --auth docker exec -it *** /bin/bash db.createUser( { user: "root", pwd: "******", roles: [ { role: "userAdminAnyDatabase", db: "admin" }, "readWriteAnyDatabase" ] })
4.4 application
Smart recommendation Demo step 3: MongoDB integration
-
aggregate
- friend_recommend
-
Dependency: POM xml
<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-mongodb</artifactId> </dependency>
-
Configuration: Resources / application properties
# mongodb spring.data.mongodb.host=192.168.111.133 spring.data.mongodb.port=27017 spring.data.mongodb.database=yizhi spring.data.mongodb.username=root spring.data.mongodb.password=******
-
Package: java/com/
- Control: Controller / friendrecommendcontroller java
- Entity: POJO / friendrecommend java
- Operation warehouse: repository/FriendRecommendRepository
- Logic: Service / friendrecommendservice java
- Logical implementation: Service / impl / friendrecommendserviceimpl java
5 MongoDB-Spark (calculated by FriendRecommend)
5.1 Spark
5.1.1 general
- A fast and universal computing engine designed for large-scale data processing
5.1.2 introduction
- Apache Spark is a fast and universal computing engine designed for large-scale data processing. Spark is a general parallel framework similar to Hadoop MapReduce, which is open-source by UC Berkeley AMP lab (AMP lab at the University of California, Berkeley). Spark has the advantages of Hadoop MapReduce; But what is different from MapReduce is that the Job intermediate output results can be saved in memory, so there is no need to read and write HDFS. Therefore, spark can be better applicable to MapReduce algorithms that need iteration, such as data mining and machine learning.
- Spark is an open source cluster computing environment similar to Hadoop, but there are still some differences between them. These useful differences make spark perform better in some workloads. In other words, spark enables memory distributed data sets. In addition to providing interactive queries, it can also optimize iterative workloads.
- Spark is implemented in the Scala language, which uses Scala as its application framework. Unlike Hadoop, spark and scala can be tightly integrated, and scala can operate distributed datasets as easily as local collection objects.
- Although spark was created to support iterative jobs on distributed datasets, it is actually a supplement to Hadoop and can run in parallel in Hadoop file system. This behavior can be supported through a third-party clustering framework called Mesos. Spark is developed by the University of California, Berkeley AMP Laboratory (Algorithms, Machines, and People Lab), which can be used to build large, low latency data analysis applications.
5.2 Spark MLlib
5.2.1 general
- Spark's machine learning library aims to simplify the engineering practice of machine learning and facilitate its expansion to a larger scale.
5.2.2 introduction
-
MLlib consists of some general learning algorithms and tools, including classification, regression, clustering, collaborative filtering, dimensionality reduction, etc. at the same time, it also includes the underlying optimization primitives and the high-level pipeline API.
-
form
name explain data type Vectors, vectors with categories, matrices, etc Mathematical statistical calculation library Basic statistics, correlation analysis, random number generator, hypothesis test, etc Algorithm evaluation AUC, accuracy, recall, F-Measure, etc Machine learning algorithm Classification algorithm, regression algorithm, clustering algorithm, collaborative filtering, etc
5.2.3 application
Smart recommendation Demo step 4: Spark integration
-
Dependency: POM xml
<!-- spark relevant --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-mllib_2.12</artifactId> <version>2.4.3</version> </dependency> <dependency> <groupId>org.mongodb.spark</groupId> <artifactId>mongo-spark-connector_2.12</artifactId> <version>2.4.1</version> </dependency> -
Configuration: Resources / APP properties
# mongodb configuration spark.mongodb.output.friend.uri = mongodb://192.168.111.133:27017/yizhi.friend_recommend?readPreference=primaryPreferred # mysql configuration jdbc.driver-class-name=com.mysql.jdbc.Driver jdbc.url=jdbc:mysql://192.168.111.133:3306/yizhi?useUnicode=true&characterEncoding=utf8&autoReconnect=true&allowMultiQueries=true&useSSL=false jdbc.username=root jdbc.password=******
-
Package: java/com/
- utils/MLibRecommend.java
- utils/SparkFriendRecommend.java
6 Redis (CACHE)
6.1 general
- High performance key value non relational database
6.2 introduction
- Redis (Remote Dictionary Server), i.e. remote dictionary service, is an open source log type key value database written in ANSI C language, supporting network, memory based and persistent, and provides API s in multiple languages. Since March 15, 2010, the development of redis has been hosted by VMware. Since May 2013, the development of redis has been sponsored by pivot.
6.3 Getting Started
-
Using Docker to create Redis container
# create directory cd /home/newborne mkdir -p redis cd redis mkdir -p data conf vim /home/newborne/redis/conf/redis.conf # ============ # From https://redis.io/topics/config/ Get redis conf # Comment bind 127.0.0.1 requirepass ****** appendonly yes # ============ # Pull image docker pull redis:5.0.2 # Deploy Redis docker run -p 6379:6379 \ -v /home/newborne/redis/conf/redis.conf:/usr/local/etc/redis/redis.conf \ -v /home/newborne/redis/data:/data \ -d redis:5.0.2 redis-server /usr/local/etc/redis/redis.conf docker exec -it *** /bin/bash > auth ****** > set name 1 > get name
6.4 application
Step 5 of smart recommendation Demo: Redis integration
-
Key
-
Dependency: POM xml
<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-redis</artifactId> </dependency> <!-- spring2.X integrate redis what is needed common-pool2--> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-pool2</artifactId> <version>2.6.0</version> </dependency> -
Configuration: Resources / application properties
# redis spring.redis.host=192.168.111.133 spring.redis.port=6379 spring.redis.database= 0 spring.redis.password=****** spring.redis.timeout=1800000 # Connection pool spring.redis.lettuce.pool.max-active=20 spring.redis.lettuce.pool.max-wait=-1 # Maximum blocking waiting time (negative number indicates no limit) spring.redis.lettuce.pool.max-idle=5 spring.redis.lettuce.pool.min-idle=0
-
Package: java/com/
- Configuration: config / redisconfig java
7 Neo4j (knowledge map)
7.1 general
- Establish a non relational database of atlas through nodes and connections
7.2 introduction
- Neo4j is a high-performance, NOSQL graphical database that stores structured data on the network rather than in tables. It is an embedded, disk based and fully transactional Java persistence engine, but it stores structured data on the network (mathematically called a graph) rather than in a table. Neo4j can also be regarded as a high-performance graph engine with all the features of a mature database. Programmers work in an object-oriented, flexible network structure rather than strict, static tables - but they can enjoy all the benefits of an enterprise class database with full transactional features.
- Neo4j has attracted more and more attention because of its embedded, high performance, lightweight and other advantages
7.3 Get Started
-
Deploying Neo4j containers using Docker
docker pull neo4j mkdir -p neo4j cd neo4j mkdir -p data plugins logs conf import docker run \ -p 7474:7474 -p 7687:7687 \ -v /home/newborne/neo4j/data:/data \ -v /home/newborne/neo4j/plugins:/plugins \ -v /home/newborne/neo4j/logs:/logs \ -v /home/newborne/neo4j/conf:/var/lib/neo4j/conf \ -v /home/newborne/neo4j/import:/var/lib/neo4j/import \ --name neo4j-apoc \ -e NEO4J_apoc_export_file_enabled=true \ -e NEO4J_apoc_import_file_enabled=true \ -e NEO4J_apoc_import_file_use__neo4j__config=true \ -e NEO4JLABS_PLUGINS=\[\"apoc\"\] \ --env NEO4J_AUTH=neo4j/****** \ neo4j http://192.168.111.133:7474 # Query node match(x:Friend{}) return x -
Deployment under Windows
7.4 application
Step 6 of smart recommendation Demo: Neo4j integration
-
node
- Friend
-
contact
- FriendRelationship
-
Dependency: POM xml
<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-neo4j</artifactId> </dependency> -
Configuration: Resources / application properties
# neo4j spring.data.neo4j.uri=bolt://192.168.111.133:7687 spring.data.neo4j.username=neo4j spring.data.neo4j.password=******
-
Package: java/com/
- Node: entity / node / friendnode java
- Relationship: entity / relationship / friendrelationship java
- Operation warehouse: repository/FriendRelationshipRepository
- Logic: Service / friendrelationshipservice java
- Logical implementation: Service / impl / friendrelationshipserviceimpl java
8 Swagger
8.1 general
- An interface document generation tool, which also provides auxiliary functions for interface test calls
8.2 introduction
- Swagger is a normative and complete framework for generating, describing, invoking and visualizing RESTful Web services.
- Swagger's goal is to define a standard and language independent interface for REST API, so that people and computers can discover and understand services without accessing source code, documents or network traffic monitoring. When properly defined by swagger, users can understand the remote service and interact with the remote service with minimal implementation logic. Similar to the interface implemented for the underlying programming, swagger eliminates the guesswork that may occur when invoking services.
- Support API to automatically generate synchronized online documents: after using Swagger, you can directly generate documents through code, and you no longer need to write interface documents manually. It is very convenient for programmers and can save time to write documents to learn new technologies.
- Provide Web page online testing API: documents alone are not enough, and the documents generated by Swagger also support online testing. After the parameters and format are set, directly enter the value corresponding to the parameters on the interface to test the interface online.
8.3 Getting Started
-
@Cacheable
Property / method name explain value Cache name, required. It specifies the namespace in which your cache is stored cacheNames Similar to value, just choose one from two key Optional attribute. You can customize the cached key using the spiel tag -
@CachePut
Property / method name explain value Cache name, required. It specifies the namespace in which your cache is stored cacheNames Similar to value, just choose one from two key Optional attribute. You can customize the cached key using the spiel tag -
@CacheEvict
Property / method name explain value Cache name, required. It specifies the namespace in which your cache is stored cacheNames Similar to value, just choose one from two key Optional attribute. You can customize the cached key using the spiel tag allEntries Whether to clear all caches. The default value is false. If true is specified, all caches will be emptied immediately after the method call beforeInvocation Whether to clear the method before execution. The default value is false. If true is specified, the cache will be emptied before the method is executed
8.4 application
Smart recommendation Demo step 7: Swagger integration
-
Dependency: POM xml
<!--swagger--> <dependency> <groupId>io.springfox</groupId> <artifactId>springfox-swagger2</artifactId> <version>2.7.0</version> </dependency> <!--swagger ui--> <dependency> <groupId>io.springfox</groupId> <artifactId>springfox-swagger-ui</artifactId> <version>2.7.0</version> </dependency> -
Package: java/com/
- config/Swagger2Config.java
9 project address
Code cloud: https://gitee.com/newborne/yizhi-demo