Analysis on the speed of inserting a large amount of data into MongoDB database

Requirement background: a certain timing task generates thousands or more JSON data. This data has not been fully written into the database, and the data of the next scheduled task has been generated. How to solve the data congestion caused by this?

When spring boot is used to insert data into MongoDB database, the save method in MongoTemplate is used to complete the data storage operation.

The specific code implementation is as follows:

JSONArray For the data I get from the scheduled task.
for (int i = 0;i < jsonArray.size();i++){
     mongoTemplate.save(jsonArray.get(i),"Stored library name");
}

This way of storing data is too slow, because it is after traversing one storage, the efficiency is too low.

It can be used mongoCollection.insertMany This method can insert data in batches with high efficiency

The specific implementation code is as follows:

IDEA,SPringBoot,MongoDB

//First, create a tool class for MongoDB to connect to the database.
//If your database has a password, please refer to the code at the end of the article. That code is the use of user name, password connection database method, code from rookie tutorial, pro test effective.
public class MongoDBJDBC {
    static  String MONGO_IP = "database IP address";
    static  Integer MONGO_PORT = Database port number, the default is27017;
    public static MongoCollection getMongoDatabase(String databaseName,String collectionName){
        MongoCollection mongoCollection = null;
        try {
            MongoClient mongoClient = new MongoClient(MONGO_IP,MONGO_PORT);
            MongoDatabase mongoDatabase = mongoClient.getDatabase(databaseName);
            mongoCollection = mongoDatabase.getCollection(collectionName);
        }catch (Exception e){
            System.out.println(e.getClass().getName()+":"+e.getMessage());
        }
        return mongoCollection;
    }
}

//After getting the MongoCollection object, use its insertMany method to insert a collection.
public void insertData(){
//Firstly, remove method is used to delete a certain data, and then insert it to update batch data.
        Query query = new Query(Criteria.where("busi_type").       
                    is(TransformTime.getStringDate()));
        mongoTemplate.remove(query,"The name of the collection in the database");    

        List<Document> list = //Get the method that needs to be inserted into the database.

//Use the tool class to get the MongoCollection object of the specified database
        MongoCollection mongoCollection = MongoDBJDBC.getMongoDatabase("Database name","Database collection name");
        mongoCollection.insertMany(list);
    }

======================I am the gorgeous dividing line==========================

How to connect to MongoDB database when it has user name and password

import java.util.ArrayList;  
import java.util.List;  
import com.mongodb.MongoClient;  
import com.mongodb.MongoCredential;  
import com.mongodb.ServerAddress;  
import com.mongodb.client.MongoDatabase;  

public class MongoDBJDBC {  
    public static void main(String[] args){  
        try {  
            //To connect to MongoDB service, if it is a remote connection, you can replace "localhost" with the IP address of the server  
            //The two parameters of ServerAddress() are server address and port  
            ServerAddress serverAddress = new ServerAddress("localhost",27017);  
            List<ServerAddress> addrs = new ArrayList<ServerAddress>();  
            addrs.add(serverAddress);  

            //MongoCredential.createScramSha1Credential()The three parameters are user name, database name and password  
            MongoCredential credential = MongoCredential.createScramSha1Credential("username", "databaseName", "password".toCharArray());  
            List<MongoCredential> credentials = new ArrayList<MongoCredential>();  
            credentials.add(credential);  

            //Get MongoDB connection through connection authentication  
            MongoClient mongoClient = new MongoClient(addrs,credentials);  

            //Connect to database  
            MongoDatabase mongoDatabase = mongoClient.getDatabase("databaseName");  
            System.out.println("Connect to database successfully");  
        } catch (Exception e) {  
            System.err.println( e.getClass().getName() + ": " + e.getMessage() );  
        }  
    }  
} 

For more Java operation methods of MongoDB database, refer to my other blogs. How can I find the portal of other articles.

Keywords: Database MongoDB Java JSON

Added by ghettopixel on Tue, 30 Jun 2020 19:43:06 +0300