Programming VIP

This paper mainly introduces the environment deployment and configuration of ELK related components, and takes the system syslog as the source data input to test and verify the data reception of elasticsearch and the data display of Kibana. 1. Introduction to basic concepts and environment configuration 1.1 basic concept of Elk ELK is an o ...

Statement: 1 ***               2. Because it is a personal summary, write the article with the most concise words               3. If there is any mistake or improper place, please point out Introduction ...

Python project practice: analyze big data with PySpark Big data, as its name implies, is a large amount of data. Generally, these data are above PB level. PB is the unit of data storage capacity, which is equal to the 50th power of 2 bytes, or about 1000 TB in value. These data are characterized by a wide variety, including video, voice, pictu ...

In the previous chapter, we built the ES cluster. Interested friends can refer to it Introduction to the core concept of elasticSearch (XIII): docker building ES cluster Here we introduce the partition management of ES cluster index ES cluster index fragmentation management introduce Shard: because ES is a distributed search engine, t ...

Introduction to persistence Redis is an in memory database. If the database state in memory is not saved to disk, the database state in the server will be lost once the server process exits. So redis provides persistence function! What is persistence The working mechanism of using permanent storage media to save data and recover the saved ...

preparation Start hadoop cluster [amelia@hadoop102 hadoop-2.7.2]$ sbin/start-dfs.sh -help: output this command parameter [amelia@hadoop102 hadoop-2.7.2]$ hadoop fs -help rm Create / sanguo folder [amelia@hadoop102 hadoop-2.7.2]$ hadoop fs -mkdir /sanguo Check whether the sanguo file exists in hadoop 2. Upload -moveFromLocal: cut and ...

catalogue Initialize Spark streaming program 1, SparkSql parameter tuning settings 1. Set session time zone 2. Sets the maximum number of bytes a single partition can hold when reading a file 3. Set the threshold for merging small files 4. Sets the number of partitions to use when shuffling data with join or aggregate 5. Set the maximum ...

Hadoop environment configuration (Linux virtual machine) This semester, I chose the course of big data management and analysis, which mainly uses Hadoop framework for data analysis and application development. First, I will configure the environment be careful It's better to put JDK and Hadoop under / usr/local When adding environment ...

By default, when the watermark passes through the end of window and the previous data arrives, these data will be deleted. In order to avoid some late data being deleted, the concept of allowed lateness is generated. In short, allowed latency is for event time. After the watermark exceeds the end of window, it is also allowed to wait for a pe ...

Hadoop learning 1.Hadoop overview Infrastructure of a distributed systemIt mainly solves the problems of massive data storage and distributed computing 1.1 three major releases of Hadoop The original version of Apache was released in 2006Cloudera integrates many big data frameworks internally, and the corresponding product is CDH releas ...