Data backup of HBase
1.1 backup the table based on the class provided by HBase
Use the class provided by HBase to export the data of a table in HBase to HDFS, and then to the test HBase table.
(1) = = export from hbase table to HDFS==
[hadoop@node01 shells]$ hbase org.apache.hadoop.hbase.mapreduce.Export myuser /hbase_data/myuser_bak
(2) = = file import hbase table==
Create backup target table in hbase shell
Import the data on HDFS into the backup target table
hbase org.apache.hadoop.hbase.mapreduce.Driver import myuser_bak /hbase_data/myuser_bak/*
All of the above are full backups of data. In the later stage, incremental data backup of tables can also be realized. Incremental backup is similar to full backup, except that time stamp should be added later.
HBase data export to HDFS
HBase org.apache.hadoop.hbase.mapreduce.export test / HBase? Data / test? Bak? Increment start timestamp end timestamp
1.2 backup table based on snapshot snapshot
The migration and copy of HBase data are realized by snapshot snapshot. This method is more commonly used and efficient, and is also the most recommended data migration method.
HBase's snapshot is actually a collection of = = metadata = = information (file list). Through the collection of these metadata information, the data of the table can be rolled back to the data at the time of snapshot.
- First of all, we need to understand the so-called LSM type system structure of HBase. We know that in HBase, the data is written to the memory first. When the data in the memory reaches certain conditions, it will flush to HDFS to form HFile. Later, it is not allowed to modify or delete in place.
- If you want to update or delete, you can only append and write new files. Since the data will not be modified or deleted in place after being written, this is where the snapshot does the article. When making a snapshot, you only need to create pointers (metadata sets) for all the files corresponding to the snapshot table. When recovering, you only need to find the corresponding files according to these pointers to recover. This is the simplest description of the principle. The figure below is a simple process to describe the snapshot:
1.3 snapshot practice
- 1. Create a snapshot of a table
snapshot 'tableName', 'snapshotName'
2. View snapshot
Find the snapshot starting with test
- 3. Restore snapshot
ps: you need to disable the table. First, set the table to unavailable state, and then restore ﹣ snapshot
disable 'tableName' restore_snapshot 'snapshotName' enable 'tableName'
4. Delete snapshot
5. Migrate snapshot
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot \ -snapshot snapshotName \ -copy-from hdfs://src-hbase-root-dir/hbase \ -copy-to hdfs://dst-hbase-root-dir/hbase \ -mappers 1 \ -bandwidth 1024 //For example: hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot \ -snapshot test \ -copy-from hdfs://node01:8020/hbase \ -copy-to hdfs://node01:8020/hbase1 \ -mappers 1 \ -bandwidth 1024
Note: this method is used to migrate the snapshot table to another cluster. MR is used to copy the data quickly. When using, remember to set the bandwidth parameter to avoid online business failure caused by full network.
6. Using bulkload to import snapshot
hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles \ hdfs://dst-hbase-root-dir/hbase/archive/datapath/tablename/filename \ tablename //For example: //Create a new table create 'newTest','f1','f2' hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles hdfs://node1:9000/hbase1/archive/data/default/test/6325fabb429bf45c5dcbbe672225f1fb newTest
dIncrementalHFiles hdfs://node1:9000/hbase1/archive/data/default/test/6325fabb429bf45c5dcbbe672225f1fb newTest