Article catalog
Current situation of domestic database
Telecommunications, finance and other industries are the most advanced in IT and the most widely used in database. At the same time, they are also the industries with the highest threshold for database technology because of their complex system, great technical difficulty and high requirements for product stability and operation performance. Entering the telecommunications and financial industries has become the embodiment of the comprehensive strength of database manufacturers. In recent months, a number of domestic database manufacturers have won the bidding or commercial implementation of projects in the telecommunications and financial industries, which undoubtedly boosted the market's confidence in domestic databases.
According to the forecast, the scale of China's relational database software market will reach US $3.82 billion by 2024, and the overall market will have a compound annual growth rate of 23.3% in the next five years. According to the survey data of Zhiyan consulting, as of 2017, foreign manufacturers accounted for more than 80% of China's database market. With the development of information technology and innovation industry, it can be expected that the penetration space of database localization rate will increase rapidly in the future.
When it comes to domestic databases, you may know Damon, Jincang, Nantah and Shentong more or less. In addition to these four, do you still know other domestic database products? In addition, Alibaba, Tencent and Huawei have also launched domestic databases. Until today, domestic database manufacturers are far more than these.
Introduction to domestic database
This article only introduces some databases, such as DM, GBASE, OSCAR, PolarDB, TDSQL and GaussDB100.
DM
Founded in 2000, Wuhan Dameng database Co., Ltd. is a basic software enterprise under China Electronic Information Industry Group (CEC), specializing in the R & D, sales and service of database management system. At the same time, it can provide users with services such as big data platform architecture consulting, data technology plan planning, product deployment and implementation. Damon's products have been successfully applied in more than 30 industries such as finance, electric power, aviation, communication and e-government. There are Dameng 7, Dameng 8, etc. Dameng database management system is a high-performance database management system with completely independent intellectual property rights launched by Dameng company, referred to as DM. At present, the latest version has reached 8.0, referred to as DM8.
GBASE
Tianjin NANDA General Data Technology Co., Ltd., established in 2004, is a well-known enterprise in the field of domestic database and big data. With the mission of "let the world use China's database", the company has created a variety of independently controllable databases and big data products such as GBase 8a/8t/8m/8s/8d/UP/InfiniData all-in-one machine, providing services to tens of thousands of users in the fields of finance, telecommunications, government affairs, national defense, enterprises and institutions, covering five continents including China, Britain, Colombia, the United States, Thailand and South Africa More than 30 countries and regions. NTU general GBASE brand has successfully entered the magic quadrant of Gartner analytical data management solutions. This Chat mainly introduces GBase8s.
OSCAR
Tianjin Shenzhou General Data Technology Co., Ltd. (hereinafter referred to as Shenzhou general) is committed to the industrialization of Shentong domestic database. It is subordinate to China Aerospace Science and Technology Corporation. It is one of the most influential basic software enterprises in China and has won the key support of major national nuclear high-tech projects. The main business of Shenzhou general company mainly includes the R & D and marketing of Shentong relational general database, Shentong KStore massive data management system, Shentong xCluster groupware, Shentong Business Intelligence Suite and other series of products. The company's customers mainly cover the fields of government, telecommunications, energy, transportation, network security, national defense and military industry, and takes the lead in realizing the large-scale commercial application of domestic databases in the telecommunications industry. The enterprise level and large-scale general relational database management system with enterprise level and security versions and independent intellectual property rights is the concentrated embodiment of Shenzhou general company's many years of R & D accumulation in the field of large-scale database and profound experience in aerospace information construction. It is also a major national "nuclear high base" project The national "863" plan strongly supports the development of basic software, which is an important achievement.
PolarDB
PolarDB is the next-generation relational cloud database developed by Alibaba cloud. It is compatible with MySQL, PostgreSQL and Oracle engines. The storage capacity can be up to 100TB. A single database can be expanded to 16 nodes at most. It is suitable for diversified database application scenarios in enterprises.
PolarDB adopts the architecture of separation of storage and computing. All computing nodes share one piece of data, providing minute level configuration, second level fault recovery, global data consistency and free data backup and disaster recovery services. PolarDB not only integrates the characteristics of stability, reliability, high performance and scalability of commercial databases, but also has the advantages of simple, open and self iteration of open source cloud databases. For example, PolarDB for MySQL can improve the performance up to 6 times that of MySQL, and the cost is only 1 / 10 of that of commercial databases. (from the official website)
TDSQL
Distributed database (TDSQL) is a distributed database product created by Tencent. It has the characteristics of strong consistency, high availability, global deployment architecture, distributed horizontal expansion, high performance and enterprise security. At the same time, it provides supporting facilities such as intelligent DBA, automatic operation, monitoring and alarm, and provides users with a complete distributed database solution.
GaussDB100
Huawei GaussDB is an enterprise AI native distributed database. GaussDB adopts MPP (Massive Parallel Processing) architecture, supports row storage and column storage, and provides PB (Petabyte, 50th power byte of 2) level data processing capacity. It can provide a cost-effective general computing platform for large-scale data management, and can also be used to support various data warehouse systems, BI (Business Intelligence) systems and decision support systems to provide services for decision analysis of upper level applications. There are GaussDB100, GaussDB200 and other series.
Use and adaptation of localization database
data type
In order to compare the differences, the common data types of Oracle and MySQL are listed at the same time.
MySQL common data types:
Common Oracle data types:
DM common data types:
GBase common data types:
OSCAR common data types:
GaussDB100 common data types:
Special grammar and precautions
Most domestic databases are compatible with Oracle or mysql. For example, DM and GaussDB100 are quite compatible with Oracle. The difference in SQL is not great. PolarDB and TDSQL are more compatible with MySQL, but some domestic databases may still have some differences in SQL. However, some problems encountered are listed below. Of course, in addition to these, there are others waiting for you to learn and explore.
GBase8S precautions
1. There is no double type, and both double types use float
Autoincrement uses the serial type.
2. Add primary key constraint
ALTER TABLE ASYNCTASK ADD CONSTRAINT PRIMARY KEY (id) CONSTRAINT PK_NR_ASYNCTASK alter table ma3j_mj_bblx drop constraint pk_ma3j_mj_bblx //delete
3. Unique constraint
ALTER TABLE DEV_BASECLASSTAB add constraint unique (CLASSNAME) constraint DEV_BASECLASSTAB_UK1;
4. Foreign key constraints
ALTER TABLE CMS_KEYWORK_EXTERNAL_TAB add constraint foreign key (KEYWORKID) references CMS_KEYWORK_TAB (KEYWORKID) on delete cascade constraint FK_KEYWORK_EX disabled;
5. Add or delete index
create index idx_test on jmtjzt_test (ordinal); DROP INDEX idx_test
6. Delete column
alter table tableName drop(column); other: alter table tableName drop column columnName
7. Modify table name
rename table tableName to newName other: alter table tableName rename to newName
8. Heavy life
rename column tableName.c1 to newName other:alter table Table name rename column Old column name to New column name;
9. null conversion
select context_,task_id from asynctask_history union all select null::blob,task_id from asynctask_history
If more than one union is queried, it needs to be converted to the corresponding type.
10. Length limit of index and primary key
The database pagesize limits the maximum length of index columns.
11. Other issues:
- Lock table
- Not enough space
- blob et al
OSCAR precautions
Agreement issues
The Shentong database provides some configuration parameters, which are listed below:
# batchbuffersie: the size of the sending buffer of the batch interface, in K,
#MAX: 1048576, MIN: 1, DEFAULT: 128
batchbuffersize=128
# loglevel: log level, which is divided into four levels: 0, 1, 2, 3 and 4. 0 means no log printing and 1 means SQL
#At the statement level, 2 represents the calling interface level, 3 represents the lightweight protocol level, and 4 represents the detailed protocol level.
loglevel=0
#logFilePath: log file path. When logFilePath is empty, JDBC log file and jdbc driver are in the same directory
logFilePath=c:/log
# maxFileSize:Maximum log file size in MB，When the maximum size is reached, the new log is used#file
#compatible_old_protocol, whether to use the old protocol to interact with the background
compatibleOldProtocol=false
There is one item to note: compatibleOldProtocol=false, whether to use the old protocol to interact with the background. When the database table field exceeds 256, if the new protocol does not support inserting data, you need to use the old protocol to interact with the Server. This configuration item is configured in Oscar config In the properties file, you can put the file anywhere, and specify CONFIGFILE in the url to load the parameters. Or put it in a unified directory with the driver package).
spring.datasource.url=jdbc:oscar://10.2.32.23:2003/OSRDB?CONFIGFILE=E:/config/oscarconfig.properties
GaussDB100 precautions
prepareStatement uses the rolling result set method to query:
After reading the driver source code of GaussDB100, we found that GaussDB100 only supports TYPE_FORWARD_ONLY type.
Program adaptation
In terms of program, we need to pay attention to the database driver and version. Different databases correspond to different drivers, and the version is also very important. We have encountered some strange problems, and finally found the problem of the driver version.
The other is to consider the differences and compatibility between databases. Especially when a program needs to adapt to multiple databases, it needs a complete and mature adaptation scheme.
Localization database performance
Database is the basic software, just like the engine of the automobile. It is the key component of the system and has the characteristic of "pulling one hair and moving the whole body". This requires that the database has a high maturity. This maturity needs to have a long accumulation in technology, products, engineering, support and industry experience.
China has only begun to develop databases in recent years, mainly drawing nutrients from open source products such as MySQL and Postgres; There are also acquisitions that start with the introduction of Informix; Innovation is never an instant thing. It's understandable to stand on the shoulders of giants. It just needs an accumulation process.
In terms of popularity, there is indeed a big gap; But technically, for example, the self-developed Damon database has far surpassed the famous open source databases such as PG and MySQL in many aspects such as standardization, function, performance and compatibility with Oracle, especially Oracle compatibility. After years of practical combat, it has been greatly improved, This is also the reason why domestic databases can win a certain market share in the traditional business field.
In terms of performance analysis, Oracle has reports such as awr to analyze database bottlenecks, while domestic databases rarely have such reports. However, cloud databases such as PolarDB and GaussDB100 generally have implementation analysis reports. Can you also easily provide me with database performance analysis. For example, GBase may require us to regularly perform database management tasks to monitor the database and coordinate DBA s to analyze problems.
Finally, China is a huge market soil. Now is the golden period for the development of China's database. The development of domestic database will be better and better in the future.
Adapt Activiti process engine
Extended description
- Write the extension configuration Dameng Properties (refer to the configuration under org/activiti/db/properties / in activiti-engine.jar for the content), and dameng Properties is placed in the config directory of src/main/resource (other databases also need corresponding syntax files).
- Write the SqlSessionFactory configuration class to reset the SqlSessionFactory of the engine.
- Write the SpringProcessEngineConfiguration configuration extension and load the SqlSessionFactory configuration class.
- At this point, the adaptation of domestic databases is realized, and only spring. Com needs to be configured datasource. Dbtype is enough.
\-----------------------------------------------------------------------------------------------------------------------------------
/**
* SqlSessionFactory to configure
*/
@Component
public class DbSqlSessionConfigurator implements ProcessEngineConfigurator {
//configuratorsAfterInit in ProcessEngineConfigurationImpl will execute here
@Override
public void configure(ProcessEngineConfigurationImpl processEngineConfiguration) {
//Create SqlSessionFactory and set ProcessEngineConfigurationImpl
processEngineConfiguration.setSqlSessionFactory(createSqlSessionFactory(processEngineConfiguration));
//Re execute initDbSqlSessionFactory
processEngineConfiguration.initDbSqlSessionFactory();
}
private SqlSessionFactory createSqlSessionFactory(ProcessEngineConfigurationImpl processEngineConfiguration) {
InputStream inputStream = null;
try {
inputStream = processEngineConfiguration.getMyBatisXmlConfigurationStream();
Environment environment = new Environment("default", processEngineConfiguration.getTransactionFactory(),
processEngineConfiguration.getDataSource());
Reader reader = new InputStreamReader(inputStream);
Properties properties = new Properties();
properties.put("prefix", processEngineConfiguration.getDatabaseTablePrefix());
String wildcardEscapeClause = "";
if ((processEngineConfiguration.getDatabaseWildcardEscapeCharacter() != null)
&& (processEngineConfiguration.getDatabaseWildcardEscapeCharacter().length() != 0)) {
wildcardEscapeClause = " escape '" + processEngineConfiguration.getDatabaseWildcardEscapeCharacter()
+ "'";
}
properties.put("wildcardEscapeClause", wildcardEscapeClause);
// set default properties
properties.put("limitBefore", "");
properties.put("limitAfter", "");
properties.put("limitBetween", "");
properties.put("limitOuterJoinBetween", "");
properties.put("limitBeforeNativeQuery", "");
properties.put("orderBy", "order by ${orderByColumns}");
properties.put("blobType", "BLOB");
properties.put("boolValue", "TRUE");
if (processEngineConfiguration.getDatabaseType() != null) {
//Set to read the syntax configuration file of Dameng database. The file path is Src / main / resource / config / Dameng properties
if("dameng".equalsIgnoreCase(processEngineConfiguration.getDatabaseType())) {
properties.load(getResourceAsStream("config/"+ processEngineConfiguration.getDatabaseType() + ".properties"));
}else {
properties.load(getResourceAsStream("org/activiti/db/properties/"
+ processEngineConfiguration.getDatabaseType() + ".properties"));
}
}
Configuration configuration = processEngineConfiguration.initMybatisConfiguration(environment, reader,
properties);
return new DefaultSqlSessionFactory(configuration);
} catch (Exception e) {
throw new ActivitiException("Error while building ibatis SqlSessionFactory: " + e.getMessage(), e);
} finally {
IoUtil.closeSilently(inputStream);
}
return null;
}
private InputStream getResourceAsStream(String resource) {
return ReflectUtil.getResourceAsStream(resource);
}
@Override
public int getPriority() {
return 0;
}
}
\-------------------------------------------------------------------------------------------------------------------------
//This configuration will be executed after the spring process engine configuration is created
@Configuration
public class ActivitiEngineConfiguration implements ProcessEngineConfigurationConfigurer {
//Set database type
@Value(value = "${spring.datasource.dbType}")
private String databaseType;
//Set DbSqlSessionConfigurator
@Autowired
private DbSqlSessionConfigurator dbSqlSessionConfigurator;
@Override
public void configure(SpringProcessEngineConfiguration processEngineConfiguration) {
processEngineConfiguration.setDatabaseType(databaseType);
List<ProcessEngineConfigurator> configurators = new ArrayList<ProcessEngineConfigurator>();
configurators.add(dbSqlSessionConfigurator);
processEngineConfiguration.setConfigurators(configurators);
}
}
\--------------------------------------------------------------------------------------------------------------------
Migration from Oracle and MySQL to domestic databases
For a large business system that has been running for some time, database migration plays a key role in the stable operation of later applications. When migrating data from two different types of databases, you need to consider the data type first, then the SQL syntax and the use of some functions, and then whether the program needs to be modified. Generally, domestic databases have corresponding migration tools.
One is to directly migrate from Oracle database to domestic database by using the built-in data migration tool in the database. This method is more suitable for internal migration or networking between different machines. Here are the migration of Oracle to dream. For other databases, please refer to the manufacturer's suggestions and tools. For example, GBase also has its own migration tools. You only need to configure the field mapping and the corresponding relationship to migrate.
For details of Dameng, please refer to the link step-by-step:
The second method is to use dmp files for migration when the destination database cannot be networked. Here is an example of the migration method of Dameng. Other database tools can use specific database commands to migrate.
The specific methods for importing and exporting dmp files are as follows.
The first method is to use the command line of dimp and dexp to import and export, and enter the installation directory bin of the dream database, such as / opt/DM8/DM8/bin. There are two files of dexp and dimp, which can be executed. Refer to the command:
./dexp USERID=SYSDBA/SYSDBA FILE=db_str.dmp LOG=db_str.log TABLES=table1,table2
DIRECTORY=/mnt/data/dexp
./dimp USERID=SYSDBA/SYSDBA FILE=/mnt/data/dexp/db_str.dmp LOG=db_str.log FULL=Y
DIRECTORY=/mnt/data/dimp
The second is to enter the desktop/client directory of the installation directory and open the DM management tool. There are import and export options in the mode, which is also very convenient to use.