stay Apache ShardingSphere 5.1.1 After the release of ,ShardingSphere Consolidate the cumulative experience of teams or individuals from around the world 1028 individual PR, To bring you 5.1.2 The new version . This version features 、 performance 、 test 、 file 、 Examples and other aspects have many optimizations .

It is worth mentioning that , This update brings the following new features ：

  • ShardingSphere-Proxy Helm Chart

  • SQL Dialect translation

  • With Driver Form use ShardingSphere-JDBC

These new capabilities make ShardingSphere The data gateway capability of has been qualitatively improved , help ShardingSphere Deploy... On the cloud , Optimize the user experience .

In addition to the new features mentioned above , This update has greatly improved SQL Resolution support , kernel 、 Operation mode 、 Elastic expansion and contraction are perfect for PostgreSQL / openGauss schema Support for , Stretch and stretch 、 Business 、DistSQL There are also many improvements in robustness and user experience .

This article will introduce you to ShardingSphere 5.1.2 Version update .

Introduction to new functions

  • Use Helm Deploy ShardingSphere-Proxy

ShardingSphere-Proxy Provides Docker Image to facilitate user containerized deployment . however , For the need to be in Kubernetes Deploy ShardingSphere-Proxy Users of , You also need to handle the database driver mounting by yourself 、 Configure mount 、 User defined algorithm mounting and other necessary steps , The deployment process is relatively tedious , The cost of operation and maintenance is relatively high .

ShardingSphere This update brings a new ShardingSphere-Proxy Helm Chart. This new feature is provided by the enterprise 、 Cloud native data enhanced computing products and solutions providers SphereEx towards Apache ShardingSphere Community donations , Push Apache ShardingSphere Moving in the direction of cloud origin .

ShardingSphere In the cluster mode, metadata is stored in the registry ,ShardingSphere-Proxy Of Helm Chart Be able to deploy automatically ZooKeeper colony , Help users quickly build ShardingSphere-Proxy colony .

Limited by open source protocols ,ShardingSphere-Proxy Binary distribution package for 、Docker Mirroring is restricted by the open source protocol , Can't pack MySQL JDBC drive , Users need to manually add MySQL JDBC Drive to classpath Can be used MySQL As ShardingSphere Storage nodes of . For such cases ,ShardingSphere-Proxy Helm Chart In the Pod Of Init The container automatically gets MySQL JDBC drive , The deployment operation cost of users is reduced .

  • new SQL Dialect translation ability

With the irreversible trend of database fragmentation , The coexistence of multiple types of databases has gradually become the norm . Use a SQL The scenarios of dialect accessing heterogeneous databases are increasing .

The existence of diverse databases , To access a database SQL Dialects are difficult to standardize , Engineers need to use different dialects for different kinds of databases , Lack of a unified query platform .

Automatically translate different types of database dialects into dialects used by back-end databases , So that engineers can use any database dialect to access all back-end heterogeneous databases , Can greatly reduce development and maintenance costs .

Apache ShardingSphere 5.1.2 An important step has been taken on the road to building a highly productive data gateway . This update brings a new SQL Dialect translation ability , It can support dialect conversion between mainstream open source databases . for example , Users can use MySQL Client connection ShardingSphere-Proxy And send based on MySQL Dialectal SQL,ShardingSphere It can automatically identify user protocols and storage node types , Done automatically SQL Dialect conversion , visit PostgreSQL Heterogeneous storage nodes , vice versa .

  • With Driver In the form of ShardingSphere-JDBC

In the past ShardingSphere In the version ,ShardingSphere-JDBC With DataSource Provide services to users in the form of . For not using DataSource Project or tool , Modification is required to introduce ShardingSphere-JDBC, Increase the user's use cost .

stay Apache ShardingSphere 5.1.2 in ,ShardingSphere-JDBC Implemented standard JDBC Driver Interface , The user can go through Driver The form of introduction ShardingSphere-JDBC.

Users can go directly through DriverManager obtain Connection：

Class.forName("org.apache.shardingsphere.driver.ShardingSphereDriver");

Connection conn = DriverManager.getConnection("jdbc:shardingsphere:classpath:config.yaml");

You can also use DataSource obtain Connection：

//  With  HikariCP  For example 

HikariDataSource dataSource = new HikariDataSource();

dataSource.setDriverClassName("org.apache.shardingsphere.driver.ShardingSphereDriver");

dataSource.setJdbcUrl("jdbc:shardingsphere:classpath:config.yaml");

Connection conn = dataSource.getConnection();

The improvement of existing capabilities

kernel

This update ,ShardingSphere A large number of improvements have been incorporated SQL Resolve the support level PR, Visible in the update log SQL The resolved optimization items occupy a large space .

ShardingSphere stay 5.1.1 Yes PostgreSQL / openGauss Of schema Provided initial support , This time 5.1.2 to update , kernel 、 Cluster pattern 、 Elastic telescopic pair PostgreSQL / openGauss Of schema Their support has also increased , For example, the metadata pair adds a pair of schema Structure support , kernel 、 Elastic scaling supports customization schema.

ShardingSphere-Proxy

With the use of ARM framework CPU The server market is becoming more and more popular ,ShardingSphere-Proxy stay Docker Aspect also provides for arm64 The mirror image of Architecture .

stay MySQL aspect ,ShardingSphere-Proxy Fixed inability to receive longer than 8 MB Packet problems , And it supports the total receiving length exceeding 16 MB The data of .

Stretch and stretch

Elastic scaling in addition to support PostgreSQL Customize schema Outside , Migration is also implemented PostgreSQL The function of automatically creating tables when , And repaired it. PostgreSQL Incremental migration encountered null The field value will report an error . In addition to the function , Elastic scaling also reduces resource consumption during migration , And right openGauss 3.0 Incremental migration provides support .

Welcome to use or upgrade ShardingSphere 5.1.2,ShardingSphere The community is looking forward to your feedback ！

Update log

The following is a ShardingSphere 5.1.2 All update logs for . It should be noted that , This update adjusts a small number of API, For adjustment items, please refer to the API Adjustment part .

New characteristics

  • kernel ： Support MySQL and PostgreSQL Of SQL Dialect conversion alpha edition

  • kernel ： Support PostgreSQL and openGauss Customize schema

  • kernel ： Support PostgreSQL and openGauss create/alter/drop view sentence

  • kernel ： Support openGauss cursor sentence

  • kernel ： Support the use of custom system libraries

  • kernel ： Support acquisition openGauss and MySQL Create table statement

  • kernel ： Support acquisition PostgreSQL Create table statement

  • Access end ： Officially support the use of Helm stay Kubernetes To quickly deploy a containing ZooKeeper Clustered ShardingSphere-Proxy colony

  • Access end ： Support ShardingSphere JDBC Driver

  • Stretch and stretch ： Support PostgreSQL Build a watch automatically

  • Stretch and stretch ： Support PostgreSQL and openGauss Customize schema Table migration for

  • Stretch and stretch ： Support string primary key table migration

  • Operation mode ： The governance center supports PG/openGauss Three level structure

  • Operation mode ： The governance center supports Database Level distributed lock

Optimize

  • kernel ： Support PostgreSQL and openGauss copy sentence

  • kernel ： Support PostgreSQL Of alter/ drop index sentence

  • kernel ： Support MySQL update force index sentence

  • kernel ： Support openGauss create/alter/drop schema sentence

  • kernel ： Optimize RoundRobinReplicaLoadBalanceAlgorithm and RoundRobinTrafficLoadBalanceAlgorithm Algorithm logic

  • kernel ： Optimize metadata loading logic when the front-end drive database type is inconsistent with the back-end

  • kernel ： Refactoring metadata loading logic

  • kernel ：show processlist Statement function performance optimization

  • kernel ： Improve loading performance in scenarios with a large number of tables

  • kernel ： Support comment Execution of statements

  • kernel ： Support PostgreSQL and openGauss In the scene of fragmentation view Execution of statements

  • kernel ： Support ORACLE Of CREATE ROLLBACK SEGMENT sentence

  • kernel ： Support parsing openGauss DROP TYPE

  • kernel ： Support parsing openGauss ALTER TYPE

  • kernel ： Support parsing Oracle DROP DISKGROUP

  • kernel ： Support parsing Oracle CREATE DISKGROUP

  • kernel ： Support parsing Oracle DROP FLASHBACK ARCHIVE

  • kernel ： Support parsing openGauss CHECKPOINT

  • kernel ： Support parsing Oracle CREATE FLASHBACK ARCHIVE

  • kernel ： Support parsing PostgreSQL Close

  • kernel ： Support parsing openGauss DROP CAST

  • kernel ： Support parsing openGauss CREATE CAST

  • kernel ： Support parsing Oracle CREATE CONTROL FILE

  • kernel ： Support parsing openGauss DROP DIRECTORY

  • kernel ： Support parsing openGauss ALTER DIRECTORY

  • kernel ： Support parsing openGauss CREATE DIRECTORY

  • kernel ： Support parsing PostgreSQL Checkpoint

  • kernel ： Support parsing openGauss DROP SYNONYM

  • kernel ： Support parsing openGauss CREATE SYNONYM

  • kernel ： Support parsing openGauss ALTER SYNONYM

  • kernel ： Support parsing PostgreSQL CALL Statement

  • kernel ： Support parsing Oracle CREATE PFILE

  • kernel ： Support parsing Oracle CREATE SPFILE

  • kernel ： Support parsing Oracle ALTER SEQUENCE

  • kernel ： Support parsing Oracle CREATE CONTEXT

  • kernel ： Support parsing Oracle ALTER PACKAGE

  • kernel ： Support parsing Oracle CREATE SEQUENCE

  • kernel ： Support parsing Oracle ALTER ATTRIBUTE DIMENSION

  • kernel ： Support parsing Oracle ALTER ANALYTIC VIEW

  • kernel ： Use ShardingSphere Spi load SQLVisitorFacade

  • kernel ： Use ShardingSphere Spi load DatabaseTypedSQLParserFacade

  • kernel ： Support parsing Oracle ALTER OUTLINE

  • kernel ： Support parsing Oracle DROP OUTLINE

  • kernel ： Support parsing Oracle drop edition

  • kernel ： Support parsing SQLServer WITH Common Table Expression

  • kernel ： Optimize SubquerySegment stay with Start and end indexes in statements

  • kernel ： restructure JoinTableSegment

  • kernel ： Support parsing Oracle DROP SYNONYM

  • kernel ： Support parsing Oracle CREATE DIRECTORY

  • kernel ： Support parsing Oracle CREATE SYNONYM

  • kernel ： Support parsing SQLServer XmlNamespaces Clause

  • kernel ： Support parsing Oracle Alter Database Dictionary

  • kernel ： Support parsing SQLServer Clause of SELECT Statement

  • kernel ： Support parsing Oracle ALTER DATABASE LINK

  • kernel ： Support parsing Oracle CREATE EDITION

  • kernel ： Support parsing Oracle ALTER TRIGGER

  • kernel ： Support parsing SQLServer REVERT Statement

  • kernel ： Support parsing PostgreSQL DROP TEXT SEARCH

  • kernel ： Support parsing PostgreSQL drop server

  • kernel ： Support parsing Oracle ALTER VIEW

  • kernel ： Support parsing PostgreSQL drop access method

  • kernel ： Support parsing PostgreSQL DROP ROUTINE

  • kernel ： Support parsing SQLServer DROP USER

  • kernel ： Support parsing Oracle DROP TRIGGER

  • kernel ： Support parsing PostgreSQL Drop subscription

  • kernel ： Support parsing PostgreSQL drop operator class

  • kernel ： Support parsing PostgreSQL DROP PUBLICATION

  • kernel ： Support parsing Oracle DROP VIEW

  • kernel ： Support parsing PostgreSQL DROP TRIGGER

  • kernel ： Support parsing Oracle DROP DIRECTORY

  • kernel ： Support parsing PostgreSQL DROP STATISTICS

  • kernel ： Support parsing PostgreSQL drop type

  • kernel ： Support parsing PostgreSQL DROP RULE

  • kernel ： Support parsing SQLServer ALTER LOGIN

  • kernel ： Support parsing PostgreSQL DROP FOREIGN DATA WRAPPER

  • kernel ： Support parsing PostgreSQL DROP EVENT TRIGGER statement

  • Access end ：ShardingSphere-Proxy MySQL The receiving length exceeds 16 MB Request packet for

  • Access end ：ShardingSphere-Proxy increase SO_BACKLOG Configuration item

  • Access end ：ShardingSphere-Proxy Enabled by default SO_REUSEADDR

  • Access end ：ShardingSphere-Proxy Docker image increase aarch64 Support

  • Access end ：ShardingSphere-Proxy MySQL Support to configure default MySQL Version number

  • Access end ：ShardingSphere-Proxy PostgreSQL / openGauss Support more character sets

  • Access end ：ShardingSphere-Proxy Add the default port configuration item

  • Stretch and stretch ：openGauss 3.0 Enable thread_pool when ,Scaling compatible HA port Data synchronization

  • Stretch and stretch ： Optimize PipelineJobExecutor in Zookeeper The logic of event handling , avoid zk Blocking events

  • Stretch and stretch ：Scaling Data synchronization is not case sensitive

  • Stretch and stretch ： improvement PostgreSQL/openGauss Copy slot cleanup

  • Stretch and stretch ： Improved lock protection during preparation

  • Stretch and stretch ： improvement PostgreSQL After the same record is deleted, the data synchronization under the reconstruction scenario

  • Stretch and stretch ：Scaling The created data source is not cached at the bottom

  • Stretch and stretch ： Try to reuse data sources , Reduce the occupation of database connection

  • DistSQL：REFRESH TABLE METADATA Support specifying PostgreSQL's schema

  • DistSQL：ALTER SHARDING TABLE RULE Add the verification of the binding table

  • Operation mode ：ShardingSphere-JDBC Support to configure database connection name

  • Distributed transactions ： Execution of... Is prohibited in transactions DistSQL

  • Distributed transactions ：autocommit = 0, DDL part DML The transaction will be opened automatically

Problem fix

  • kernel ： Repair PostgreSQL and openGauss show Statement parsing exception
  • kernel ： Repair PostgreSQL and openGauss time extract function Parsing exceptions
  • kernel ： Repair PostgreSQL and openGauss select mod fucntion Parsing exceptions
  • kernel ： Repair multiple in the read / write separation scenario schema join Statement execution exception
  • kernel ： Repair encryption scenario create schema Statement routing exception
  • kernel ： Repair drop schema if exist Statement exception
  • kernel ： Repair execution LAST_INSERT_ID() Routing error
  • kernel ： Repair use database The problem of executing exceptions without data source
  • kernel ： Repair with set var Of fucntion Create statement
  • Access end ： Repair ShardingSphere-Proxy PostgreSQL / openGauss Describe PreparedStatement Null pointer due to field case mismatch
  • Access end ： Repair ShardingSphere-Proxy PostgreSQL / openGauss perform schema DDL It does not return correctly after tag The problem of
  • Stretch and stretch ： Repair MySQL unsigned Type in the Scaling Something went wrong in the process
  • Stretch and stretch ： Fix the problem of connection leakage when the consistency check failed to create a data source
  • Stretch and stretch ： Repair ShardingSphereDataSource Initialize the problem of ignoring rules other than shards
  • Stretch and stretch ： Support job Closed during the preparation phase
  • Stretch and stretch ： Fix the data source url and jdbcurl Compatibility issues
  • Stretch and stretch ： Repair openGauss Data replication slot creation timing problem , Avoid possible incremental data loss
  • Stretch and stretch ： improvement job State persistence , Ensure that the old state is not overwritten under special circumstances
  • Stretch and stretch ： Repair PostgreSQL Use TestDecoder Cannot resolve correctly when performing incremental migration null
  • DistSQL： Repair single machine and memory mode ,SET VARIABLE The modification does not take effect
  • DistSQL： Repair SHOW INSTANCE LIST Inconsistency with actual data
  • DistSQL： Fix the case sensitive problem of sharding rules
  • Operation mode ： Repair Scaling The metadata of the new version is lost after the function changes the table splitting rules
  • Distributed transactions ： Repair according to catalog obtain indexinfo Empty question

restructure

  • Stretch and stretch ： restructure jobConfig, Convenient for new types job Reuse and extension
  • Operation mode ： Optimize the storage structure of the registry computing nodes
  • Operation mode ： Use uuid replace [email protected] As the unique identifier of the instance

API adjustment

  • DistSQL：EXPORT SCHEMA CONFIG Adjusted for EXPORT DATABASE CONFIG
  • DistSQL：IMPORT SCHEMA CONFIG Adjusted for IMPORT DATABASE CONFIG
  • Operation mode ： adjustment db-discovery Algorithm configuration
  • DistSQL：SHOW SCHEMA RESOURCES Adjusted for SHOW DATABASE RESOURCES
  • DistSQL：COUNT SCHEMA RULES Adjusted for COUNT DATABASE RULES
  • jurisdiction ： Authority provider ALL_PRIVILEGES_PERMITTED Updated to ALL_PERMITTED
  • jurisdiction ： Authority provider SCHEMA_PRIVILEGES_PERMITTED Updated to DATABASE_PERMITTED

Related links

Download link ：

https://github.com/apache/shardingsphere/blob/master/RELEASE-NOTES.md

Update log ：

https://github.com/apache/shardingsphere/blob/master/RELEASE-NOTES.md

Project address ：

https://shardingsphere.apache.org/

Community building

this Apache ShardingSphere 5.1.2 Release of version , share 54 position Contributor submitted 1028 individual PR, Thank you for your support .

author

Wu Weijie ,SphereEx Infrastructure R & D Engineer ,Apache ShardingSphere PMC. Focus on Apache ShardingSphere Access terminal and ShardingSphere subprojects ElasticJob The development of .

Apache ShardingSphere 5.1.2 Release ｜ New drive API + Cloud native deployment , More articles on building high-performance data gateways

  1. Blockbuster released | The world's first cloud native application standard definition and architecture model OAM Official open source

    author : OAM Project leader Reading guide :2019 year 10 month 17 Japan , Alibaba partners . Jiang Jiangwei, general manager of Alibaba cloud intelligent basic products business department ( flower : Little evil ) stay Qcon Shanghai heavyweight announced , Alibaba cloud and Microsoft jointly launched the open application model Op ...

  2. Apache ShardingSphere 5.0.0 Kernel optimization and upgrade Guide

    After nearly two years of optimization and polishing ,Apache ShardingSphere 5.0.0 GA The version was finally officially released this month , Compared with 4.1.1 GA edition ,5.0.0 GA The version has a lot of optimization at the kernel level . First , be based on ...

  3. Major upgrade ！ Lingque cloud releases the original open platform of the whole stack cloud ACP 3.0

    The development of cloud native technology is changing the pattern of the global software industry , With the gradual improvement of cloud native technology ecosystem , The cloud native platform of lingque cloud has also entered a mature stage . In recent days, , Lingque cloud releases major product upgrades , Launch the full stack cloud native open platform ACP 3.0. As a for enterprise users ...

  4. Apache ShardingSphere： Distributed database middleware driven by open source

    2021 year 7 month 21 Japan 2021 Amazon cloud technology China Summit site ,SphereEx cofounder .Apache ShardingSphere PMC Pan Juan was invited to participate in the summit , With <Apache Sh ...

  5. DistSQL： Use like a database Apache ShardingSphere

    Apache ShardingSphere 5.0.0-beta The first article of in-depth analysis revisited with you ShardingSphere The principle of the kernel , The kernel level of this version is described in detail , especially SQL In terms of ability ...

  6. SQL I can still be in Apache ShardingSphere Realize these functions on ？

    In the last year 10 month 5.0.0-alpha After the release ,Apache ShardingSphere Experienced for a long time 8 More than months of continuous development and optimization , Finally in the 6 month 25 The first day of the year officially ushered in 5.0.0-beta edition ...

  7. SphereEx Establishment of the company , Push Apache ShardingSphere Community development is accelerating

    In recent days, ,SphereEx With the help of China Sequoia seed fund and Chuxin capital , The company and team have been established . Various major media platforms and official account have successively reported , And grab the headlines . Take as an example Apache ShardingSphere Core team ...

  8. Apache Storm 1.1.0 Release Overview

    Let me write it out front I have long been concerned about data mining and machine learning related frontier research . Welcome to talk to me , Private wechat :846731084 I tested this version myself , On the whole, it's more stable , The new features were not tested one by one , On the basis of kafk-client Come on ...

  9. 【 Re posting 】IBM Launch a new flash solution , Help enterprise storage architecture do subtraction

    IBM Launch a new flash solution , Help enterprise storage architecture do subtraction http://stor-age.zhiding.cn/stor-age/2020/0227/3124851.shtml Feeling There is a mistake in the drawing FlashS ...

  10. Apache ShardingSphere Landing trip in Jingdong Baitiao scene

    Jingdong Baitiao uses Apache ShardingSphere Solved the problem of 100 billion data storage and capacity expansion , It has laid the foundation for the promotion activities . 2014 Beginning of the year ," Jingdong ious " As an Internet credit payment product in the industry , Data explosion ...

Random recommendation

  1. I am right. git know

        Git --- The stupid content tracker, Stupid content tracker .Linus Torvalds This is how to introduce Git Of . Git Is used for Linux Version control of kernel development ...

  2. vmware install cent os 6.5 + oracle 11g xe + jboss eap 6.2 + weblogic 12c+ webshpere mq 7.5

    Preface : mac The development speed of the system is really fast , Just a few years ,mac os Online banking has been available on ( Bank of China . CMB . Business circles . Alipay and others have fully supported mac os 了 ),windows Classic and commonly used software on :qq. Fetion . Wang Wang . Youdao Dictionary . Youdao cloud pen ...

  3. LeetCode Alien Dictionary

    The link to the original question is here :https://leetcode.com/problems/alien-dictionary/ subject : There is a new alien language which uses ...

  4. html5 meta Head set

    <meta name="viewport" content="height=[pixel_value | device-height], width=[pixel_ ...

  5. ng-options

    tr td place td select.form-control(required="true" ng-model="addkc.dd" ng-options=& ...

  6. zabbix Deployment practice 【5 Use email SMTP SSL Push alarm email 】

    Because the traditional email push script uses smtp 25 port , It is no longer applicable to major email providers , Already directed SSL transition , Here we use QQ Mailbox as an example , Use SSL 465 port Sign in zabbix-server Get into cd /usr/lib/za ...

  7. HDU1024 DP The optimization of the Maximum M Subparagraphs and questions

    Max Sum Plus Plus Time Limit: 2000/1000 MS (Java/Others)    Memory Limit: 65536/32768 K (Java/Others ...

  8. Replace with a prototype PRD when , What should the prototype contain

    As the pace of the Internet gets faster and faster , The traditional requirements document has been difficult to adapt to the market , Especially for agile teams , Redundant and detailed requirements documents have become a burden ( Even leaders can't read such a long document ). At present, most teams prefer to use prototypes directly instead of needing to ...

  9. .NET Control .

    In the title of unit 4 , It involves the use of many controls that have not been used before , front 12 None of the questions is very difficult , So I finished it quickly , But the latter is because timer Control not found , Some operations cannot be implemented , So I didn't do , But I also read it carefully . When will you put ti ...

  10. Hadoop colony datanode Solutions to disk imbalance 【 turn 】

    One . introduction : Hadoop Of HDFS Cluster is very prone to imbalance of disk utilization between machines , For example, add a new data node to the cluster , The disk size is different between nodes, and so on . When hdfs When there is an imbalance , Will cause a lot of problems , Than ...