Good wind relies on strength – code conversion practice of accelerating SQL Server Migration with babelfish

Preface

Although there is no doubt , The traditional commercial license database has rich functions and perfect support , But its strict pricing model 、 Cumbersome license terms and high total cost of ownership （TCO） This makes enterprises want to adopt open source solutions with lower cost . In some ways , Open source databases provide the same or better functionality at a lower cost . Moving from a commercial database to an open source database can save enterprises a lot of costs in licensing and support .

PostgreSQL It's enterprise class , Open source database system with rich functions , It is highly reliable and has excellent performance , Ideal for real-time and mission critical applications .Amazon Aurora Is a relational database service , High speed and availability of existing high-end commercial databases , There is also the simplicity and cost-effectiveness of open source databases .Aurora And MySQL and PostgreSQL Fully compatible with , Enable existing applications and tools to run without modification . With the typical PostgreSQL Database comparison , It triples performance , And increased scalability 、 Persistence and security .

From traditional SQL Server Database migration can be time-consuming and resource intensive , Any migration involves three main steps ： Mobile architecture 、 Migrate data and modify client applications . As we can see in the figure below ： When migrating databases , You can use Amazon Schema Conversion Tool（SCT） coordination Amazon Database Migration Service (DMS) Automatically migrate database schema and data , But when migrating the application itself , Usually more work needs to be done , Including rewriting the application code that interacts with the database , take T-SQL Code migration to PL/pgSQL in , This is complicated 、 Time consuming and risky .

Babelfish for Aurora PostgreSQL  yes Amazon Aurora PostgreSQL A new feature in the compatible version , You can understand Microsoft SQL Server Proprietary SQL Language T-SQL, And support the same communication protocol , therefore , modify SQL Server And move it to Aurora The amount of work required will be reduced , Thus, faster 、 Lower risk and more cost-effective migration .

Babelfish By supporting Aurora PostgreSQL Of Microsoft SQL Server data type 、 Syntax and functions to support T-SQL and SQL Server Behavior . But please pay attention to ,Babelfish No right T-SQL Of 100% Full support , There are still some Differences and limitations , In some cases, manual code conversion is required .

This article will enumerate and demonstrate some high frequency and common typical code conversion cases , Help you complete the migration work more efficiently and quickly .

Learn more about Babelfish for Aurora PostgreSQL Information :

https://aws.amazon.com/cn/rds/aurora/babelfish/

Learn more about Babelfish Of Aurora PostgreSQL And SQL Server The difference between :

https://docs.amazonaws.cn/AmazonRDS/latest/AuroraUserGuide/babelfish-compatibility.html

Environmental preparation

Before we start our presentation , Suppose in your work environment , There is already a ready to migrate SQL Server Source library , Then besides , You also need to set up the following related components ：

• Babelfish Compass

  This is an open source SQL Server Migrate to Babelfish Grammar evaluation tool for , Can be in  GitHub  Upload and download . It can be in Windows and Linux Running under the platform , need Java Environmental support , The current version is v2022-04.

• Babelfish for Aurora PostgreSQL

  Babelfish from 2021 The first version was released in autumn 1.0.0 Start , The current version has been updated to 1.2.1, Corresponding Aurora PostgreSQL The version is 13.6. You can Official documents Instructions to operate , You can create one in a few simple steps Babelfish for Aurora PostgreSQL Cluster environment . During the configuration process, you should pay attention to the selection of database migration mode , Also, if there are Chinese data, please select... In the sorting rules “chinese_prc_ci_as”.

up to now , A contain SQL Server The source and Aurora PostgreSQL The target and the environment for the migration assessment tool are ready . Next , Please refer to the content of this blog , It only takes you a few minutes to Generate a Babelfish Migration assessment report .

Learn more about  GitHub Information :

https://github.com/babelfish-for-postgresql/babelfish_compass  

Learn more about   Official documents Information :

 https://docs.amazonaws.cn/AmazonRDS/latest/AuroraUserGuide/babelfish-create.html 

Generate a Babelfish For the migration evaluation report, please refer to :

https://aws.amazon.com/cn/blogs/database/migrate-from-sql-server-to-amazon-aurora-using-babelfish/ 

Code conversion

1

  Conversion evaluation

Babelfish Compass The evaluation report generated by the tool is a guide to evaluate the migration work content and workload , You can modify the items listed here , Write one by one SQL Transcoding content .

Evaluation report Summary The section lists migration SQL Server Source to Babelfish Target T-SQL The syntax features of are compatible with statistics , Including support 、 I won't support it 、 Semantic review 、 Manual review and negligible items . The most important one is the content that does not support features , These contain unsupported features SQL sentence , If not modified , stay Babelfish for Aurora PostgreSQL Most implementations in the environment will report errors ：“‘???’ is not currently supported in Babelfish”, And others SQL The statement does not report an error , But it will not really take effect .

In the evaluation report, we can view these unsupported features SQL Classified statistics , The evaluation report in the figure below lists the features that are not supported in each category SQL sentence , It shows the DDL Script in Babelfish The main features not supported in are adding constraint statements to tables ,Merge sentence 、 modify the database 、 Modify role 、 Execute some system stored procedures, etc .

2

Principle of conversion

Babelfish by Aurora PostgreSQL The database cluster provides an additional endpoint , Enable them to understand SQL Server Line level protocols and common SQL Server sentence . After the move , You can still use the same T-SQL Development tools and drivers , Connect to TDS The port completes the related development .

You can also use native PostgreSQL Connected to the PostgreSQL This end is for development , Again from T-SQL This end is called . This is a compatibility mode , Can help us solve most of the problems Babelfish Yes T-SQL Compatibility problem .

• Choose the conversion mode ： As mentioned above , For some unsupported SQL sentence , We can choose in T-SQL To rewrite , It can also be in PostgreSQL And then from T-SQL Call in . The principle of transformation depends on the connection development mode of the application , for example .net Applications connect to TDS End development , Then the preferred conversion mode is in T-SQL Conversion in . If in T-SQL This end cannot be overwritten or has modified performance problems , Then try in PostgreSQL Modify in process .

• Code readability ： For the to be modified SQL sentence , There may be several ways to rewrite . Simple 、 Efficient 、 Good readability is always the first choice . for example , In most cases , Use Case Statements are better than using ..Then Easier to understand .

3

Simple transcoding

Code transformations that fall into this category , It is characterized by simple modification , However, the quantity is often listed in the evaluation report for all unsupported features SQL Most of the statements . Generally speaking, such transcoding work only needs to mask the relevant options 、 Comment the whole statement or simply modify it . The reason for this change , It's because of PostgreSQL and SQL Server The characteristic difference between the two Babelfish The limitation of .SQL Server Some options or actions in , stay Babelfish Not supported and will not affect the function execution , Can be ignored directly . Although this kind of SQL Statement rewriting is simple , But it can achieve the same effect .

Before the demonstration , Let's take a look at the structure of the two tables that will be used next ：

create table dept(
    deptno int NOT NULL PRIMARY KEY,
    dname varchar(14),
    loc varchar(13)
) 

create table employees (
    empno int NOT NULL PRIMARY KEY,
    ename varchar(10),  
    job varchar(9),  
    mgr int,  
    hiredate datetime,  
    sal money,  
    comm money,  
    deptno int
)

* Slide left to see more

• ALTER TABLE..CHECK CONSTRAINT

The original statement ：

ALTER TABLE [dbo].[employees]  WITH CHECK ADD CONSTRAINT [FK_DEPT] FOREIGN KEY([deptno])
REFERENCES [dbo].[dept] ([deptno])
GO
ALTER TABLE [dbo].[Employees] CHECK CONSTRAINT [FK_DEPT]
GO

* Slide left to see more

Modified statement ：

ALTER TABLE [dbo].[employees] ADD CONSTRAINT [FK_DEPT] FOREIGN KEY([deptno])
REFERENCES [dbo].[dept] ([deptno])
GO

* Slide left to see more

explain ：

1. stay Babelfish China does not support it. CHECK CONSTRAINT Statement to enable table constraints ,ALTER After adding constraints to the table, constraints are automatically enabled .

2. stay Babelfish Is not supported when adding constraints to WITH CHECK/NOHECK Option to check constraints on existing data .

3. This unsupported ALTER TABLE Feature statements are the most common in the migration process , It is usually modified in Babelfish Create new tables and constraints on , Then import the data of the table , The constraint of the table will automatically check the imported data , Ensure that data constraints are valid .

• ALTER ROLE..

The original statement

ALTER ROLE [???] ADD MEMBER [NT AUTHORITY\SYSTEM]
GO

* Slide left to see more

Modified statement

/* ALTER ROLE [???] ADD MEMBER [NT AUTHORITY\SYSTEM]
GO */

* Slide left to see more

explain ：

1. at present Babelfish Only... In the user database is supported dbo user , You cannot create a user with lower privileges , For example, read-only permission to some tables .

2. Most of these statements are logged in by the user with the permission of the operating system SQL Server Poured out after the source DDL sentence , Mask statements can be annotated directly .

• ALTER DATABASE..

The original statement ：

ALTER DATABASE [???] SET RECOVERY FULL 
GO

* Slide left to see more

Modified statement ：

ALTER DATABASE [???] SET RECOVERY FULL 
GO

* Slide left to see more

explain ：

Babelfish I won't support it ALTER DATABASE grammar ,Aurora PostgreSQL Is a fully managed database , It will restrict some database modification statements , These statements can be directly annotated to mask

• ALTER AUTHORIZATION ON object

The original statement ：

ALTER AUTHORIZATION ON [dbo].[employees] TO  SCHEMA OWNER 
GO

* Slide left to see more

Change the following sentence ：

/* ALTER AUTHORIZATION ON [dbo].[employees] TO  SCHEMA OWNER 
GO */

* Slide left to see more

explain ：

Babelfish I won't support it AUTHORIZATION The creation of 、 Modification and deletion , You can directly annotate the mask .

• EXEC sys.sp_addextendedproperty

The original statement ：

EXEC sys.sp_addextendedproperty @name=N'MS_Description', @value=N' Number ' , @level0type=N'SCHEMA',@level0name=N'dbo', @level1type=N'TABLE',@level1name=N'dept', @level2type=N'COLUMN',@level2name=N'deptno'
GO

* Slide left to see more

Change the following sentence （ stay PostgreSQL End modification ）：

COMMENT ON COLUMN dept.deptno IS ' Number ';

explain ：

Babelfish The use of system stored procedures... Is not supported sp_addextendedproperty Add a description for the field , This can be masked by direct annotation SQL sentence , And connect to PostgreSQL End use comment Add field description .

• OBJECTPROPERTY

The original statement

select name from sysobjects where objectproperty(id, N'IsTable') = 1 and name not like N'#%%' order by name
select * from sysobjects where id = object_id(N'temp_tableSpaceInfo') AND objectproperty(id, N'IsUserTable') = 1

* Slide left to see more

Change the following sentence ：

select name from sysobjects where xtype in ('U','IT','S') and name not like N'#%%' order by name
select * from sysobjects where id = object_id(N'temp_tableSpaceInfo') AND xtype='U'

* Slide left to see more

explain ：

Babelfish Built in metadata functions are not supported OBJECTPROPERTYEX, According to the SQL The semantics should be rewritten properly

• SET ROWCOUNT

The original statement ：

CREATE PROCEDURE [dbo].[P_Rowcount]
@id int
as
set nocount on
set rowcount @id 
begin
select * from employees order by empno
end
GO

* Slide left to see more

Change the following sentence ：

CREATE PROCEDURE [dbo].[P_Rowcount]
@id int
as
set nocount on
begin
select top (@id) * from employees order by empno
end
GO

* Slide left to see more

After the second modification ：

CREATE PROCEDURE [dbo].[P_Rowcount]
@id int
as
set nocount on
begin
select * from employees order by empno offset 0 rows fetch first @id rows only;
end
GO

* Slide left to see more

explain ：

Babelfish I won't support it SET ROWCOUNT Statement to return the specified number of rows , According to the SQL The semantics should be rewritten properly . From the example, we can see that there are many rewriting methods , In a complex business scenario, you should choose between code readability and performance impact .

• CURRENT OF

The original statement ：

CREATE PROCEDURE [dbo].[P_CurrentOf] AS
BEGIN
  DECLARE @empno int
  DECLARE NoResponce CURSOR FOR
    SELECT empno FROM employees;
  OPEN NoResponce;
  FETCH NEXT FROM NoResponce INTO @empno;
  DELETE FROM employees WHERE CURRENT OF NoResponce;
END
GO

* Slide left to see more

Change the following sentence ：

CREATE PROCEDURE [dbo].[P_CurrentOf] AS
BEGIN
  DECLARE @empno int
  DECLARE NoResponce CURSOR FOR
    SELECT empno FROM employees;
  OPEN NoResponce;
  FETCH NEXT FROM NoResponce INTO @empno;
  DELETE FROM employees WHERE empno = @empno;
END
GO

* Slide left to see more

remarks ：

Where Current Of Statement allows you to update or delete the last cursor Records taken ,Babelfish I won't support it Current Of sentence , According to the SQL Statement context semantics selects variables

• IDENTITY

The original statement ：

SELECT IDENTITY(INT,1,1) AS rowid,* INTO #tmp
FROM employees
ORDER BY empno

‍* Slide left to see more

Change the following sentence ：

SELECT row_number() over () as rowid, * INTO #tmp
FROM employees
ORDER BY empno

* Slide left to see more

explain ：

Babelfish I won't support it IDENTITY function , Used when carrying INTO Clause SELECT Statement to insert an identity column into a new table , You can use row_number() over () Rewrite the way .

4

  Complex transcoding

Compared with the simple code transformation introduced earlier , The rest of this SQL The statement will be more complicated , There are a lot of changes . meanwhile , You also need to carefully review SQL The relationship between contexts in a statement , To ensure that the modified statement has the same effect as the original statement .

• MERGE

Before this case demonstration , Create two MERGE Used source and target tables

create table source
(
   id      int not null primary key ,
   country varchar(20) null,
   city    varchar(20)
);

insert into source
  (id, country, city)
 VALUES
  (1, 'RUSSIA',  'MOSCOW'),
  (2, 'FRANCE',  'PARIS'),
  (3, 'ENGLAND', 'LONDON'),
  (4, 'USA',     'NEW YORK'),
  (5, 'GERMANY', 'BERLIN'),
  (6, 'BRAZIL',  'BRASILIA');

create table target
(
   id      int not null primary key ,
   country varchar(20) null,
   city    varchar(20)
);

insert into target
  (id, country, city)
 VALUES
  (1, 'JAPAN',   'TOKYO'),
  (4, 'USA',     'DENVER'),
  (7, 'CHINA',   'BEI JING');

* Slide left to see more

The original statement ：

MERGE INTO target AS C2
USING source AS C1 
ON C2.id = C1.id
WHEN MATCHED
   THEN UPDATE 
      SET
         C2.country = C1.country,
         C2.city = c1.city
WHEN NOT MATCHED
   THEN INSERT (id, country, city)
            VALUES (C1.id, C1.country, C1.city);

* Slide left to see more

Modified statement ：

begin
update target set country = C1.country, city = C1.city from (select id, country, city from source) C1 where target.id = C1.id;
insert into target (id, country, city) select * from source as C1 where not exists (select id from target where id = C1.id);
end
go

* Slide left to see more

Second amendment （ stay PostgreSQL End modification ）：

with upsert as
(update target c2 set country=c1.country, city=c1.city 
 from source c1 where c1.id=c2.id
  RETURNING c2.*
)
insert into target select a.id, a.country, a.city 
from source a where a.id not in (select b.id from upsert b);

* Slide left to see more

The third modification （ stay PostgreSQL End modification ）：

insert into target (id,country,city) select id,country,city 
from source
on conflict (id)
do update set country=excluded.country,city=excluded.city;

* Slide left to see more

explain ：

1. MERGE Is a commonly used data merge UPDATE statement ,Babelfish I won't support it MERGE sentence , Generally speaking, according to SQL Semantic in T-SQL Split into multiple DML sentence , It can also be in PostgreSQL Equivalent rewriting at the end .

2. PostgreSQL Not yet MERGE sentence , have access to UPSET or CONFLICT Statements for ,INSERT ON CONFLICT The execution cost of is less than UPDATE sentence .

• FULLTEXT  Full text search

Babelfish I won't support it SQL Server Full text search for , The following statements and system stored procedures are not supported

CREATE、ALTER、DROP FULLTEXT CATALOG
CREATE、ALTER、DROP FULLTEXT INDEX
CREATE、ALTER、DROP FULLTEXT STOPLIST
exec sp_fulltext_database 'enable';

* Slide left to see more

Amazon Aurora PostgreSQL The compatible version adds a pair of pg_bigm Extender support .pg_bigm The extender is in PostgreSQL Full text search is available in . This extension allows users to create 2-gram（ Double group ）, To improve the speed of full-text search . The following example demonstrates how to PostgreSQL Enable full-text search through extension ：

set search_path=dbo;
create extension pg_bigm;

CREATE TABLE fulltext_doc (doc text);
INSERT INTO fulltext_doc VALUES('Babelfish help SQL transfer   Cost optimization ');
INSERT INTO fulltext_doc VALUES('Babelfish help SQL transfer   performance optimization ');
INSERT INTO fulltext_doc VALUES('Babelfish help SQL transfer   Enhance the use experience ');
INSERT INTO fulltext_doc VALUES('Babelfish help SQL transfer   Provided in the  2-gram  Full text search tool ');
INSERT INTO fulltext_doc VALUES('Babelfish help SQL transfer   Provided in the  3-gram  Full text search tool ');

CREATE INDEX fulltext_doc_idx ON fulltext_doc USING gin (doc gin_bigm_ops);
alter table fulltext_doc owner to dbo;

* Slide left to see more

After the full-text search is set successfully, you can search in TDS Port by T-SQL call ：

• SWITCHOFFSET

The original statement ：

SELECT CAST(SWITCHOFFSET(TODATETIMEOFFSET(SYSUTCDATETIME(),'+00:00'),'+08:00') AS DATETIME)

* Slide left to see more

Modified statement （ stay PostgreSQL Create a user-defined function at the end ）：

CREATE OR REPLACE FUNCTION dbo.f_get_cst()
RETURNS sys.datetime AS $$ 
  BEGIN
      RETURN cast(timezone('Asia/Shanghai',now()) as sys.datetime);
END;
$$ LANGUAGE plpgsql;

* Slide left to see more

explain ：

Babelfish I won't support it SWITCHOFFSET and TODATETIMEOFFSET Time zone offset built-in functions such as , Can be in PostgreSQL Create a user-defined function on the TDS Port by T-SQL Call to implement the same function

• XML  Method

Before this case presentation , Create a and xml Parse related tables ：

create table t_xml_test (
  id int,
  country  nvarchar(max),
  industry nvarchar(max)
); 

insert t_xml_test values(1, 'China', 'Manufacturing and foreign trade business');
insert t_xml_test values(2, 'USA', 'Financial and Bioindustry');
insert t_xml_test values(3, 'Russia', 'Resource export');

* Slide left to see more

The original statement ：

create procedure p_xml_test
 @xml xml
as
begin    
   set nocount on
   select * from t_xml_test
   where id in (select imgXML.Item.value('id[1]','int') from @xml.nodes('/root/country') as imgXML(Item)); 
   set nocount off
end
go

* Slide left to see more

Modified statement （ First, in the PostgreSQL Create a user-defined function to parse XML）：

CREATE OR REPLACE FUNCTION xmlQueryID(in_xml xml) 
RETURNS TABLE (id text) 
AS $$
DECLARE
BEGIN
  RETURN QUERY
    select * from (
      WITH xmldata(data) AS (VALUES (in_xml::xml))
      SELECT xmltable.*
      FROM XMLTABLE('/root/country' PASSING (SELECT data FROM xmldata) COLUMNS id text)) as foo;
END;
$$ LANGUAGE plpgsql;

* Slide left to see more

Connect TDS Port in T-SQL Revision in China SQL Statement and call PostgreSQL User defined functions created by the client ：

create procedure p_xml_test
 @xml xml
as
begin    
   set nocount on    
   select * from t_xml_test
   where id in (select * from xmlQueryID(@xml)) ;
   set nocount off
end
go

* Slide left to see more

stay T-SQL Call stored procedure test in , The query results show xml Parsing is normal , The data is correct ：

explain ：Babelfish Parsing is not supported XML Method of data , Include VALUES、XML.NODES And other ways , Can be in PostgreSQL Create a user-defined function on the TDS Port by T-SQL Call to complete XML Data analysis work .

summary

Through the previous case introduction , We show you how to use Babelfish transfer SQL Server Some of the most common unsupported features SQL The transformation method of . At present ,Babelfish for PostgreSQL The project continues to move forward , The version is constantly updated . Each new release adds some important features , Including increasing syntax compatibility and SQL Server Support for native functionality . It is recommended that you plan and implement SQL Server Check frequently during migration Babelfish Feature support description for , Use the latest feature support to complete code conversion .

meanwhile , stay 2021 year 10 month 28 Japan , Amazon cloud technology officially announced the launch  Babelfish for PostgreSQL Open source project . This enables the user to PostgreSQL On the server Babelfish.

More in detail SQL Server Migrate to Amazon Aurora PostgreSQL Please refer to the official for code conversion Migration manual , But please pay attention to , These transformations are all in PostgreSQL End rewrite , It is necessary to consider how to T-SQL Side of the call .

Learn more about Babelfish Information supported by the feature ：https://babelfishpg.org/docs/usage/limitations-of-babelfish/ 

Learn more about Babelfish for PostgreSQL Information about ：https://aws.amazon.com/cn/about-aws/whats-new/2021/10/babelfish-postgresql-open-source-project/ 

Learn more about the migration manual ：https://aws.amazon.com/cn/dms/resources/ 

Author of this article

Tangxiaohua

Amazon cloud technology database solution technical expert , More than 20 years of experience in database industry , Responsible for technical consulting and solutions based on Amazon cloud computing database products . Focus on cloud relational database architecture design 、 test 、 Operation and maintenance 、 Optimization and migration .

hear , Click below 4 Button

You won't encounter bug 了 ！