One hour is worth seven days! Ingenuity in the work of programmers

The original plan 7 Day's work ,1 Hours to complete ！ Did I hang up ？

The project I'm working on recently , Help you learn programming ：https://github.com/liyupi/code-nav

Hello everyone , I'm fish skin , Today, I share my little joy in my work , Also hope to bring you some programming thinking .

Evil rise

Here's the thing , Recently, we are developing a Internal use only The data analysis system of , I do the back end , Another guy is the front end .

The function we want to achieve is ： The user can input anything on the interface SQL Data query statement , And save it , Generate a data Kanban . Later, users can open the Kanban at any time to browse and analyze SQL The latest data found , Instead of typing it over and over again SQL sentence .

for instance ！

Suppose we have a big data warehouse , Huge amounts of data are stored , There are men and women ：

Product students may only want to analyze part of the data , So I wrote the following SQL Statement to query all men ：

select * from table
	where  Gender  = ' male ';

Will be SQL Statement save , I need one “ Men's data board ”, after , You can view and analyze all male data on the Kanban page .

Data kanban

To achieve this requirement , One of the easiest ways is , Directly input the user's input on the interface SQL The string is sent to the back end for saving , When you need to look at the data , The back end then queries the data from the database with this string .

Write SQL The configuration process ：

Open Kanban and browse the data flow ：

Since the user is allowed to input at will , So here's the problem .

If you are careless and have a wrong number SQL sentence ：

#  error  
sleetc * from table
#  correct  
select * from table;

Or a little confused SQL The grammar of ：

#  error  
select table from a;
#  correct  
select a from table;

Even little troublemakers don't play by the rules , Enter some messy characters ：

#  error  
select q^q from table;
#  correct  
select q from table;

If you put these wrong SQL Statement to the back end , The back end directly uses it to look up the database , It will inevitably lead to query errors , I found out about loneliness .

For real-time queries , It's no problem , If the query fails, it's a big deal to modify the statement query again .

But what I need to do is to allow users to permanently save query statements as Kanban configurations , Convenient for subsequent automatic query data . And write SQL The configured user may not be the same person as the user watching the data , If small A It's not found in the configuration SQL The statement is wrong , Then come and check the data kanban B You'll be confused , Why can't you see the data ？ The data is not ready yet , Or the query data is 0 Line? , Or do I have no browsing rights ？

He would never have thought of , Has been configured successfully SQL sentence , It turned out to be wrong ！

therefore , You need to configure the user input SQL check , See if it's legal .

Make a metaphor , The front end is an employee at the bottom （ Ignorant little development ）, The back end is the team leader , Database is the big boss . After small developers make a demand , It should be checked by the team leader first , After the group leader said there was no problem , And then give it to the big boss .

So how to check SQL Sentence? ？

Because the user's input is completely uncertain , They wrote SQL Sentences can be stinky and long . So I just thought of the need , I feel a headache , I feel like I'm in trouble , Don't plan for yourself conservatively 7 Days to complete .

You can think about it first, if you want to achieve SQL Statement verification , What would you do ？

Here's my thinking process .

Racking my brains

First , We need to be clear ： It's on the front end , Or back-end verification ？

Actually , Whether on the front end or the back end , Verification is very important , Can effectively prevent a lot of wrong input . But because it is the back-end program that operates the database directly , It's the last line of defense for databases , So I suggest Write the verification logic on the back end . The database is very tender , He can't handle it himself , Need a back-end program to help him grasp .

So how to check on the back end SQL Well ？

Find a ready-made one

First , Never ask Baidu in case of trouble , No, search the warehouse . Now there are many open source projects on the Internet , You might as well have a look , Is there any ready-made SQL Verify class library . Ideally , There is a tool class function , I passed it on to him SQL String as argument , He went straight back to me true or false.

However , I found myself thinking peach, All kinds of open source projects have been searched , There's no one that can be used out of the box PostgreSQL Verification Library .

It seems , You can only do it yourself , There's plenty to eat .

Simulation query

To achieve their own verification , My first thought was to simulate a query . The user just wrote SQL After the statement , Even if he doesn't need to browse the data query results now , I can also save the configuration when he , In his writing SQL Go to the database once . If the inquiry is correct , Just explain SQL The sentence is legal , Allow to save .

This is the most direct way , Is the most convenient , Basically no development costs , Equisetum odoratum ！ It's like a small developer writing bad code , Give it to the team leader , But the group leader doesn't talk about martial arts , I can't understand the code （ Or maybe I don't want to see it ）, Simply throw the code directly to the big boss , The big boss said no problem , Small development goes online again . The group leader is ecstatic ！

however , There's a fatal problem ： The user is configuring SQL When the sentence is , The data table may not be ready yet , Whether the statement is correct or not , Can't find the data .

therefore , Will be SQL Statement is sent directly to the database , Make sure the data table exists first . If exist , It can be verified by simulating query ; If it does not exist , It can only be verified in other ways on the back end .

It's like when the team leader wants to throw the bad code directly to the big boss , The big boss is not here , At this time , It's up to you to check .

Regular expressions

To verify a string in a program , The first thing I thought of was Regular expressions , That is to use specific syntax to match the same kind of string with similar rules , The common ones are checking mobile phone numbers 、 Checkbox 、 Check ID card, etc .

Before using regular expressions for validation , We need to analyze the string first , See if they have a similar structure 、 Which parts are similar . such as QQ mailbox , The structure is very regular , Basic it is [email protected], therefore , You can use regular expressions /^\[email protected]$/ To verify .

Looking back at our needs , What to check is SQL sentence , It seems to be quite regular , It's just about which table to query 、 Which lines to choose 、 Select which columns 、 How to sort and so on , The general structure is like this ：

SELECT select_list 
[ INTO new_table ] 
FROM table_source 
[ WHERE search_condition ] 
[ GROUP BY group_by_expression ] 
[ HAVING search_condition ] 
[ ORDER BY order_expression [ ASC | DESC ] ]

According to this structure , It's easy to write rough regular expressions . however , In data business SQL It's a lot more complicated than that , It contains all kinds of four operations 、IF ... ELSE conditional 、CASE ... WHEN ... Branch , character string 、 Date type handler , There are also various aggregation functions and so on , Like this one down here SQL：

select a as b, 
	sum(case when (false) then d / a else 2 end) as c
	from table
	where a = 1
	group by b, c;

If the above fragmentary syntax is matched by regular expressions , But it's too much trouble ！ Think about naokuo's pain again .

Analytic expression

Since writing a set of regular expressions is cumbersome , All I can think of is SQL It's broken . It can be done in a way similar to Compiler Principle syntax analysis , To get a SQL Parser , Will be complete SQL Statement into an abstract syntax tree （AST）, Each node is a small expression , In this way, more precise verification can be achieved SQL The validity of the statement .

SQL Expression abstract syntax tree

If you start from scratch to achieve such a set of SQL Parser , It's too much trouble , And I don't have some professional knowledge and I can't write it out . therefore , I'll search the Internet first , See if there is a ready-made parser engine .

The search results are quite satisfactory , Found some famous parsing engines , But I took a look , Read for a long time , It's hard to use their source code directly . The way to compromise is to write a parser according to their source code .

Think of it here. , There's not just a chill on the top of my head , Feel like you're timing yourself 7 There are fewer days .

Substitute stealthily

the second day , I thought about it again , There are so many ready-made class libraries on the Internet , No one can meet my needs ？ Even if it's not completely available , Can we find a relatively easy to use one ？

After all, it's too much trouble to write this complicated verification logic by yourself , So I have to struggle again ！

therefore , I took out the Royal duckling , Start talking to it ：SQL check 、SQL check 、SQL check ...

I ： When will it be used SQL Check ？

Yellow duck ： When you need to look up the database .

I ： What will go to the database ？

Yellow duck ： frame 、 Database connection pool 、 Or agency .

I ： So when these things look up the database , Will you check it for us ？

Yellow duck ： Check... Check , You know , Must the function you need be verification ？

wait , It seems that I suddenly realized ！

Since there's no way to find the ready-made ones directly SQL Verify class library , Let's have a Substitute stealthily , Think about whether other libraries contain SQL Parsing function , If parsing fails , It doesn't mean SQL illegal , Check failed ！

I began to think about the technology I used to access databases , Suddenly thought of , Ali's Druid The database connection pool class library seems to have a SQL The function of statement formatting , It's a great way to put the clutter of SQL Rearrange . If you can do it to SQL format , Does it mean , This class library has the ability to SQL Sentence parsing ？

Take a close look at Druid Documents , I found that there is a class called SQLUtils, This class has a method called parseStatements, There are many different kinds of SQL Dialect analysis , such as MySQL、PostgreSQL etc. .

//  analysis , Accept  sql  Statements and database dialects are parameters 
SQLUtils.parseStatements(sql, POSTGRESQL);

When parsing fails , It throws an exception , Express SQL Illegal statement , Just to meet my needs ！

Final , My code is as follows ：

try {
  String sql = "select * from a";
  SQLUtils.parseStatements(sql, POSTGRESQL);
  return true;
} catch (ParserException e) {
  LOGGER.error(" Parse failure ", e);
	return false;
}

I wrote the code in a few minutes , And then I spent some time typing in all kinds of SQL Statement to test , Although it can only implement basic syntax checking , But in terms of both effectiveness and cost , I think it's good , A lot of time saved can continue to refine and optimize other code of the project .

The key is , My heart is not tired , The hair is sticking out again ！

---

Through this matter , Bring me three thoughts ：

1. Before we look for the project code 、 When looking for class libraries , If there's no way to find one that directly meets the needs , Then we can turn our thinking from the whole to the part , Think about whether other projects include the features you're looking for . It's like looking it up in a dictionary , You have to look up the words apple, But there's only the first letter in the catalog a, This is the time , You can't just stare at a see , It's about seeing what's in the dictionary , Actually apple It's hidden in a In .
2. The forefathers planted trees , Descendants cool , Now there are too many project codes available online , If it's not for learning , A lot of things don't have to be realized by themselves .
3. Pay attention to accumulation when writing code , Learn more about technology , And sum it up in your arsenal , Otherwise, you can't find the trees planted by predecessors , It's a pity .

Of course , If conditions permit , The front end can also be checked , But there's no need at the moment , Let's use CodeMirror Make one SQL Code highlight instead .

If it really lets you implement the front end SQL check , What would you do ？

I'm fish skin , Originality is not easy. , If you think the article is good , Hope friends give the thumbs-up Under the support , Give me some creative motivation .

I'm still developing my Programming navigation （https://www.code-nav.cn）, A project to help you find programming resources , Welcome to use ！

Various programming resources

How did I teach myself in college , Get Tencent 、 Byte and other big factories offer Of , You can read this article , No more confusion ！

I studied computer for four years , Mutual encouragement ！