Using streamapi assertion combination and local cache for fuzzy query (nearly 1000 times more efficient than MySQL)

Use StreamAPI Assertion combines with local cache to do fuzzy query （ Than mysql Efficiency improvement is nearly 1000 times ）

I have a need recently , Fuzzy query is required , The first thing I thought of was mysql Paging fuzzy query , However, Party A's demand is to query all data in a fuzzy way , And this fuzzy query is not easy to optimize by adding indexes

When you get this demand , I guess 2w A fuzzy query of the whole table in the database of data , It takes about 10s about ：

Then I think my data volume is not large , It is difficult and almost impossible for the data in the table to exceed 100000 data , If I exist directly JVM cache Is it faster in

Do as you say , Here I use Stream in Assertion combination Fuzzy query in the same way , Unfamiliar friends can see the small examples in the appendix

Here I combine assertions based on the fields of the fuzzy query ：

@Override
public List<T> likeQuery(Map<String, Object> queryMap) {
    
    //  You need to combine assertions first , Because it's an and operation , So initialize a successful assertion 
    Predicate<Book> p = Objects::nonNull;
    for (Entry<String, Object> entry : queryMap.entrySet()) {
    
        String k = entry.getKey();
        String v = String.valueOf(entry.getValue());
        Predicate<Book> p2 = null;
        if (k.equalsIgnoreCase("bookName")) {
    
            p2 = book -> book.matchName(v);
        } else if (k.equalsIgnoreCase("author")) {
    
            p2 = book -> book.matchAuthor(v);
        } else if (k.equalsIgnoreCase("publishArea")) {
    
            p2 = book -> book.matchPublishArea(v);
        }
        //  Assertion combination 
        if (p2 != null){
    
            p = p.and(p2);
        }
    }
    //  I use it myself List Caching done , All through here this obtain 
    return this.stream()
            .filter(p)
            .collect(Collectors.toList());
}

Then write the comparison logic in the entity class ：

@Override
public  boolean matchName(String str) {
    
    if(this.bookName == null) return false;
    return this.bookName.contains(str);
}

Then it's time to see the effect , The fuzzy query of 20000 measured data will not exceed 100ms, Generally lower , And it also includes http Requested time

adopt AB Measure the pressure of the interface , Total settings 1000 Threads , Make 100000 requests and record the time

100000 requests take only 31 second

see JVM Heap space occupancy

Discovery cache 2 Ten thousand data , Heap space occupation is also within a reasonable range , And GC Memory usage will be greatly reduced

The number of threads is also within a reasonable range

Fuzzy query on the front end , Very silky , There's no carton

Summary ：

It is recommended not to use parallel streams to improve performance , Because there are many uncontrollable risks in the process of processing , For example, if you use thread unsafe collection classes for parallel stream operations , It is very likely to cause thread safety problems

The author suggests using fork/join The framework splits tasks manually , Ensure thread safety

appendix ,Stream Examples of assertive composite fuzzy queries

Predicate:<T> Assertion interface ; Enter an object , Return to one Boolean value , The abstract functions to be implemented are boolean test(T t)

for example ：

/** * 4.Predicate<T>: Assertion interface  */
@Test
public void test04(){
    
    List<String> list= Arrays.asList("Hello"," I am Liang Fengxian ","Lambda","www","ok");
    // Filter strings longer than three 
    System.out.println(filterStr(list,(str)->str.length()>3));
}
// demand , Add a string that meets the condition to the collection 
private List<String> filterStr(List<String> list, Predicate<String> predicate) {
    
    List<String> strList = new ArrayList<>();
    for (String str : list) {
    
        if (predicate.test(str)) {
    
            strList.add(str);
        }
    }
    return strList;
}

Assertion combination ( Predicate combination ), And or not

public class PredicateFunction {
    
    // Extract common code 
    static List<Apple> filterApples(List<Apple> inventory, Predicate<Apple> p){
    
        List<Apple> result = new ArrayList<>();
        inventory.forEach(apple -> {
    
            if(p.test(apple)){
    
                result.add(apple);
            }
        });
        return result;
    }

    public static void main(String[] args) {
    
        String[] colors = {
    "green","yellow","blue"};
        Random r = new Random();
        List<Apple> list = new ArrayList<>();
        for (int i = 0; i < 10; i++) {
    
            list.add(new Apple(colors[r.nextInt(colors.length)],r.nextInt(200)));
        }
        System.out.println(filterApples(list,Apple::isGreenApple));
        System.out.println(filterApples(list,Apple::isHealthyApple));
        // Combine two conditions 
        Predicate<Apple> p1 = Apple::isGreenApple;
        Predicate<Apple> p2 = Apple::isHealthyApple;
        // Or operations 
        System.out.println(filterApples(list,p1.or(p2)));
        //and operation 
        System.out.println(filterApples(list,p1.and(p2)));
    }
}

@Data
@AllArgsConstructor
class Apple{
    
    private String color;
    private int weight;
    public static boolean isGreenApple(Apple apple){
    
        return "green".equalsIgnoreCase(apple.getColor());
    }
    public static boolean isHealthyApple(Apple apple){
    
        return apple.getWeight() > 150;
    }
}