Huawei cloud image engine service

Preface

This article will be divided into the following 3 There are three parts to introduce ：

The first 1 Chapter What is graph Computing

The first 2 Chapter Figure engine service introduction

The first 3 Chapter Introduction to query and analysis functions

This paper mainly introduces the definition and characteristics of graph calculation , I hope this article can let you know about graph computing and Huawei cloud graph engine service , Master the use of graph engine for query and Analysis

One 、 What is graph Computing

1. chart . Everywhere

in real life , From person to person , Articles and articles , There are diverse relationships between people and objects , We will get a variety of pictures , As shown below , It can be seen that pictures are everywhere in our real life .

Communication network

The vertices : devices, routers; edge : network flow

Social networks

The vertices : users, posts; edge : relations, Likes

User product map

The vertices : users, items; edge : Ratings

Weather changing Wiki article

The vertices : Wiki articles; edge : Links

2. What is graph Computing

• Definition ：“ Figure calculation ” In order to “ Relationship ” Based on the real world “ chart ” Abstract expression of structure , And the calculation mode on this data structure

• describe ： G = （V,E,D） V = vertex （ Vertex or node ） E = edge （ edge ） D = data （ attribute & The weight ）

• Good at ： Good at dealing with connected 、 Massive 、 The query of changeable data 、 analysis ;

characteristic ：

• Data independent structuring , Diverse data

• Multi data association , Have communication ability

• Data dynamics , Real time interactive analysis

• Explainable

Natural graph is data relation ：

• social connections

• Information dissemination network

• Communication network

• Organizational structure

• ……

Suitable for the scene ：

• Opinion leaders dig

• Friend recommendation

• User segmentation

• Organizational structure analysis

• ……

3. Figure computational advantage

• Strong presentation skills , Suitable for expressing a variety of complex relationships 、 Support rich semantics

In terms of expression ability , The data expressed in traditional database is relatively single , Graph calculation can support N Yes N The expression of , More diverse forms , Better presentation skills

Rich data expression , Scalability support

relational database Figure calculation

(1:1 or 1:N) (N:N)

• Big data , Potential relationship mining Fast and efficient

In terms of performance , The query speed of graph computing for multiple relationships is much higher than that of traditional database , At the same time, graph has high-performance parallel computing ability

It can be seen from both expression ability and performance , Graph computing has greater application advantages ,

Fast multi hop relation query

Hops	relational database	Figure Engine Services	Number of records returned
2	0.162	0.025289	213597
3	63.589	0.779019	1031115
4	1368.662	1.452095	1227152
5	Hang in the air	1.474496	1230000

• Graph calculation may become AI The key basic technology of the next hop

The performance of graph computing in high-dimensional sparse scenes is expected to be improved by a hundredfold

NIPS/SIGMOD/NDSI Such as the database 、 Figure calculation and AI Summary of the views of the field Summit ：

• The math , stay AI In calculation , Graph calculation and deep learning are two sides of the coin , It has equivalence 、 Interchangeable

4. Figure calculation ∙ Steadily rising to the peak

5. Figure computing development - Technology is getting better , The ecology is becoming more and more stable

Graph computing technology has become more and more perfect , Many companies have released their own graph computing products , The ecology has also become more stable

2、 Figure engine service introduction

1. Figure Engine Services ： Analysis and query of large-scale integrated graph

Huawei cloud map engine service is a large-scale integrated map analysis and query platform , There is a rich library of graph analysis algorithms , High performance graph computing kernel , Distributed high performance storage engine , Support the extension of attribute graph , At the same time, the open source interface is compatible , And the results are diversified in format , With a large scale , High performance , Integration of query and analysis , It's easy to use

2. Figure overall solution of engine service

Users can batch import their own historical data into the graph engine service , At the same time, it also provides an incremental data import method , It is convenient for users to update data in real time , It can also be provided through the graph engine SDK Easy access to graph Engine Services , Easy to use , It's easy to operate .

3. Figure engine service usage page

• Figure query area ： Support standard graph query language Gremlin, Compatible with your usage habits

• Figure analysis area ： Provide rich graph analysis algorithms , Simple and easy to use

• Visualization area ： Support , WYSIWYG visual presentation

• Result recording area ： The operation record is visible , Support JSON Format result Export , Easy access to

4. Use scenarios

The usage scenarios of Huawei cloud image engine service are also very rich

Internet

Good friends / goods / information

News recommendation

Abnormal behavior analysis and public opinion analysis

Knowledge map

Intelligent Q & a knowledge disambiguation

Learning path recommendation

Financial risk control

Analysis of tracking reliability of lost contact personnel in real-time fraud detection

Smart city

Path planning

Pipeline pressure regulation and urban road network regulation

3、 Query function

What is? Gremlin？

Gremlin yes Apache TinkerPop Graph traversal language under the framework Gremlin Is a functional data flow language , It enables users to express complex attribute graphs in a concise way

（property graph） Traverse or query each Gremlin Traversal consists of a series of steps

（step, There may be nested ） form , Every step is in the data stream （data stream） Perform an atomic operation on

In addition to Huawei graphics engine service ,TA They also use Gremlin, Has become the industry's de facto standard

    

1. Gremlin Basic operation （ One ）

Common statements are as follows , The complex query function can be realized through the easy combination of various statements

• map(x)

select(“a”,”b”) id() mean() count()value(“age”) order()sum() groupCount()

• flatMap(x)

out(“knows”) values(“name”) properties() v()in(“created”) match(x,y,z) outE(“knows”)

• filter(x)

has(“name”,“gremlin”) and(x,y)coin(0.5)dedup(10)where(“a”,eq(“b”)or(x,y)sample(10)

• sideEffect(x)

groupCount(“m”)tree(“m”)subgraph(“m”)store(“m”)group(“m”)

2. Gremlin Basic operation （ Two ）

for example ： “gremlin” The age distribution of people you know

We can break it down into the following five steps

(1) All the people

(2) Then find the name “gremlin” People who

(3) gremlin” People I know

(4) Based on the previous step , Check the age of these people

(5) Get the age distribution

Go through the five steps above , We can do that gremlin” The age distribution of people you know

3. Query function summary

Support industry standard graph query language ：Gremlin

Gremlin Traversal consists of a series of steps （ There may be nested ） form , Each step performs an atomic operation on the data stream

Basic operation ：map(x), filterMap(x)、filter(x)、sideEffect(x) etc. ：

give an example ：”gremlin“ The age distribution of people you know

g.V().has("name","gremlin").cout("knows").values("age").groupCount()

The above is a brief introduction to Huawei cloud image engine

4、 Analysis function

1. Application scenarios and algorithms

2. Algorithm in practice ： Social networks — Individual value exploration

background ：

Take Sina Weibo as an example , How to rate each user ？

（ Traditional scoring = Pay attention to several + Number of fans + Number of microblogs ）

be based on PageRank User evaluation of

Using various centrality （centrality）,TrustRank Algorithms like search for leaders in social networks （ High value users ）

3. Algorithm in practice ： Social networks — Friend recommendation

background ：

you are here Facebook Update your contacts last week , Will be pushed by many contacts in the background , You'll find that Facebook The push is controlled and efficient , Neither too much bother , It can open up your contacts again

 

 Friends' recommendation based on ternary closure theory

Based on ternary closure theory , Combined with the triangle counting on the graph , Clustering coefficient , Shortest path ,k Friend degree , Correlation prediction and other algorithms , Conduct social network tightness analysis , Realize friend recommendation

4. Algorithm in practice ： Social networks — Community recommendation

background ：

I know about the people in Lao Wang's circle of friends , Can you estimate Lao Wang's economic situation 、 Credit risks ？

 

    

“ Birds of a feather flock together , Birds of a feather flock together ”

Using community algorithm （K-core, Louvain,Label Propagation etc. ） Carry out community / Group analysis

5. Algorithm in practice ： social contact / multimedia / Online retailers — Real-time recommendation

background ：

On a movie platform , With a huge user base 、 Movie library , How to make real-time and accurate recommendations when users conduct a series of behaviors ？

How to solve the problem of data sparsity , How to consider the impact of complex relationships ……?

use Pixie、GRank And other algorithms for large amounts of data 、 Real time recommendation in complex scenarios

6. Algorithm in practice ： Analysis function summary

Application scenarios ： Social networks 、 Precision marketing 、 Credit insurance, etc

Functional division ： Find a connection （ Link analysis 、 degree / neighbor ）、 Find a path 、 Find a group ( Community class 、 Communication class )、 Look for features

Graph algorithm ：PageRank、 shortest path 、K-hop、 Clustering coefficient 、 Trigonometric counting 、Centrality、 Maximum associated subgraph 、

Degree Correlation、K-core、 Tag spread 、Louvain、PPR、 Relationship prediction 、 Propagation model 、node2vec etc.

Examples of algorithm practice ：

Individual value exploration ：

PageRank、Centrality、TrustRank etc.

Friend recommendation ：

Based on ternary closure theory

Trigonometric counting , Clustering coefficient , Shortest path ,k Friends, etc

The community found ：

K-core, Louvain,Label Propagation And so on

Real-time recommendation ：

Pixie、GRank etc.