go---- mgo

brief introduction

       mgo By Golang Write open source mongodb drive . because mongodb No official development Golang drive , So this driver is widely used .mongodb The official website also recommends this open source driver , And the author is github It also means that we have received mongodb Official sponsorship . But because of the author's personal arrangement , Update of the driver 、bug Repair 、issue Maintenance was slightly criticized .

       mgo In terms of functions, it is relatively perfect ,api It's also easy to use . because mongodb Rich play ,mgo Huge code , Most of them are related to mongodb Protocol code . The core structure for handling connections and requests , The logic is clear .

Simple use

1 2 3

func dial() {     session ,_ := mgo.Dial("mongodb://127.0.0.1") }

mgo The core data structure for callers is mgo.Session,dial Function demonstrates how to get a session

1 2 3

func foo1() {     session.DB("test").C("coll").Insert(bson.M{ "name":"zhangsan"}) }

foo1 Function through the generated session, towards test Database coll Set writes a piece of data .

but mgo The correct use of is not the case , Instead, you should start from the source each time you use it session Copy

1 2 3 4 5

func foo2() {     s := session.Copy()     defer s.Close()     s.DB("test").C("coll").Insert(bson.M{ "name":"zhangsan"}) }

foo2 Function from source session Copy out a temporary session, Use temporary session Write a piece of data , Close the temporary when the function exits session.

session Copy and concurrency of

        Why should I use it every time Copy, Instead of using it directly Dial Generated session What's the example? ？ Personally think that , This is related to mgo.Session Of Socket Caching mechanism . Look at Session The core data structure of .

1 2 3 4 5 6 7 8 9 10 11

type Session struct {     m                sync.RWMutex     ...     slaveSocket      *mongoSocket     masterSocket     *mongoSocket     ...     consistency      Mode     ...     poolLimit        int     ... }

Here's a list mgo.Session Five private member variables of , And Copy The mechanism is concerned with ,m,slaveSocket,masterSocket.

m yes mgo.Session Concurrency lock for , So all Session Instances are thread safe .

slaveSocket,masterSocket On behalf of Session To mongodb Cache of a physical connection between master and slave nodes . and Session The policy of always takes precedence over cached connections . Whether to cache the connection , from consistency That is the Session The mode of . Suppose in a concurrent program , Use the same Session example , Don't use Copy, And the Session The mode of the instance will just cache the connection , that , All pass this Session Instance operation , Will arrive through the same connection mongodb. although mongodb Its own network model is non blocking communication , Requests can be made over a link , Non blocking processing ; But after a relatively simple performance test , stay mongodb3.0 in ,10 The efficiency of concurrent write of a single connection is twice as high as that of a single connection （ stay mongodb3.4 There is basically no difference in ）. therefore , Use Session Copy An important reason for this is , Requests can be concurrently distributed across multiple connections .

The above is just a matter of efficiency , But the second problem is fatal .mgo.Session Cache one master one slave connection , The instance itself is not responsible for maintenance . in other words , When slaveSocket,masterSocket Either one , Connection is broken ,Session You will not reset the cache , The Session If users do not actively reset the cache , The caller will always get EOF. This happens during master-slave switchover , It also happens when the network jitters . Actively maintain the database in the business code Session The usability of , Obviously, it is not liked .

1 2 3 4 5 6 7

func (s *Session) Copy() *Session {     s.m.Lock()     scopy := copySession(s, true)     s.m.Unlock()     scopy.Refresh()     return scopy }

Above is Copy Implementation of function , Solved the problem of using global Session Two questions of . among ,copySession Will source Session Shallow copy to temporary Session in , Such a source Session The configuration of is copied to the temporary Session in . pivotal Refresh, Will source Session Shallow copy to temporary Session Connection cache pointer to , That is to say slaveSocket,masterSocket Set to empty , This is temporary Session There is no cache connection , Instead, try to get an idle connection .

Session Where does the connection come from ？ Connection pool

The use of Session Copy The need for mechanisms , So here comes the question ,Copy Out of the temporary Session How to get one to mongodb Physical connection of . The answer is connection pooling .mgo I have maintained a set to mongodb Cluster connection pool . This connection pooling mechanism is based on mongodb The database server is the smallest unit , Every mongodb Will be in mgo Inside , Corresponding to one mongoServer Examples of structures , An example represents mgo Held connection to the database . Let's look at the definition of the connection pool .

1 2 3 4 5 6 7 8

type mongoServer struct {     sync.RWMutex     ...     unusedSockets []*mongoSocket     liveSockets   []*mongoSocket     ...     info          *mongoServerInfo }

among ,info It represents the information of the database server corresponding to the instance in the cluster —— whether master,ReplicaSetName etc. . And two Slice, Is the legendary connection pool .unusedSockets Store currently idle connections ,liveSockets Store currently active connections ,Session The cached connections are stored in liveSockets Slicing , But temporarily Session The obtained connection is located at unusedSockets Slicing .

Every mongoServer Will belong to a mongoCluster structure , amount to mgo In the internal , It simulates mongo The model of database cluster .

1 2 3 4 5 6 7 8 9

type mongoCluster struct {     sync.RWMutex     ...     servers      mongoServers     masters      mongoServers     ...     setName      string     ... }

As defined ,mongoCluster Hold a series of mongoServer Example , Spread into two arrays in a master-slave structure .

Every Session Will store their own , To operate mongoCluster References to .

A long journey Session

        The following describes one Copy Out of the temporary Session How to get a mongodb Physically connected .

When temporary Session By Copy come out , And by calling a series of api, Set a database operation to Session After the inside , Everything is ready now , Just connect . New Session First, it will check whether there is a connection available in its cache , Of course, he didn't know that he was a bare knuckle commander who had nothing . because mgo The implementation of the , Poor man, he has to check twice , Use the read lock once , Use write lock once . The author's intention should be to expect the same session During concurrent operations , Before the second exclusive lock check , Happens to cache to a connection , Then you can reduce one operation on the connection pool . But this time , This kind of good thing didn't happen here Session On the body ,“ feel ” Twice “ Pocket ” After repeated confirmation , He finally found himself penniless . Without connection, he asked the organization for help , That's it session To be operated mongodb colony , That is to say mongoCluster structure .

“ organization ” Asked this Session A series of questions , There are two main problems , One is ” Do you want to connect from the main library or from the library ”, Two is “ What is the maximum size of the connection pool you expect ”. The first question is ,Session Good answer , He first looked at his own pattern , Whether you must go to the master database or the slave database , It depends on the situation . Then I looked at whether the operation in my hand was reading or writing , Of course, the write operation cannot be completed in the slave library . The second question is a bit difficult , But he doesn't have to think for himself , Because this is from the source Session The configuration from there , It is also a little ancestral property .

This Cluster At this time, he acts like a shopkeeper , He started with the master and the slave , From my own mongoServer I picked out one of them , Then ask him , Do you have a free connection now . If there is , That's lucky Session You can successfully obtain this idle connection , Happily put it in your pocket and go home to work . But if not , Just right unusedSockets It's empty , Then the shopkeeper will ask another question , Have you exceeded the maximum number of connections this guy expects . If not more than , That's fine , As a waiter mongoServer So I went to work , He will go to the database server he is responsible for to apply for a new connection , Hand in Session In the hand . But if this guy does the math , If you still apply for a new connection , I'm afraid the limit is exceeded , It would be Session Classmate, I'm sorry , Please wait . Every time 100ms, Man, spin once , Waiting for the unusedSockets Available connections appear in .

Of course someone will ask , Then spin on like this , If the connection has been blocked by other Session Occupy , Will it be a dead circle , The answer is no . As a database server administrator, this guy can say , He also has to make sure that the server he is responsible for is still alive . therefore , Guys, classmates, every 15s Will send a message to the server ping command . As Administrator , Man, it doesn't matter whether the connection pool size is too large , That's their ordinary session Trivia to consider . Guys, you should ping When , Also go unusedSockets Look inside , If there is one, the best , Just take one ; If not , Go directly to the server and ask for a new one .ping After that , The new connection will be put into unusedSockets in . In this case , Get connection request in spin , You can get the connection .

After feeling his pocket , Find organization , Ask the man , Man, do a little more work , temporary Session Finally, I got the physical connection to the database I dreamed of , Put him in his pocket （ Of course, there are some patterns Session I won't do that ）. With satisfaction, I wrote out the operation in my hand through this connection , Wait until the database gives him the answer he wants , His life is over . adopt Close Method , We stripped him of the hard won connection in his pocket , Put back to the corresponding mongoServer Of unusedSockets in . Not long after ,GC And killed this Session.