Lsyncd set up synchronous image - use lsyncd to realize real-time synchronization between local and remote servers

Lysncd namely Live Syncing Daemon, It's an open source real-time data synchronization tool ( Background processes ), be based on inotify and rsync.

lsyncd Will closely monitor the local The server Reference directory on , When there are files or directory changes under the directory , Notify remote immediately The server , And pass rsync or rsync+ssh Method to realize file synchronization . The advantage of doing so is , You can use Lsyncd Building a VPS Synchronous mirror , Application scenarios such as CDN Mirror image 、 Website data backup 、 Website moving, etc .

One 、Lsyncd Installation method

Lsyncd Website is ：http://axkibe.github.io/lsyncd/,Linux The installation command is as follows ：

# Ubuntu
sudo apt install lsyncd
 
# CentOS
sudo yum install epel-release
sudo yum install lsyncd
 
#  Compilation and installation 
cmake .
make
sudo make install

Copy

After successful installation , You can ：/usr/share/doc/lsyncd/examples, notice Lsyncd Configuration examples , You can refer to these examples and write your own configuration .

Two 、Lsyncd Configuration instructions

open Lsyncd Configuration of , As you can see, it mainly includes two parts ： One is setting , The other is synchronization .settings There are global settings ,– The beginning indicates a comment , The code is as follows ：

settings {
        logfile = "/var/log/lsyncd/lsyncd.log",
        statusFile = "/var/log/lsyncd/lsyncd.status"
}
sync {
    default.rsyncssh,
    source = "/root/linsir", -- Source directory 
    host = "192.168.2.16", -- Destination host 
    targetdir = "/root/remote", -- Remote directory 
    delete = true,
    delay = 0,
    exclude={                 
        ".txt"             -- exclude 
    },
    rsync = {
           binary = "/usr/bin/rsync",
           archive = true, -- file 
            compress = true, -- Compress 
            verbose = true, 
           owner = true,   -- Belong to 
            perms = true,   -- jurisdiction 
            _extra = {"--bwlimit=2000"},
            },
        ssh = {
            port = 3322
            }
}

Copy

Here are some descriptions of common options ：

logfile  Define log files stausFile  Define state file nodaemon=true  Indicates that daemon mode is not enabled , Default statusInterval  take lsyncd The status of is written in the above statusFile The interval of , Default 10 second inotifyMode  Appoint inotify Monitored events , The default is CloseWrite, It can also be Modify or CloseWrite or Modify maxProcesses  The maximum number of synchronization processes . If at the same time 20 Files need to be synchronized , and maxProcesses = 8, Then you can see 8 individual rysnc process maxDelays  How many monitored events have accumulated to activate a synchronization , Even in the back delay The delay time has not yet arrived

sync  It defines synchronization parameters , Can continue to use maxDelays To rewrite settings Global variable of . Generally, the first parameter specifies lsyncd In what mode ：rsync、rsyncssh、direct Three models ：

default.rsync ： Synchronization between local directories , Use rsync, It can also be used ssh Form of remote rsync effect , or daemon Connect to the remote server by rsyncd process ; default.direct ： Synchronization between local directories , Use cp、rm Wait for the command to complete the differential file backup ; default.rsyncssh ： Synchronize to remote host Directory ,rsync Of ssh Pattern , Need to use key To certify

source  Synchronized source directory , Using absolute paths .target Define destination address . There are several ways to write corresponding to different modes ：

/tmp/dest ： Local directory synchronization , Can be used for direct and rsync Pattern 172.29.88.223:/tmp/dest ： Sync to remote The server Catalog , Can be used for rsync and rsyncssh Pattern , The splicing command is similar to /usr/bin/rsync -ltsd –delete –include-from=- –exclude=* SOURCE TARGET, The rest is rsync Content. , For example, specify username, Password free synchronization 172.29.88.223::module ： Sync to remote The server Catalog , be used for rsync Pattern

Other optimization options are ：

init  This is an optimization option , When init = false, Only synchronize the files with changes after the process is started , Even if the original directory is different, it will not be synchronized . The default is true delay  Cumulative events , wait for rsync Synchronization delay time , Default 15 second （ The maximum accumulation is 1000 Events that cannot be merged ）. That is to say 15s Changes under the internal monitoring directory , It will accumulate once rsync Sync , Avoid too frequent synchronization .（ Can be merged means ,15s The same file has been modified twice in the , Finally, only the latest files are synchronized ） delete  In order to maintain target And souce Full synchronization ,Lsyncd Default meeting delete = true To allow synchronous deletion . In addition to false, also startup、running value , Please refer to Lsyncd 2.1.x ‖ Layer 4 Config ‖ Default Behavior.

excludeFrom  Exclusion options , Specify the excluded list file later , Such as excludeFrom = “/etc/lsyncd.exclude”, If it is a simple exclusion , have access to exclude = LIST. The exclusion rule here is written in the same way as the original rsync It's a little different , It's simpler ：

Any part of the monitoring path matches a text , Will be excluded , for example /bin/foo/bar You can match rules foo If the rule is slashed / start , Then match all from the beginning If the rule is based on / ending , Then match the end of the monitoring path ? Match any character , But does not include / * matching 0 Or more characters , But does not include / ** matching 0 Or more characters , It can be /

rsync The configuration options are ：

bwlimit The speed limit , Company kb/s, And rsync identical （ Such an important option is not marked in the document ） compress Compressed transmission defaults to true. Between bandwidth and cpu Trade off between loads , Local directory synchronization can be set to false perms File permissions are reserved by default . There are other rsyncssh Configuration items unique to the mode , Such as host、targetdir、rsync_path、password_file, See the following example .rsyncOps={“-avz”,”–delete”} This way of writing is 2.1.* Version is no longer supported . lsyncd.conf There can be multiple sync, Respective source, Respective target, Respective modes , They don't influence each other .

3、 ... and 、Lsyncd Local synchronization

First , Create a log file , For tracking .

mkdir /var/log/lsyncd
touch /var/log/lsyncd/lsyncd.{log,status}

Copy

then , create profile .

mkdir /etc/lsyncd
vim /etc/lsyncd/lsyncd.conf.lua

Copy

The configuration file writes the following （ Pay attention to the Source and Target Replace your source directory path and target directory path respectively ）：

settings = {
        logfile = "/var/log/lsyncd/lsyncd.log",
        statusFile = "/var/log/lsyncd/lsyncd.status"
}
 
sync {
        default.rsync,
        source = "/var/www/source_dir",
        target = "/home/snail/backup",
}

Copy

Last , start-up lsyncd service ：systemctl restart lsyncd, Add files to the source directory , See if the target directory will be increased synchronously . without , Please check the log .

Four 、Lsyncd Remote synchronization

use Lsyncd Realize two VPS host The server Real time synchronization between . First , Enter your source VPS host , Use S-S-H Command remote login to the target VPS On a host .

ssh [email protected]_remote_server_IP 
# Or specify the port 
ssh [email protected]_remote_server_IP -p 1234 

Copy

next , Carry out orders , In the target VPS Generate key on :

ssh-keygen -t rsa  #  Enter all by default 

Copy

After the key generation is successful , You need to aim VPS Copy the key file on to the source VPS On a host , The path is consistent .

then , Re enter the source VPS On , Carry out orders ：ssh-copy-id [email protected]_remote_server_IP , Enter the remote login password again , Confirm to add the key .

After the key is added successfully , Now you can source VPS Use directly on ：ssh [email protected]_remote_server_IP, You can enter the remote without entering a password VPS 了 . Then follow the local synchronization method above , Create logs and configuration files , Pay attention to replacing your own source directory and target directory path .

settings = {
    insist = true,
    logfile = "/var/log/lsyncd/lsyncd.log",
    statusFile = "/var/log/lsyncd/lsyncd.status"
}
 
sync {
    default.rsyncssh,
    source = "/home/www",
    host = "115.28.65.xxx",
    targetdir = "/home/backup"
}

Copy

If your goal is VPS The port of is not the default 22, You can specify the port in the following way ：

settings = {
    insist = true,
    logfile = "/var/log/lsyncd/lsyncd.log",
    statusFile = "/var/log/lsyncd/lsyncd.status"
}
 
sync {
    default.rsyncssh,
    source = "/home/www",
    host = "115.28.65.xxx",
    targetdir = "/home/backup"

    ssh = {
        port = 36000
    }
    
}

Copy

Last , Restart : systemctl restart lsyncd.

Just a moment , You can see that there are already synchronized files in the target directory .

meanwhile , open the log file , You can also see the operation records .

Lsyncd Other pattern examples （ Please replace the path with IP Address ）：

settings {
    logfile ="/usr/local/lsyncd-2.1.5/var/lsyncd.log",
    statusFile ="/usr/local/lsyncd-2.1.5/var/lsyncd.status",
    inotifyMode = "CloseWrite",
    maxProcesses = 8,
    }

--  Remote directory synchronization ,rsync Pattern  + rsyncd daemon
sync {
    default.rsync,
    source    = "/tmp/src",
    target    = "[email protected]::module1",
    delete="running",
    exclude = { ".*", ".tmp" },
    delay = 30,
    init = false,
    rsync     = {
        binary = "/usr/bin/rsync",
        archive = true,
        compress = true,
        verbose   = true,
        password_file = "/etc/rsyncd.d/rsync.pwd",
        _extra    = {"--bwlimit=200"}
        }
    }
--   Remote directory synchronization ,rsync Pattern  + ssh shell
sync {
    default.rsync,
    source    = "/tmp/src",
    target    = "172.29.88.xxx:/tmp/dest",
    -- target    = "[email protected]:/remote/dest",
    --  above target, Note that if you are an ordinary user , Must have write permission 
    maxDelays = 5,
    delay = 30,
    -- init = true,
    rsync     = {
        binary = "/usr/bin/rsync",
        archive = true,
        compress = true,
        bwlimit   = 2000
        -- rsh = "/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no"
        --  If you want to specify another port , Please use the above rsh
        }
    }
--   Remote directory synchronization ,rsync Pattern  + rsyncssh, The effect is the same as above 
sync {
    default.rsyncssh,
    source    = "/tmp/src2",
    host      = "172.29.88.xxx",
    targetdir = "/remote/dir",
    excludeFrom = "/etc/rsyncd.d/rsync_exclude.lst",
    -- maxDelays = 5,
    delay = 0,
    -- init = false,
    rsync    = {
        binary = "/usr/bin/rsync",
        archive = true,
        compress = true,
        verbose   = true,
        _extra = {"--bwlimit=2000"},
        },
    ssh      = {
        port  =  1234
        }
    }

Copy

5、 ... and 、 summary

Lsyncd Configuration is quite simple , If you are troublesome, you don't need to study their parameters , Directly use the Lsyncd Remote synchronization sample code , Other options remain the default .Lsyncd The speed of synchronization depends on your VPS Host bandwidth , And it is incremental synchronous backup .

How to realize one to many real-time synchronization ？ The solution is to add multiple target directory paths in the configuration repeatedly .

sync{ 
default.rsync, 
source='/sourcedir', 
target='targethost1:/targetdir' 
}

sync{ 
default.rsync, 
source='/sourcedir', 
target='targethost2:/targetdir' 
}

sync{ 
default.rsync, 
source='/sourcedir', 
target='targethost3:/targetdir' 
}

Copy

Set up Lsyncd Boot up . The code is as follows ：

/etc/init.d/lsyncd start
chkconfig lsyncd on

Copy