知乎專欄 | 多維度架構 | 微信號 netkiller-ebook | QQ群:128659835 請註明“讀者” |
目錄
rsync is an open source utility that provides fast incremental file transfer. rsync is freely available under the GNU General Public License version 2 and is currently being maintained by Wayne Davison.
過程 70.1. rsync
安裝rsync
在AS3 第二張CD上找到rsync-2.5.6-20.i386.rpm
[root@linuxas3 root]# cd /mnt [root@linuxas3 mnt]# mount cdrom [root@linuxas3 mnt]# cd cdrom/RedHat/RPMS [root@linuxas3 RPMS]# rpm -ivh rsync-2.5.6-20.i386.rpm
配置/etc/rsyncd.conf
在rh9,as3系統上rsync安裝後,並沒有創建rsyncd.conf文檔,要自己創建rsyncd.conf文檔
[root@linuxas3 root]# vi /etc/rsyncd.conf uid=nobody gid=nobody max connections=5 use chroot=no log file=/var/log/rsyncd.log pid file=/var/run/rsyncd.pid lock file=/var/run/rsyncd.lock #auth users=root secrets file=/etc/rsyncd.passwd [postfix] path=/var/mail comment = backup mail ignore errors read only = yes list = no auth users = postfix [netkiller] path=/home/netkiller/web comment = backup 9812.net ignore errors read only = yes list = no auth users = netkiller [pgsqldb] path=/var/lib/pgsql comment = backup postgresql database ignore errors read only = yes list = no
選項說明
uid = nobody gid = nobody use chroot = no # 不使用chroot max connections = 4 # 最大連接數為4 pid file = /var/run/rsyncd.pid #進程ID檔案 lock file = /var/run/rsync.lock log file = /var/log/rsyncd.log # 日誌記錄檔案 secrets file = /etc/rsyncd.pwd # 認證檔案名,主要保存用戶密碼,權限建議設為600,所有者root [module] # 這裡是認證的模組名,在client端需要指定 path = /var/mail # 需要做鏡像的目錄 comment = backup xxxx # 註釋 ignore errors # 可以忽略一些無關的IO錯誤 read only = yes # 只讀 list = no # 不允許列檔案 auth users = postfix # 認證的用戶名,如果沒有這行,則表明是匿名 [other] path = /path/to... comment = xxxxx
密碼檔案
在server端生成一個密碼檔案/etc/rsyncd.pwd
[root@linuxas3 root]# echo postfix:xxx >>/etc/rsyncd.pwd [root@linuxas3 root]# echo netkiller:xxx >>/etc/rsyncd.pwd [root@linuxas3 root]# chmod 600 /etc/rsyncd.pwd
啟動rsync daemon
[root@linuxas3 root]# rsync --daemon
添加到啟動檔案
echo "rsync --daemon" >> /etc/rc.d/rc.local [ OK ]
cat /etc/rc.d/rc.local 確認一下
測試
[root@linux docbook]# rsync rsync://netkiller.8800.org/netkiller [root@linux tmp]# rsync rsync://netkiller@netkiller.8800.org/netkiller Password: [chen@linux temp]$ rsync -vzrtopg --progress --delete postfix@netkiller.8800.org::postfix /tmp Password:
過程 70.2. installation setp by setp
installation
$ sudo apt-get install rsync
enable
$ sudo vim /etc/default/rsync RSYNC_ENABLE=true
config /etc/rsyncd.conf
$ sudo vim /etc/rsyncd.conf uid=nobody gid=nobody max connections=5 use chroot=no pid file=/var/run/rsyncd.pid lock file=/var/run/rsyncd.lock log file=/var/log/rsyncd.log #auth users=root secrets file=/etc/rsyncd.secrets [neo] path=/home/neo/www comment = backup neo ignore errors read only = yes list = no auth users = neo [netkiller] path=/home/netkiller/public_html comment = backup netkiller ignore errors read only = yes list = no auth users = netkiller [mirror] path=/var/www/netkiller.8800.org/html/ comment = mirror netkiller.8800.org exclude = .svn ignore errors read only = yes list = yes [music] path=/var/music comment = backup music database ignore errors read only = yes list = no [pgsqldb] path=/var/lib/pgsql comment = backup postgresql database ignore errors read only = yes list = no auth users = neo,netkiller
/etc/rsyncd.secrets
$ sudo vim /etc/rsyncd.secrets neo:123456 netkiller:123456
$ sudo chmod 600 /etc/rsyncd.secrets
start
$ sudo /etc/init.d/rsync start
test
$ rsync -vzrtopg --progress --delete neo@localhost::neo /tmp/test1/ $ rsync -vzrtopg --progress --delete localhost::music /tmp/test2/
firewall
$ sudo ufw allow rsync
CentOS 6 之前的版本可以使用 xinetd, CentOS 7 不建議使用
yum install xinetd
配置 /etc/xinetd.d/rsync
vim /etc/xinetd.d/rsync # default: off # description: The rsync server is a good addition to an ftp server, as it \ # allows crc checksumming etc. service rsync { disable = yes flags = IPv6 socket_type = stream wait = no user = root server = /usr/bin/rsync server_args = --daemon log_on_failure += USERID }
disable = yes 改為 disable = no
# vim /etc/rsyncd.conf
chkconfig xinetd on /etc/init.d/xinetd restart
systemctl enable rsyncd systemctl start rsyncd systemctl restart rsyncd systemctl stop rsyncd
啟動配置項 /etc/sysconfig/rsyncd
# cat /etc/sysconfig/rsyncd OPTIONS=""
啟動腳本
# cat /usr/lib/systemd/system/rsyncd.service [Unit] Description=fast remote file copy program daemon ConditionPathExists=/etc/rsyncd.conf [Service] EnvironmentFile=/etc/sysconfig/rsyncd ExecStart=/usr/bin/rsync --daemon --no-detach "$OPTIONS" [Install] WantedBy=multi-user.target
# Minimal configuration file for rsync daemon # See rsync(1) and rsyncd.conf(5) man pages for help # This line is required by the /etc/init.d/rsyncd script pid file = /var/run/rsyncd.pid port = 873 address = 192.168.1.171 #uid = nobody #gid = nobody uid = root gid = root use chroot = yes read only = yes #limit access to private LANs hosts allow=192.168.1.0/255.255.255.0 10.0.1.0/255.255.255.0 hosts deny=* max connections = 5 motd file = /etc/rsyncd/rsyncd.motd #This will give you a separate log file #log file = /var/log/rsync.log #This will log every file transferred - up to 85,000+ per user, per sync #transfer logging = yes log format = %t %a %m %f %b syslog facility = local3 timeout = 300 [home] path = /home list=yes ignore errors auth users = linux secrets file = /etc/rsyncd/rsyncd.secrets comment = linuxsir home exclude = beinan/ samba/ [beinan] path = /opt list=no ignore errors comment = optdir auth users = beinan secrets file = /etc/rsyncd/rsyncd.secrets [www] path = /www/ ignore errors read only = true list = false hosts allow = 172.16.1.1 hosts deny = 0.0.0.0/32 auth users = backup secrets file = /etc/backserver.pas [web_user1] path = /home/web_user1/ ignore errors read only = true list = false hosts allow = 202.99.11.121 hosts deny = 0.0.0.0/32 uid = web_user1 gid = web_user1 auth users = backup secrets file = /etc/backserver.pas [pub] comment = Random things available for download path = /path/to/my/public/share read only = yes list = yes uid = nobody gid = nobody auth users = pub secrets file = /etc/rsyncd.secrets
命令行選項 -v, --verbose 詳細模式輸出 -q, --quiet 精簡輸出模式 -c, --checksum 打開校驗開關,強制對檔案傳輸進行校驗 -a, --archive 歸檔模式,表示以遞歸方式傳輸檔案,並保持所有檔案屬性,等於-rlptgoD -r, --recursive 對子目錄以遞歸模式處理 -R, --relative 使用相對路徑信息 -b, --backup 創建備份,也就是對於目的已經存在有同樣的檔案名時,將老的檔案重新命名為~filename。可以使用--suffix選項來指定不同的備份檔案首碼。 --backup-dir 將備份檔案(如~filename)存放在在目錄下。 -suffix=SUFFIX 定義備份檔案首碼 -u, --update 僅僅進行更新,也就是跳過所有已經存在於DST,並且檔案時間晚于要備份的檔案。(不覆蓋更新的檔案) -l, --links 保留軟鏈結 -L, --copy-links 想對待常規檔案一樣處理軟鏈結 --copy-unsafe-links 僅僅拷貝指向SRC路徑目錄樹以外的鏈結 --safe-links 忽略指向SRC路徑目錄樹以外的鏈結 -H, --hard-links 保留硬鏈結 -p, --perms 保持檔案權限 -o, --owner 保持檔案屬主信息 -g, --group 保持檔案屬組信息 -D, --devices 保持設備檔案信息 -t, --times 保持檔案時間信息 -S, --sparse 對稀疏檔案進行特殊處理以節省DST的空間 -n, --dry-run現實哪些檔案將被傳輸 -W, --whole-file 拷貝檔案,不進行增量檢測 -x, --one-file-system 不要跨越檔案系統邊界 -B, --block-size=SIZE 檢驗算法使用的塊尺寸,預設是700位元組 -e, --rsh=COMMAND 指定使用rsh、ssh方式進行數據同步 --rsync-path=PATH 指定遠程伺服器上的rsync命令所在路徑信息 -C, --cvs-exclude 使用和CVS一樣的方法自動忽略檔案,用來排除那些不希望傳輸的檔案 --existing 僅僅更新那些已經存在於DST的檔案,而不備份那些新創建的檔案 --delete 刪除那些DST中SRC沒有的檔案 --delete-excluded 同樣刪除接收端那些被該選項指定排除的檔案 --delete-after 傳輸結束以後再刪除 --ignore-errors 及時出現IO錯誤也進行刪除 --max-delete=NUM 最多刪除NUM個檔案 --partial 保留那些因故沒有完全傳輸的檔案,以是加快隨後的再次傳輸 --force 強制刪除目錄,即使不為空 --numeric-ids 不將數字的用戶和組ID匹配為用戶名和組名 --timeout=TIME IP超時時間,單位為秒 -I, --ignore-times 不跳過那些有同樣的時間和長度的檔案 --size-only 當決定是否要備份檔案時,僅僅察看檔案大小而不考慮檔案時間 --modify-window=NUM 決定檔案是否時間相同時使用的時間戳窗口,預設為0 -T --temp-dir=DIR 在DIR中創建臨時檔案 --compare-dest=DIR 同樣比較DIR中的檔案來決定是否需要備份 -P 等同於 --partial --progress 顯示備份過程 -z, --compress 對備份的檔案在傳輸時進行壓縮處理 --exclude=PATTERN 指定排除不需要傳輸的檔案模式 --include=PATTERN 指定不排除而需要傳輸的檔案模式 --exclude-from=FILE 排除FILE中指定模式的檔案 --include-from=FILE 不排除FILE指定模式匹配的檔案 --version 打印版本信息 --address 綁定到特定的地址 --config=FILE 指定其他的配置檔案,不使用預設的rsyncd.conf檔案 --port=PORT 指定其他的rsync服務連接埠 --blocking-io 對遠程shell使用阻塞IO -stats 給出某些檔案的傳輸狀態 --progress 在傳輸時現實傳輸過程 --log-format=formAT 指定日誌檔案格式 --password-file=FILE 從FILE中得到密碼 --bwlimit=KBPS 限制I/O頻寬,KBytes per second -h, --help 顯示幫助信息
模擬運行,顯示日誌,但不做複製操作。
rsync -anvzP /www/* root@172.16.0.1/www
速度限制,限製為 100k Bytes/s
rsync -auvzP--bwlimit=100 /www/* root@172.16.0.1/www
rsync -auzv --rsh=ssh root@202.130.101.33:/www/example.com/* /backup/example.com/ # --rsh=ssh 可以省略 rsync -auzv root@202.130.101.33:/www/example.com/* /backup/example.com/
如果需要特別參數,可以這樣寫,這裡指定連接SSH的連接埠為20
rsync -auzv --rsh='ssh -p20' root@202.130.101.34:/www/example.com/* /backup/example.com/
transfer file from src to dest directory
neo@netkiller:/tmp$ mkdir rsync neo@netkiller:/tmp$ cd rsync/ neo@netkiller:/tmp/rsync$ ls neo@netkiller:/tmp/rsync$ mkdir src dest neo@netkiller:/tmp/rsync$ echo file1 > src/file1 neo@netkiller:/tmp/rsync$ echo file2 > src/file2 neo@netkiller:/tmp/rsync$ echo file3 > src/file3
skipping directory
neo@netkiller:/tmp/rsync$ mkdir src/dir1 neo@netkiller:/tmp/rsync$ mkdir src/dir2 neo@netkiller:/tmp/rsync$ rsync src/* dest/ skipping directory src/dir1 skipping directory src/dir2
recurse into directories
neo@netkiller:/tmp/rsync$ rsync -r src/* dest/ neo@netkiller:/tmp/rsync$ ls dest/ dir1 dir2 file1 file2 file3
backup
neo@netkiller:/tmp/rsync$ rsync -r --backup --suffix=.2008-11-21 src/* dest/ neo@netkiller:/tmp/rsync$ ls dest/ dir1 dir2 file1 file1.2008-11-21 file2 file2.2008-11-21 file3 file3.2008-11-21 neo@netkiller:/tmp/rsync$
backup-dir
neo@netkiller:/tmp/rsync$ rsync -r --backup --suffix=.2008-11-21 --backup-dir mybackup src/* dest/ neo@netkiller:/tmp/rsync$ ls dest/ dir1 dir2 file1 file1.2008-11-21 file2 file2.2008-11-21 file3 file3.2008-11-21 mybackup neo@netkiller:/tmp/rsync$ ls dest/mybackup/ file1.2008-11-21 file2.2008-11-21 file3.2008-11-21
rsync -r --backup --suffix=.2008-11-21 --backup-dir ../mybackup src/* dest/ neo@netkiller:/tmp/rsync$ ls dest mybackup src neo@netkiller:/tmp/rsync$ ls src/ dir1 dir2 file1 file2 file3
update
neo@netkiller:/tmp/rsync$ rm -rf dest/* neo@netkiller:/tmp/rsync$ rsync -r -u src/* dest/ neo@netkiller:/tmp/rsync$ echo netkiller>>src/file2 neo@netkiller:/tmp/rsync$ rsync -v -r -u src/* dest/ building file list ... done file2 sent 166 bytes received 42 bytes 416.00 bytes/sec total size is 38 speedup is 0.18
update by time and size
neo@netkiller:/tmp/rsync$ echo Hi>src/dir1/file1.1 neo@netkiller:/tmp/rsync$ rsync -v -r -u src/* dest/ building file list ... done dir1/file1.1 sent 166 bytes received 42 bytes 416.00 bytes/sec total size is 41 speedup is 0.20
--archive
rsync -a src/ dest/
--compress
rsync -a -z src/ dest/
--delete
src
svn@netkiller:~$ ls src/ dir1 dir2 file1 file2 file3
dest
neo@netkiller:~$ rsync -v -u -a --delete -e ssh svnroot@127.0.0.1:/home/svnroot/src /tmp/dest svnroot@127.0.0.1's password: receiving file list ... done created directory /tmp/dest src/ src/file1 src/file2 src/file3 src/dir1/ src/dir2/ sent 104 bytes received 309 bytes 118.00 bytes/sec total size is 0 speedup is 0.00
src
svn@netkiller:~$ rm -rf src/file2 svn@netkiller:~$ rm -rf src/dir2
dest
neo@netkiller:~$ rsync -v -u -a --delete -e ssh svnroot@127.0.0.1:/home/svnroot/src /tmp/dest svnroot@127.0.0.1's password: receiving file list ... done deleting src/dir2/ deleting src/file2 src/ sent 26 bytes received 144 bytes 68.00 bytes/sec total size is 0 speedup is 0.00
http://samba.anu.edu.au/rsync/examples.html
$ rsync -v -u -a --delete --rsh=ssh --stats localfile username@hostname:/home/username/
for example:
I want to copy local workspace of eclipse directory to another computer.
$ rsync -v -u -a --delete --rsh=ssh --stats workspace neo@192.168.245.131:/home/neo/
$ rsync -v -u -a --delete --rsh=ssh --stats neo@192.168.245.131:/home/neo/* /tmp/
rsync使用方法
rsync rsync://認證用戶@主機/模組
rsync -vzrtopg --progress --delete 認證用戶@主機::模組 /mirror目錄
例 70.1. examples
用rsync刪除目標目錄
mkdir /root/blank
rsync --delete-before -a -H -v --progress --stats /root/blank/ ./cache/
例 70.2. backup to a central backup server with 7 day incremental
#!/bin/sh # This script does personal backups to a rsync backup server. You will end up # with a 7 day rotating incremental backup. The incrementals will go # into subdirectories named after the day of the week, and the current # full backup goes into a directory called "current" # tridge@linuxcare.com # directory to backup BDIR=/home/$USER # excludes file - this contains a wildcard pattern per line of files to exclude EXCLUDES=$HOME/cron/excludes # the name of the backup machine BSERVER=owl # your password on the backup server export RSYNC_PASSWORD=XXXXXX ######################################################################## BACKUPDIR=`date +%A` OPTS="--force --ignore-errors --delete-excluded --exclude-from=$EXCLUDES --delete --backup --backup-dir=/$BACKUPDIR -a" export PATH=$PATH:/bin:/usr/bin:/usr/local/bin # the following line clears the last weeks incremental directory [ -d $HOME/emptydir ] || mkdir $HOME/emptydir rsync --delete -a $HOME/emptydir/ $BSERVER::$USER/$BACKUPDIR/ rmdir $HOME/emptydir # now the actual transfer rsync $OPTS $BDIR $BSERVER::$USER/current
例 70.3. backup to a spare disk
I do local backups on several of my machines using rsync. I have an extra disk installed that can hold all the contents of the main disk. I then have a nightly cron job that backs up the main disk to the backup. This is the script I use on one of those machines. #!/bin/sh export PATH=/usr/local/bin:/usr/bin:/bin LIST="rootfs usr data data2" for d in $LIST; do mount /backup/$d rsync -ax --exclude fstab --delete /$d/ /backup/$d/ umount /backup/$d done DAY=`date "+%A"` rsync -a --delete /usr/local/apache /data2/backups/$DAY rsync -a --delete /data/solid /data2/backups/$DAY The first part does the backup on the spare disk. The second part backs up the critical parts to daily directories. I also backup the critical parts using a rsync over ssh to a remote machine.
例 70.4. mirroring vger CVS tree
The vger.rutgers.edu cvs tree is mirrored onto cvs.samba.org via anonymous rsync using the following script. #!/bin/bash cd /var/www/cvs/vger/ PATH=/usr/local/bin:/usr/freeware/bin:/usr/bin:/bin RUN=`lps x | grep rsync | grep -v grep | wc -l` if [ "$RUN" -gt 0 ]; then echo already running exit 1 fi rsync -az vger.rutgers.edu::cvs/CVSROOT/ChangeLog $HOME/ChangeLog sum1=`sum $HOME/ChangeLog` sum2=`sum /var/www/cvs/vger/CVSROOT/ChangeLog` if [ "$sum1" = "$sum2" ]; then echo nothing to do exit 0 fi rsync -az --delete --force vger.rutgers.edu::cvs/ /var/www/cvs/vger/ exit 0 Note in particular the initial rsync of the ChangeLog to determine if anything has changed. This could be omitted but it would mean that the rsyncd on vger would have to build a complete listing of the cvs area at each run. As most of the time nothing will have changed I wanted to save the time on vger by only doing a full rsync if the ChangeLog has changed. This helped quite a lot because vger is low on memory and generally quite heavily loaded, so doing a listing on such a large tree every hour would have been excessive.
例 70.5. automated backup at home
I use rsync to backup my wifes home directory across a modem link each night. The cron job looks like this #!/bin/sh cd ~susan { echo date dest=~/backup/`date +%A` mkdir $dest.new find . -xdev -type f \( -mtime 0 -or -mtime 1 \) -exec cp -aPv "{}" $dest.new \; cnt=`find $dest.new -type f | wc -l` if [ $cnt -gt 0 ]; then rm -rf $dest mv $dest.new $dest fi rm -rf $dest.new rsync -Cavze ssh . samba:backup } >> ~/backup/backup.log 2>&1 note that most of this script isn't anything to do with rsync, it just creates a daily backup of Susans work in a ~susan/backup/ directory so she can retrieve any version from the last week. The last line does the rsync of her directory across the modem link to the host samba. Note that I am using the -C option which allows me to add entries to .cvsignore for stuff that doesn't need to be backed up.
例 70.6. Fancy footwork with remote file lists
One little known feature of rsync is the fact that when run over a remote shell (such as rsh or ssh) you can give any shell command as the remote file list. The shell command is expanded by your remote shell before rsync is called. For example, see if you can work out what this does: rsync -avR remote:'`find /home -name "*.[ch]"`' /tmp/ note that that is backquotes enclosed by quotes (some browsers don't show that correctly).
#!/usr/bin/perl my $path = "/data"; #本地目錄 my $ip="172.16.xxx.xxx"; #遠程目錄 my $maxchild=5; #同時並發的個數 open FILE,"ls $path|"; while() { chomp; my $filename = $_; my $i = 1; while($i<=1){ my $un = `ps -ef |grep rsync|grep -v grep |grep avl|wc -l`; $i =$i+1; if( $un < $maxchild){ system("rsync -avl --size-only $path/$_ $ip:$path &") ; }else{ sleep 5; $i = 1; } } }