Home | Mirror | Search

2. rsync - fast remote file copy program (like rcp)

rsync is an open source utility that provides fast incremental file transfer. rsync is freely available under the GNU General Public License version 2 and is currently being maintained by Wayne Davison.

2.1. 安裝Rsync與配置守護進程

2.1.1. install with source

過程 73.1. rsync

  1. 安裝rsync

    在AS3 第二張CD上找到rsync-2.5.6-20.i386.rpm

    [root@linuxas3 root]# cd /mnt
    [root@linuxas3 mnt]# mount cdrom
    [root@linuxas3 mnt]# cd cdrom/RedHat/RPMS
    [root@linuxas3 RPMS]# rpm -ivh rsync-2.5.6-20.i386.rpm
    			
  2. 配置/etc/rsyncd.conf

    在rh9,as3系統上rsync安裝後,並沒有創建rsyncd.conf文檔,要自己創建rsyncd.conf文檔

    [root@linuxas3 root]# vi /etc/rsyncd.conf
    
    uid=nobody
    gid=nobody
    max connections=5
    use chroot=no
    log file=/var/log/rsyncd.log
    pid file=/var/run/rsyncd.pid
    lock file=/var/run/rsyncd.lock
    #auth users=root
    secrets file=/etc/rsyncd.passwd
    
    [postfix]
    path=/var/mail
    comment = backup mail
    ignore errors
    read only = yes
    list = no
    auth users = postfix
    
    [netkiller]
    path=/home/netkiller/web
    comment = backup 9812.net
    ignore errors
    read only = yes
    list = no
    auth users = netkiller
    
    [pgsqldb]
    path=/var/lib/pgsql
    comment = backup postgresql database
    ignore errors
    read only = yes
    list = no
    			
    1. 選項說明

      uid = nobody
      gid = nobody
      use chroot = no         # 不使用chroot
      max connections = 4     # 最大連接數為4
      pid file = /var/run/rsyncd.pid           #進程ID檔案
      lock file = /var/run/rsync.lock
      log file = /var/log/rsyncd.log    # 日誌記錄檔案
      secrets file = /etc/rsyncd.pwd    # 認證檔案名,主要保存用戶密碼,權限建議設為600,所有者root
      
      [module]            # 這裡是認證的模組名,在client端需要指定
      path = /var/mail    # 需要做鏡像的目錄
      comment = backup xxxx # 註釋
      ignore errors         # 可以忽略一些無關的IO錯誤
      read only = yes       # 只讀
      list = no             # 不允許列檔案
      auth users = postfix  # 認證的用戶名,如果沒有這行,則表明是匿名
      
      [other]
      path = /path/to...
      comment = xxxxx
      		    	
    2. 密碼檔案

      在server端生成一個密碼檔案/etc/rsyncd.pwd

      [root@linuxas3 root]# echo postfix:xxx >>/etc/rsyncd.pwd
      [root@linuxas3 root]# echo netkiller:xxx >>/etc/rsyncd.pwd
      [root@linuxas3 root]# chmod 600 /etc/rsyncd.pwd
      		    	
    3. 啟動rsync daemon

      [root@linuxas3 root]# rsync --daemon
      		    	
  3. 添加到啟動檔案

    echo "rsync --daemon" >> /etc/rc.d/rc.local                                    [  OK  ]
    			

    cat /etc/rc.d/rc.local 確認一下

  4. 測試

    [root@linux docbook]#  rsync rsync://netkiller.8800.org/netkiller
    [root@linux tmp]# rsync rsync://netkiller@netkiller.8800.org/netkiller
    Password:
    
    [chen@linux temp]$  rsync -vzrtopg --progress --delete postfix@netkiller.8800.org::postfix /tmp
    Password:
    			

2.1.2. install with aptitude

過程 73.2. installation setp by setp

  1. installation

    $ sudo apt-get install rsync
    			
  2. enable

    $ sudo vim /etc/default/rsync
    
    RSYNC_ENABLE=true
    			
  3. config /etc/rsyncd.conf

    $ sudo vim /etc/rsyncd.conf
    
    uid=nobody
    gid=nobody
    max connections=5
    use chroot=no
    pid file=/var/run/rsyncd.pid
    lock file=/var/run/rsyncd.lock
    log file=/var/log/rsyncd.log
    #auth users=root
    secrets file=/etc/rsyncd.secrets
    
    [neo]
    path=/home/neo/www
    comment = backup neo
    ignore errors
    read only = yes
    list = no
    auth users = neo
    
    [netkiller]
    path=/home/netkiller/public_html
    comment = backup netkiller
    ignore errors
    read only = yes
    list = no
    auth users = netkiller
    
    [mirror]
    path=/var/www/netkiller.8800.org/html/
    comment = mirror netkiller.8800.org
    exclude = .svn
    ignore errors
    read only = yes
    list = yes
    
    [music]
    path=/var/music
    comment = backup music database
    ignore errors
    read only = yes
    list = no
    
    [pgsqldb]
    path=/var/lib/pgsql
    comment = backup postgresql database
    ignore errors
    read only = yes
    list = no
    auth users = neo,netkiller
    			
  4. /etc/rsyncd.secrets

    $ sudo vim  /etc/rsyncd.secrets
    
    neo:123456
    netkiller:123456
    			

    $ sudo chmod 600 /etc/rsyncd.secrets
    			
  5. start

    $ sudo /etc/init.d/rsync start
    			
  6. test

    $ rsync -vzrtopg --progress --delete neo@localhost::neo /tmp/test1/
    $ rsync -vzrtopg --progress --delete localhost::music /tmp/test2/
    			
  7. firewall

    $ sudo ufw allow rsync
    			

2.1.3. xinetd

yum install xinetd
		

配置 /etc/xinetd.d/rsync

vim /etc/xinetd.d/rsync

# default: off
# description: The rsync server is a good addition to an ftp server, as it \
#	allows crc checksumming etc.
service rsync
{
	disable	= yes
	flags		= IPv6
	socket_type     = stream
	wait            = no
	user            = root
	server          = /usr/bin/rsync
	server_args     = --daemon
	log_on_failure  += USERID
}
		

disable = yes 改為 disable = no

# vim /etc/rsyncd.conf

chkconfig xinetd on
/etc/init.d/xinetd restart
		

2.2. rsyncd.conf

# Minimal configuration file for rsync daemon
# See rsync(1) and rsyncd.conf(5) man pages for help

# This line is required by the /etc/init.d/rsyncd script
pid file = /var/run/rsyncd.pid
port = 873
address = 192.168.1.171
#uid = nobody
#gid = nobody
uid = root
gid = root

use chroot = yes
read only = yes


#limit access to private LANs
hosts allow=192.168.1.0/255.255.255.0 10.0.1.0/255.255.255.0
hosts deny=*

max connections = 5
motd file = /etc/rsyncd/rsyncd.motd

#This will give you a separate log file
#log file = /var/log/rsync.log

#This will log every file transferred - up to 85,000+ per user, per sync
#transfer logging = yes

log format = %t %a %m %f %b
syslog facility = local3
timeout = 300

[home]
path = /home
list=yes
ignore errors
auth users = linux
secrets file = /etc/rsyncd/rsyncd.secrets
comment = linuxsir home
exclude =   beinan/  samba/

[beinan]
path = /opt
list=no
ignore errors
comment = optdir
auth users = beinan
secrets file = /etc/rsyncd/rsyncd.secrets

[www]
path = /www/
ignore errors
read only = true
list = false
hosts allow = 172.16.1.1
hosts deny = 0.0.0.0/32
auth users = backup
secrets file = /etc/backserver.pas


[web_user1]
path = /home/web_user1/
ignore errors
read only = true
list = false
hosts allow = 202.99.11.121
hosts deny = 0.0.0.0/32
uid = web_user1
gid = web_user1
auth users = backup
secrets file = /etc/backserver.pas


[pub]
  	comment = Random things available for download
  	path = /path/to/my/public/share
  	read only = yes
  	list = yes
  	uid = nobody
  	gid = nobody
  	auth users = pub
  	secrets file = /etc/rsyncd.secrets
		

2.3. upload

$ rsync -v -u -a --delete --rsh=ssh --stats localfile username@hostname:/home/username/
		

for example:

I want to copy local workspace of eclipse directory to another computer.

$ rsync -v -u -a --delete --rsh=ssh --stats workspace neo@192.168.245.131:/home/neo/
		

2.4. download

$ rsync -v -u -a --delete --rsh=ssh --stats neo@192.168.245.131:/home/neo/* /tmp/
		

2.5. mirror

rsync使用方法

rsync rsync://認證用戶@主機/模組

rsync -vzrtopg --progress --delete 認證用戶@主機::模組 /mirror目錄
		

2.6. step by step to learn rsync

  1. transfer file from src to dest directory

    neo@netkiller:/tmp$ mkdir rsync
    neo@netkiller:/tmp$ cd rsync/
    neo@netkiller:/tmp/rsync$ ls
    neo@netkiller:/tmp/rsync$ mkdir src dest
    neo@netkiller:/tmp/rsync$ echo file1 > src/file1
    neo@netkiller:/tmp/rsync$ echo file2 > src/file2
    neo@netkiller:/tmp/rsync$ echo file3 > src/file3
    				
  2. skipping directory

    neo@netkiller:/tmp/rsync$ mkdir src/dir1
    neo@netkiller:/tmp/rsync$ mkdir src/dir2
    neo@netkiller:/tmp/rsync$ rsync src/* dest/
    skipping directory src/dir1
    skipping directory src/dir2
    				
  3. recurse into directories

    neo@netkiller:/tmp/rsync$ rsync -r src/* dest/
    neo@netkiller:/tmp/rsync$ ls dest/
    dir1  dir2  file1  file2  file3
    				
  4. backup

    neo@netkiller:/tmp/rsync$ rsync -r --backup --suffix=.2008-11-21 src/* dest/
    neo@netkiller:/tmp/rsync$ ls dest/
    dir1  dir2  file1  file1.2008-11-21  file2  file2.2008-11-21  file3  file3.2008-11-21
    neo@netkiller:/tmp/rsync$
    				

    backup-dir

    neo@netkiller:/tmp/rsync$ rsync -r --backup --suffix=.2008-11-21 --backup-dir mybackup src/* dest/
    neo@netkiller:/tmp/rsync$ ls dest/
    dir1  dir2  file1  file1.2008-11-21  file2  file2.2008-11-21  file3  file3.2008-11-21  mybackup
    neo@netkiller:/tmp/rsync$ ls dest/mybackup/
    file1.2008-11-21  file2.2008-11-21  file3.2008-11-21
    				

    rsync -r --backup --suffix=.2008-11-21 --backup-dir ../mybackup src/* dest/
    neo@netkiller:/tmp/rsync$ ls
    dest  mybackup  src
    neo@netkiller:/tmp/rsync$ ls src/
    dir1  dir2  file1  file2  file3
    				
  5. update

    neo@netkiller:/tmp/rsync$ rm -rf dest/*
    neo@netkiller:/tmp/rsync$ rsync -r -u src/* dest/
    neo@netkiller:/tmp/rsync$ echo netkiller>>src/file2
    neo@netkiller:/tmp/rsync$ rsync -v -r -u src/* dest/
    building file list ... done
    file2
    
    sent 166 bytes  received 42 bytes  416.00 bytes/sec
    total size is 38  speedup is 0.18
    				

    update by time and size

    neo@netkiller:/tmp/rsync$ echo Hi>src/dir1/file1.1
    neo@netkiller:/tmp/rsync$ rsync -v -r -u src/* dest/
    building file list ... done
    dir1/file1.1
    
    sent 166 bytes  received 42 bytes  416.00 bytes/sec
    total size is 41  speedup is 0.20
    				
  6. --archive

    rsync -a src/ dest/
    				
  7. --compress

    rsync -a -z src/ dest/
    				
  8. --delete

    src

    svn@netkiller:~$ ls src/
    dir1  dir2  file1  file2  file3
    				

    dest

    neo@netkiller:~$ rsync -v -u -a --delete -e ssh svnroot@127.0.0.1:/home/svnroot/src /tmp/dest
    svnroot@127.0.0.1's password:
    receiving file list ... done
    created directory /tmp/dest
    src/
    src/file1
    src/file2
    src/file3
    src/dir1/
    src/dir2/
    
    sent 104 bytes  received 309 bytes  118.00 bytes/sec
    total size is 0  speedup is 0.00
    				

    src

    svn@netkiller:~$ rm -rf src/file2
    svn@netkiller:~$ rm -rf src/dir2
    				

    dest

    neo@netkiller:~$ rsync -v -u -a --delete -e ssh svnroot@127.0.0.1:/home/svnroot/src /tmp/dest
    svnroot@127.0.0.1's password:
    receiving file list ... done
    deleting src/dir2/
    deleting src/file2
    src/
    
    sent 26 bytes  received 144 bytes  68.00 bytes/sec
    total size is 0  speedup is 0.00
    				

2.7. rsync examples

http://samba.anu.edu.au/rsync/examples.html

2.7.1. rsync delete

例 73.1. examples

用rsync刪除目標目錄



mkdir /root/blank
rsync --delete-before -a -H -v --progress --stats /root/blank/ ./cache/


2.7.2. backup to a central backup server with 7 day incremental

例 73.2. backup to a central backup server with 7 day incremental

				
#!/bin/sh

# This script does personal backups to a rsync backup server. You will end up
# with a 7 day rotating incremental backup. The incrementals will go
# into subdirectories named after the day of the week, and the current
# full backup goes into a directory called "current"
# tridge@linuxcare.com

# directory to backup
BDIR=/home/$USER

# excludes file - this contains a wildcard pattern per line of files to exclude
EXCLUDES=$HOME/cron/excludes

# the name of the backup machine
BSERVER=owl

# your password on the backup server
export RSYNC_PASSWORD=XXXXXX


########################################################################

BACKUPDIR=`date +%A`
OPTS="--force --ignore-errors --delete-excluded --exclude-from=$EXCLUDES
      --delete --backup --backup-dir=/$BACKUPDIR -a"

export PATH=$PATH:/bin:/usr/bin:/usr/local/bin

# the following line clears the last weeks incremental directory
[ -d $HOME/emptydir ] || mkdir $HOME/emptydir
rsync --delete -a $HOME/emptydir/ $BSERVER::$USER/$BACKUPDIR/
rmdir $HOME/emptydir

# now the actual transfer
rsync $OPTS $BDIR $BSERVER::$USER/current
				
				

2.7.3. backup to a spare disk

例 73.3. backup to a spare disk

				
I do local backups on several of my machines using rsync. I have an
extra disk installed that can hold all the contents of the main
disk. I then have a nightly cron job that backs up the main disk to
the backup. This is the script I use on one of those machines.

    #!/bin/sh

    export PATH=/usr/local/bin:/usr/bin:/bin

    LIST="rootfs usr data data2"

    for d in $LIST; do
	mount /backup/$d
	rsync -ax --exclude fstab --delete /$d/ /backup/$d/
	umount /backup/$d
    done

    DAY=`date "+%A"`

    rsync -a --delete /usr/local/apache /data2/backups/$DAY
    rsync -a --delete /data/solid /data2/backups/$DAY



The first part does the backup on the spare disk. The second part
backs up the critical parts to daily directories.  I also backup the
critical parts using a rsync over ssh to a remote machine.
				
				

2.7.4. mirroring vger CVS tree

例 73.4. mirroring vger CVS tree

				
The vger.rutgers.edu cvs tree is mirrored onto cvs.samba.org via
anonymous rsync using the following script.

    #!/bin/bash

    cd /var/www/cvs/vger/
    PATH=/usr/local/bin:/usr/freeware/bin:/usr/bin:/bin

    RUN=`lps x | grep rsync | grep -v grep | wc -l`
    if [ "$RUN" -gt 0 ]; then
	    echo already running
	    exit 1
    fi

    rsync -az vger.rutgers.edu::cvs/CVSROOT/ChangeLog $HOME/ChangeLog

    sum1=`sum $HOME/ChangeLog`
    sum2=`sum /var/www/cvs/vger/CVSROOT/ChangeLog`

    if [ "$sum1" = "$sum2" ]; then
	    echo nothing to do
	    exit 0
    fi

    rsync -az --delete --force vger.rutgers.edu::cvs/ /var/www/cvs/vger/
    exit 0

Note in particular the initial rsync of the ChangeLog to determine if
anything has changed. This could be omitted but it would mean that the
rsyncd on vger would have to build a complete listing of the cvs area
at each run. As most of the time nothing will have changed I wanted to
save the time on vger by only doing a full rsync if the ChangeLog has
changed. This helped quite a lot because vger is low on memory and
generally quite heavily loaded, so doing a listing on such a large
tree every hour would have been excessive.
				
				

2.7.5. automated backup at home

例 73.5. automated backup at home

				
I use rsync to backup my wifes home directory across a modem link each
night. The cron job looks like this

    #!/bin/sh
    cd ~susan
    {
    echo
    date
    dest=~/backup/`date +%A`
    mkdir $dest.new
    find . -xdev -type f \( -mtime 0 -or -mtime 1 \) -exec cp -aPv "{}"
    $dest.new \;
    cnt=`find $dest.new -type f | wc -l`
    if [ $cnt -gt 0 ]; then
      rm -rf $dest
      mv $dest.new $dest
    fi
    rm -rf $dest.new
    rsync -Cavze ssh . samba:backup
    } >> ~/backup/backup.log 2>&1


note that most of this script isn't anything to do with rsync, it just
creates a daily backup of Susans work in a ~susan/backup/ directory so
she can retrieve any version from the last week. The last line does
the rsync of her directory across the modem link to the host
samba. Note that I am using the -C option which allows me to add
entries to .cvsignore for stuff that doesn't need to be backed up.
				
				

2.7.6. Fancy footwork with remote file lists

例 73.6. Fancy footwork with remote file lists

				
One little known feature of rsync is the fact that when run over a
remote shell (such as rsh or ssh) you can give any shell command as
the remote file list. The shell command is expanded by your remote
shell before rsync is called. For example, see if you can work out
what this does:

	rsync -avR remote:'`find /home -name "*.[ch]"`' /tmp/

note that that is backquotes enclosed by quotes (some browsers don't
show that correctly).
				
				

2.8. rsync for windows

http://www.rsync.net/resources/howto/windows_rsync.html

2.9. 多進程 rsync 腳本

		
#!/usr/bin/perl

my $path = "/data";          #本地目錄
my $ip="172.16.xxx.xxx";     #遠程目錄
my $maxchild=5;              #同時並發的個數

open FILE,"ls $path|";
while()
{

        chomp;
        my $filename = $_;
        my $i = 1;
        while($i<=1){
                my $un = `ps -ef |grep rsync|grep -v grep |grep avl|wc -l`;
                $i =$i+1;
                if( $un < $maxchild){
                        system("rsync -avl --size-only $path/$_   $ip:$path &") ;
                }else{
                        sleep 5;
                        $i = 1;
                }
        }
}
		
		

2.10. 數度限制

限製為 100k Bytes/s

rsync -auvzP--bwlimit=100 /www/* root@172.16.0.1/www
		
comments powered by Disqus