Difference between revisions of "Backup Server"
| (19 intermediate revisions by 4 users not shown) | |||
| Line 1: | Line 1: | ||
| ⚫ | |||
| − | [[Image:Phd101705s.gif||]] |
||
| + | == Personal Backups == |
||
| − | IRIDIA has a server where users can store their backups. It can be configured to automatically retrieve the data to backup from users' computers during the night. |
||
| + | Everyone has 20GB available space on fallopius. You are responsible for backup up your own data into the space you have allocated on fallopius. |
||
| ⚫ | |||
| ⚫ | |||
| − | The server name is <tt>fallopius.ulb.ac.be</tt> and it is in IRIDIA. It will be moved to another building. When? Hard to say... |
||
| + | Backups are initiated from the client machine ("pushed" onto the server). |
||
| − | You need a valid login to access the computer. Ask the system adminstrato to create one for you. |
||
| + | You can use you personal favourite to push your backups on the server. Please make sure that your data does not exceed the limit (i.e. purge old data regularly). |
||
| − | The server has globally 677 Gb available for backup. The HDs are configured according to RAID 5 architecture to assure more realiability and not to lose also your backups! |
||
| + | Examples of good backup programs are rdiff-backup, rsync, hdup, unison and many others. See below for a list . |
||
| − | == How to Access the Server == |
||
| + | === Pull Backup === |
||
| − | The server can be accessed only via SSH. |
||
| + | Backups are initiated from the backup server ("pulled"). |
||
| ⚫ | |||
| + | ==== Halva's old system ==== |
||
| − | The server is the only machine in IRIDIA which sets quota for the users. Each user has a soft quota of 20 Gb, and a hard quota of 22 Gb. When the soft quota is exceeded, the user has 7 days to reduce its occupied space. After 7 days, or in any case if the hard quota is exceeded, the user account is blocked and the only thing he can do is to remove files. |
||
| + | There is an old automated backup system that is still in place, but no longer supported. It was written by Halva and uses rdiff-backup. A script pulls the contents of the user's home directory onto the backup server. Instructions here: [[Old Backup System (Halva's rdiff system)]]. Good luck. |
||
| ⚫ | |||
| − | The system administrator can modify the quota of a user with the command: |
||
| + | === iridia.ulb.ac.be === |
||
| − | <tt>edquota -u username</tt> |
||
| + | All data on the iridia server is backed up by a separate (supported) nightly script that uses dar (see below). This is an incremental backup program that is like a network version of tar. The scripts are run by the crontab of the user servicebackup on iridia using the script /home/servicebackup/iridia_service_backup_system/backup.sh. The shell functions that do the meat of the work are included from the file include.sh in the same directory. |
||
| − | editing the values of the soft and hard limits. |
||
| + | The backed up data is stored in the directory are stored on fallopius in the directory /home/iridiaserver/service_backup_dar. |
||
| ⚫ | |||
| + | To restore data you will need to familiarise yourself with the dar program, or use the script /home/servicebackup/iridia_service_backup_system/restore.sh on iridia. |
||
| + | The data that is backed up is divided into 'services'. A list of these services can be found in the directory /home/servicebackup/iridia_service_backup_system/services. Each service is a directory that must at a minimum contain scripts called <SERVICE_NAME>_backup.sh and <SERVICE_NAME>_restore.sh. These two scripts should make use of the functions provided by the file /home/servicebackup/iridia_service_backup_system/include.sh which provide wrappers around the dar commands. Optionally, inside each service directory there can be scripts called <SERVICE_NAME>_suspend.sh and <SERVICE_NAME>_resume.sh which will be called before and after the backup, and can be used to stop and start any processes that might interfere with the integrity of the backed up data if they are left running. |
||
| − | Using <tt>ssh</tt> and <tt>scp</tt>, each user can copy his data on <tt>fallopius</tt>. Other programs that might be used are [http://www.nongnu.org/duplicity/ duplicity] and [http://www.nongnu.org/rdiff-backup/ rdiff-backup]. |
||
| + | A mail gets sent every day to a list of addresses defined in the backup.sh script with the success / failure of the backup in the subject line. |
||
| − | For example you might use the rsync command like this: |
||
| + | ===== Troubleshooting ===== |
||
| − | <tt>time rsync -a -z -e ssh "/home/user/" user@fallopius.ulb.ac.be:/home/user/remote/ --delete --progress --update</tt> |
||
| + | Logfiles for each backup are stored in /home/servicebackup/iridia_service_backup_system/log on iridia.ulb.ac.be |
||
| − | [http://developer.apple.com/macosx/backuponmacosx.html Command-line backup solutions on Mac OS X] |
||
| + | A quick way of debugging the system is to run the script /home/servicebackup/iridia_service_backup_system/just_test_backup.sh . This script does a test backup of a small directory and outputs to the logfile test.log. |
||
| ⚫ | |||
| + | There is an ugly soft link from /usr/lib/libdar.so.3.0.1 -> /home/iridiaserver/dar/lib/libdar.so.3.0.1 This link has been known to disappear after an upgrade of the system. |
||
| − | The server can be configure to automatically retrieve the data to backup from users' computers. This procedure is described more in detail in the README file that each user finds in his home directory at the first login. Please read that file first, and if you have more doubt ask the system administrator. |
||
| + | ===== svn ===== |
||
| − | Once the setup of an account is completed, the server proceeds to the automatic retrieval of the data each evening at 22:00 on odd days (the 1st, 3rd, 5th, and so on). At the end of the procedure, an email is sent to the owner of the accunt with the result of the backup. |
||
| + | The scripting system is maintained and versioned in the svn repository: |
||
| − | === Mac OS X and rsync === |
||
| + | file:///usr/local/share/svn_repositories/iridia_service_backup_system |
||
| − | [http://lartmaker.nl/rsync/ Mini-HOWTO: Fixing rsync on Tiger (Mac OS X 10.4.x)] |
||
| − | === How to remove old incremental data === |
||
| + | == Backup Software == |
||
| − | To remove old data just log in fallopius.ulb.ac.be and give a command similar to the following: |
||
| ⚫ | |||
| − | <tt>rdiff-backup --force --remove-older-than 2005-12-31 mmanfrin@iridia.ulb.ac.be</tt> |
||
| + | DAR (http://dar.sourceforge.net/) is an incremental backup program that is like a network version of tar. It is used by the iridia server. |
||
| − | (in this example we remove the incremental backups of the machine iridia.ulb.ac.be for the account mmanfrin previous to 31 Dec 2005 ) |
||
| + | === rdiff-backup === |
||
| − | == Where Does the Name Come from? == |
||
| + | The current supported version is 1.1.17. Follow the steps below for Debian etch: |
||
| − | In IRIDIA we are used to name the computer after famous scientits. We looked for someone who invented something that might be related with the concept of "backup". |
||
| + | $ wget http://iridia.ulb.ac.be/~abrutschy/rdiff-backup_1.1.17-1_etch_i386.deb |
||
| − | A backup is, after all, something that makes you safer, that lets you work without worries. It is a form of protection. In a book published in 1564, "De Morbo Gallico", Gabriello Fallopio (1523-1563), an Italian anatomist, describes a device he used to protect people against venereal illnesses quite common at that time. The device he refers to is the '''condom'''. |
||
| + | $ sudo dpkg -i rdiff-backup_1.1.17-1_etch_i386.deb |
||
| + | and for Ubuntu Intrepid/Debian Lenny: |
||
| − | Less worthy of note, there are also two organs in the female body named after him: the fallopian tubes. |
||
| + | $ wget http://iridia.ulb.ac.be/~abrutschy/rdiff-backup_1.1.17-1_lenny_intrepid_i386.deb |
||
| − | [[Image:Phd103003s.gif||]] |
||
| + | $ sudo dpkg -i rdiff-backup_1.1.17-1_lenny_intrepid_i386.deb |
||
| + | |||
| + | and for Ubuntu Jaunty: |
||
| + | |||
| + | $ wget http://iridia.ulb.ac.be/~abrutschy/rdiff-backup_1.1.17-1_jaunty_i386.deb |
||
| + | $ sudo dpkg -i rdiff-backup_1.1.17-1_jaunty_i386.deb |
||
| + | |||
| + | In case you use another system, download and install from the source ( http://iridia.ulb.ac.be/wiki/images/1/18/Rdiff-backup_1.1.17.orig.tar.gz or http://download.savannah.gnu.org/releases/rdiff-backup/rdiff-backup-1.1.17.tar.gz). Note that it could not work if your system has a version that is different). |
||
| + | |||
| + | Add the following lines to "/etc/apt/preferences" in order to prevent apt to install another version of rdiff-backup: |
||
| + | |||
| + | Package: rdiff-backup |
||
| + | Pin: version 1.1.17-1 |
||
| + | Pin-Priority: 1001 |
||
| + | |||
| + | ===== Clean up some of your used space ===== |
||
| + | |||
| + | Imagining that you don't need the entire history of your backup, you can use rdiff to delete files that are older than a specified amount of time. |
||
| + | |||
| + | /usr/bin/rdiff-backup -v0 --force --remove-older-than 3W <fallopiususername>@fallopius::/home/<fallopiususername>/backup/<localusername>@<localmachine>.ulb.ac.be/ |
||
| + | |||
| + | Please use once in a while |
||
| + | |||
| + | === Unison === |
||
| + | |||
| + | If you know unison, note that you can also use that for your personal backups. Installed version is 2.27.57. For documentation look here http://www.cis.upenn.edu/~bcpierce/unison/. |
||
| + | |||
| + | == Info for Administrators == |
||
| + | |||
| + | Here you find some [[Information for administrators]]. |
||
Latest revision as of 18:49, 14 February 2012
IRIDIA Backup System
Personal Backups
Everyone has 20GB available space on fallopius. You are responsible for backup up your own data into the space you have allocated on fallopius.
Push Backup
Backups are initiated from the client machine ("pushed" onto the server).
You can use you personal favourite to push your backups on the server. Please make sure that your data does not exceed the limit (i.e. purge old data regularly).
Examples of good backup programs are rdiff-backup, rsync, hdup, unison and many others. See below for a list .
Pull Backup
Backups are initiated from the backup server ("pulled").
Halva's old system
There is an old automated backup system that is still in place, but no longer supported. It was written by Halva and uses rdiff-backup. A script pulls the contents of the user's home directory onto the backup server. Instructions here: Old Backup System (Halva's rdiff system). Good luck.
Server Backup
iridia.ulb.ac.be
All data on the iridia server is backed up by a separate (supported) nightly script that uses dar (see below). This is an incremental backup program that is like a network version of tar. The scripts are run by the crontab of the user servicebackup on iridia using the script /home/servicebackup/iridia_service_backup_system/backup.sh. The shell functions that do the meat of the work are included from the file include.sh in the same directory.
The backed up data is stored in the directory are stored on fallopius in the directory /home/iridiaserver/service_backup_dar. To restore data you will need to familiarise yourself with the dar program, or use the script /home/servicebackup/iridia_service_backup_system/restore.sh on iridia.
The data that is backed up is divided into 'services'. A list of these services can be found in the directory /home/servicebackup/iridia_service_backup_system/services. Each service is a directory that must at a minimum contain scripts called <SERVICE_NAME>_backup.sh and <SERVICE_NAME>_restore.sh. These two scripts should make use of the functions provided by the file /home/servicebackup/iridia_service_backup_system/include.sh which provide wrappers around the dar commands. Optionally, inside each service directory there can be scripts called <SERVICE_NAME>_suspend.sh and <SERVICE_NAME>_resume.sh which will be called before and after the backup, and can be used to stop and start any processes that might interfere with the integrity of the backed up data if they are left running.
A mail gets sent every day to a list of addresses defined in the backup.sh script with the success / failure of the backup in the subject line.
Troubleshooting
Logfiles for each backup are stored in /home/servicebackup/iridia_service_backup_system/log on iridia.ulb.ac.be
A quick way of debugging the system is to run the script /home/servicebackup/iridia_service_backup_system/just_test_backup.sh . This script does a test backup of a small directory and outputs to the logfile test.log.
There is an ugly soft link from /usr/lib/libdar.so.3.0.1 -> /home/iridiaserver/dar/lib/libdar.so.3.0.1 This link has been known to disappear after an upgrade of the system.
svn
The scripting system is maintained and versioned in the svn repository: file:///usr/local/share/svn_repositories/iridia_service_backup_system
Backup Software
dar
DAR (http://dar.sourceforge.net/) is an incremental backup program that is like a network version of tar. It is used by the iridia server.
rdiff-backup
The current supported version is 1.1.17. Follow the steps below for Debian etch:
$ wget http://iridia.ulb.ac.be/~abrutschy/rdiff-backup_1.1.17-1_etch_i386.deb $ sudo dpkg -i rdiff-backup_1.1.17-1_etch_i386.deb
and for Ubuntu Intrepid/Debian Lenny:
$ wget http://iridia.ulb.ac.be/~abrutschy/rdiff-backup_1.1.17-1_lenny_intrepid_i386.deb $ sudo dpkg -i rdiff-backup_1.1.17-1_lenny_intrepid_i386.deb
and for Ubuntu Jaunty:
$ wget http://iridia.ulb.ac.be/~abrutschy/rdiff-backup_1.1.17-1_jaunty_i386.deb $ sudo dpkg -i rdiff-backup_1.1.17-1_jaunty_i386.deb
In case you use another system, download and install from the source ( http://iridia.ulb.ac.be/wiki/images/1/18/Rdiff-backup_1.1.17.orig.tar.gz or http://download.savannah.gnu.org/releases/rdiff-backup/rdiff-backup-1.1.17.tar.gz). Note that it could not work if your system has a version that is different).
Add the following lines to "/etc/apt/preferences" in order to prevent apt to install another version of rdiff-backup:
Package: rdiff-backup Pin: version 1.1.17-1 Pin-Priority: 1001
Clean up some of your used space
Imagining that you don't need the entire history of your backup, you can use rdiff to delete files that are older than a specified amount of time.
/usr/bin/rdiff-backup -v0 --force --remove-older-than 3W <fallopiususername>@fallopius::/home/<fallopiususername>/backup/<localusername>@<localmachine>.ulb.ac.be/
Please use once in a while
Unison
If you know unison, note that you can also use that for your personal backups. Installed version is 2.27.57. For documentation look here http://www.cis.upenn.edu/~bcpierce/unison/.
Info for Administrators
Here you find some Information for administrators.
