Difference between revisions of "Backup Server"

From IridiaWiki
Jump to navigationJump to search
 
(19 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
= IRIDIA Backup System =
[[Image:Phd101705s.gif||]]
 
   
  +
== Personal Backups ==
IRIDIA has a server where users can store their backups. It can be configured to automatically retrieve the data to backup from users' computers during the night.
 
   
  +
Everyone has 20GB available space on fallopius. You are responsible for backup up your own data into the space you have allocated on fallopius.
== Server Information ==
 
   
 
=== Push Backup ===
The server name is <tt>fallopius.ulb.ac.be</tt> and it is in IRIDIA. It will be moved to another building. When? Hard to say...
 
   
  +
Backups are initiated from the client machine ("pushed" onto the server).
You need a valid login to access the computer. Ask the system adminstrato to create one for you.
 
   
  +
You can use you personal favourite to push your backups on the server. Please make sure that your data does not exceed the limit (i.e. purge old data regularly).
The server has globally 677 Gb available for backup. The HDs are configured according to RAID 5 architecture to assure more realiability and not to lose also your backups!
 
   
  +
Examples of good backup programs are rdiff-backup, rsync, hdup, unison and many others. See below for a list .
== How to Access the Server ==
 
   
  +
=== Pull Backup ===
The server can be accessed only via SSH.
 
   
  +
Backups are initiated from the backup server ("pulled").
== Quota ==
 
   
  +
==== Halva's old system ====
The server is the only machine in IRIDIA which sets quota for the users. Each user has a soft quota of 20&nbsp;Gb, and a hard quota of 22&nbsp;Gb. When the soft quota is exceeded, the user has 7 days to reduce its occupied space. After 7 days, or in any case if the hard quota is exceeded, the user account is blocked and the only thing he can do is to remove files.
 
   
  +
There is an old automated backup system that is still in place, but no longer supported. It was written by Halva and uses rdiff-backup. A script pulls the contents of the user's home directory onto the backup server. Instructions here: [[Old Backup System (Halva's rdiff system)]]. Good luck.
   
 
== Server Backup ==
The system administrator can modify the quota of a user with the command:
 
   
  +
=== iridia.ulb.ac.be ===
<tt>edquota -u username</tt>
 
   
  +
All data on the iridia server is backed up by a separate (supported) nightly script that uses dar (see below). This is an incremental backup program that is like a network version of tar. The scripts are run by the crontab of the user servicebackup on iridia using the script /home/servicebackup/iridia_service_backup_system/backup.sh. The shell functions that do the meat of the work are included from the file include.sh in the same directory.
editing the values of the soft and hard limits.
 
   
  +
The backed up data is stored in the directory are stored on fallopius in the directory /home/iridiaserver/service_backup_dar.
== Manual Backup ==
 
  +
To restore data you will need to familiarise yourself with the dar program, or use the script /home/servicebackup/iridia_service_backup_system/restore.sh on iridia.
   
  +
The data that is backed up is divided into 'services'. A list of these services can be found in the directory /home/servicebackup/iridia_service_backup_system/services. Each service is a directory that must at a minimum contain scripts called <SERVICE_NAME>_backup.sh and <SERVICE_NAME>_restore.sh. These two scripts should make use of the functions provided by the file /home/servicebackup/iridia_service_backup_system/include.sh which provide wrappers around the dar commands. Optionally, inside each service directory there can be scripts called <SERVICE_NAME>_suspend.sh and <SERVICE_NAME>_resume.sh which will be called before and after the backup, and can be used to stop and start any processes that might interfere with the integrity of the backed up data if they are left running.
Using <tt>ssh</tt> and <tt>scp</tt>, each user can copy his data on <tt>fallopius</tt>. Other programs that might be used are [http://www.nongnu.org/duplicity/ duplicity] and [http://www.nongnu.org/rdiff-backup/ rdiff-backup].
 
   
  +
A mail gets sent every day to a list of addresses defined in the backup.sh script with the success / failure of the backup in the subject line.
For example you might use the rsync command like this:
 
   
  +
===== Troubleshooting =====
<tt>time rsync -a -z -e ssh "/home/user/" user@fallopius.ulb.ac.be:/home/user/remote/ --delete --progress --update</tt>
 
   
  +
Logfiles for each backup are stored in /home/servicebackup/iridia_service_backup_system/log on iridia.ulb.ac.be
[http://developer.apple.com/macosx/backuponmacosx.html Command-line backup solutions on Mac OS X]
 
   
  +
A quick way of debugging the system is to run the script /home/servicebackup/iridia_service_backup_system/just_test_backup.sh . This script does a test backup of a small directory and outputs to the logfile test.log.
== Automatic Backup ==
 
   
  +
There is an ugly soft link from /usr/lib/libdar.so.3.0.1 -> /home/iridiaserver/dar/lib/libdar.so.3.0.1 This link has been known to disappear after an upgrade of the system.
The server can be configure to automatically retrieve the data to backup from users' computers. This procedure is described more in detail in the README file that each user finds in his home directory at the first login. Please read that file first, and if you have more doubt ask the system administrator.
 
   
  +
===== svn =====
Once the setup of an account is completed, the server proceeds to the automatic retrieval of the data each evening at 22:00 on odd days (the 1st, 3rd, 5th, and so on). At the end of the procedure, an email is sent to the owner of the accunt with the result of the backup.
 
   
  +
The scripting system is maintained and versioned in the svn repository:
=== Mac OS X and rsync ===
 
  +
file:///usr/local/share/svn_repositories/iridia_service_backup_system
[http://lartmaker.nl/rsync/ Mini-HOWTO: Fixing rsync on Tiger (Mac OS X 10.4.x)]
 
   
=== How to remove old incremental data ===
 
   
  +
== Backup Software ==
To remove old data just log in fallopius.ulb.ac.be and give a command similar to the following:
 
   
 
=== dar ===
<tt>rdiff-backup --force --remove-older-than 2005-12-31 mmanfrin@iridia.ulb.ac.be</tt>
 
   
  +
DAR (http://dar.sourceforge.net/) is an incremental backup program that is like a network version of tar. It is used by the iridia server.
(in this example we remove the incremental backups of the machine iridia.ulb.ac.be for the account mmanfrin previous to 31 Dec 2005 )
 
   
  +
=== rdiff-backup ===
== Where Does the Name Come from? ==
 
   
  +
The current supported version is 1.1.17. Follow the steps below for Debian etch:
In IRIDIA we are used to name the computer after famous scientits. We looked for someone who invented something that might be related with the concept of "backup".
 
   
  +
$ wget http://iridia.ulb.ac.be/~abrutschy/rdiff-backup_1.1.17-1_etch_i386.deb
A backup is, after all, something that makes you safer, that lets you work without worries. It is a form of protection. In a book published in 1564, "De Morbo Gallico", Gabriello Fallopio (1523-1563), an Italian anatomist, describes a device he used to protect people against venereal illnesses quite common at that time. The device he refers to is the '''condom'''.
 
  +
$ sudo dpkg -i rdiff-backup_1.1.17-1_etch_i386.deb
   
  +
and for Ubuntu Intrepid/Debian Lenny:
Less worthy of note, there are also two organs in the female body named after him: the fallopian tubes.
 
   
  +
$ wget http://iridia.ulb.ac.be/~abrutschy/rdiff-backup_1.1.17-1_lenny_intrepid_i386.deb
[[Image:Phd103003s.gif||]]
 
  +
$ sudo dpkg -i rdiff-backup_1.1.17-1_lenny_intrepid_i386.deb
  +
  +
and for Ubuntu Jaunty:
  +
  +
$ wget http://iridia.ulb.ac.be/~abrutschy/rdiff-backup_1.1.17-1_jaunty_i386.deb
  +
$ sudo dpkg -i rdiff-backup_1.1.17-1_jaunty_i386.deb
  +
  +
In case you use another system, download and install from the source ( http://iridia.ulb.ac.be/wiki/images/1/18/Rdiff-backup_1.1.17.orig.tar.gz or http://download.savannah.gnu.org/releases/rdiff-backup/rdiff-backup-1.1.17.tar.gz). Note that it could not work if your system has a version that is different).
  +
  +
Add the following lines to "/etc/apt/preferences" in order to prevent apt to install another version of rdiff-backup:
  +
  +
Package: rdiff-backup
  +
Pin: version 1.1.17-1
  +
Pin-Priority: 1001
  +
  +
===== Clean up some of your used space =====
  +
  +
Imagining that you don't need the entire history of your backup, you can use rdiff to delete files that are older than a specified amount of time.
  +
  +
/usr/bin/rdiff-backup -v0 --force --remove-older-than 3W <fallopiususername>@fallopius::/home/<fallopiususername>/backup/<localusername>@<localmachine>.ulb.ac.be/
  +
  +
Please use once in a while
  +
  +
=== Unison ===
  +
  +
If you know unison, note that you can also use that for your personal backups. Installed version is 2.27.57. For documentation look here http://www.cis.upenn.edu/~bcpierce/unison/.
  +
  +
== Info for Administrators ==
  +
  +
Here you find some [[Information for administrators]].

Latest revision as of 18:49, 14 February 2012

IRIDIA Backup System

Personal Backups

Everyone has 20GB available space on fallopius. You are responsible for backup up your own data into the space you have allocated on fallopius.

Push Backup

Backups are initiated from the client machine ("pushed" onto the server).

You can use you personal favourite to push your backups on the server. Please make sure that your data does not exceed the limit (i.e. purge old data regularly).

Examples of good backup programs are rdiff-backup, rsync, hdup, unison and many others. See below for a list .

Pull Backup

Backups are initiated from the backup server ("pulled").

Halva's old system

There is an old automated backup system that is still in place, but no longer supported. It was written by Halva and uses rdiff-backup. A script pulls the contents of the user's home directory onto the backup server. Instructions here: Old Backup System (Halva's rdiff system). Good luck.

Server Backup

iridia.ulb.ac.be

All data on the iridia server is backed up by a separate (supported) nightly script that uses dar (see below). This is an incremental backup program that is like a network version of tar. The scripts are run by the crontab of the user servicebackup on iridia using the script /home/servicebackup/iridia_service_backup_system/backup.sh. The shell functions that do the meat of the work are included from the file include.sh in the same directory.

The backed up data is stored in the directory are stored on fallopius in the directory /home/iridiaserver/service_backup_dar. To restore data you will need to familiarise yourself with the dar program, or use the script /home/servicebackup/iridia_service_backup_system/restore.sh on iridia.

The data that is backed up is divided into 'services'. A list of these services can be found in the directory /home/servicebackup/iridia_service_backup_system/services. Each service is a directory that must at a minimum contain scripts called <SERVICE_NAME>_backup.sh and <SERVICE_NAME>_restore.sh. These two scripts should make use of the functions provided by the file /home/servicebackup/iridia_service_backup_system/include.sh which provide wrappers around the dar commands. Optionally, inside each service directory there can be scripts called <SERVICE_NAME>_suspend.sh and <SERVICE_NAME>_resume.sh which will be called before and after the backup, and can be used to stop and start any processes that might interfere with the integrity of the backed up data if they are left running.

A mail gets sent every day to a list of addresses defined in the backup.sh script with the success / failure of the backup in the subject line.

Troubleshooting

Logfiles for each backup are stored in /home/servicebackup/iridia_service_backup_system/log on iridia.ulb.ac.be

A quick way of debugging the system is to run the script /home/servicebackup/iridia_service_backup_system/just_test_backup.sh . This script does a test backup of a small directory and outputs to the logfile test.log.

There is an ugly soft link from /usr/lib/libdar.so.3.0.1 -> /home/iridiaserver/dar/lib/libdar.so.3.0.1 This link has been known to disappear after an upgrade of the system.

svn

The scripting system is maintained and versioned in the svn repository: file:///usr/local/share/svn_repositories/iridia_service_backup_system


Backup Software

dar

DAR (http://dar.sourceforge.net/) is an incremental backup program that is like a network version of tar. It is used by the iridia server.

rdiff-backup

The current supported version is 1.1.17. Follow the steps below for Debian etch:

 $ wget http://iridia.ulb.ac.be/~abrutschy/rdiff-backup_1.1.17-1_etch_i386.deb
 $ sudo dpkg -i rdiff-backup_1.1.17-1_etch_i386.deb

and for Ubuntu Intrepid/Debian Lenny:

 $ wget http://iridia.ulb.ac.be/~abrutschy/rdiff-backup_1.1.17-1_lenny_intrepid_i386.deb
 $ sudo dpkg -i rdiff-backup_1.1.17-1_lenny_intrepid_i386.deb

and for Ubuntu Jaunty:

 $ wget http://iridia.ulb.ac.be/~abrutschy/rdiff-backup_1.1.17-1_jaunty_i386.deb
 $ sudo dpkg -i rdiff-backup_1.1.17-1_jaunty_i386.deb

In case you use another system, download and install from the source ( http://iridia.ulb.ac.be/wiki/images/1/18/Rdiff-backup_1.1.17.orig.tar.gz or http://download.savannah.gnu.org/releases/rdiff-backup/rdiff-backup-1.1.17.tar.gz). Note that it could not work if your system has a version that is different).

Add the following lines to "/etc/apt/preferences" in order to prevent apt to install another version of rdiff-backup:

 Package: rdiff-backup
 Pin: version 1.1.17-1
 Pin-Priority: 1001
Clean up some of your used space

Imagining that you don't need the entire history of your backup, you can use rdiff to delete files that are older than a specified amount of time.

 /usr/bin/rdiff-backup -v0 --force --remove-older-than 3W <fallopiususername>@fallopius::/home/<fallopiususername>/backup/<localusername>@<localmachine>.ulb.ac.be/

Please use once in a while

Unison

If you know unison, note that you can also use that for your personal backups. Installed version is 2.27.57. For documentation look here http://www.cis.upenn.edu/~bcpierce/unison/.

Info for Administrators

Here you find some Information for administrators.