Friday, August 22, 2008

How to backup your stuff UNIX-style

Foreword:

This "How to" here involves a lot of shell commands. And you can't just copy & paste them 1:1 ... chances are your system is different from mine, so you need to think before you copy my commands.

People looking for a nice "Point and click" tutorial should move on. There is no pointing and clicking here. Simply because I don't believe in "pointing and clicking" and I also don't know any reliable GUI backup method ... "reliable" in the sense: We can get our stuff back after a disaster. So for a "GUI point and click backup How-To" I am simply the wrong person.

So we're going to use the shell here. And then plenty of it. So why should you be interested?

1. You could learn a lot.
2. You could learn a lot.
3. You could learn even more.
4. You could gain some confidence with the shell and your new Linux OS

5. This tutorial here works, GUI or not.

You can and you will get your data back if you do it right. But again: THINK. No tutorial can replace your brain and your intelligence. And did I mention that you could learn a lot ... ? :wink:

IMPORTANT: All the commands here need to be executed as super-user "root". So either put a "sudo" in front of each command that I mention or open a root shell via "sudo su -" and execute the commands directly in there.

Also: I strongly advise against using UUID's in /etc/fstab . UUID's make matters unnecessarily complicated by being "unique". So this means that if something goes wrong and you have to restore from a backup onto a new disk the UUID's will be different ... and that will cause problems. So it would be better to get rid of the UUID's in /etc/fstab right away (e.g. right after you installed your system) and use the more traditional device names in /etc/fstab ... Example:
Code: Select all
# /etc/fstab: static file system information.
#
#
/dev/hda1 / ext3 rw,errors=remount-ro 0 1
/dev/hda5 none swap rw 0 0
none /proc proc defaults 0 0
/dev/hda9 /data ext3 rw 0 2
/dev/hda8 /home ext3 rw 0 2
/dev/hda6 /tmp ext3 rw 0 2
/dev/hda7 /var ext3 rw 0 2
/dev/hdb1 /data2 ext3 rw 0 2
Device names are "dumb" and they don't care if they are on your old harddisk or a new one ... for as long as the harddisk and its partitions exist your system will boot. With UUID's: Nope. You'd have to find out each UUID for each disk (costs you a lot of time) and those UUID's are usually so long and so cryptic that there is a high chance that you will screw up. As I said: Get rid of them, change your /etc/fstab so that it uses device names!

Edit: Also, don't forget to edit /boot/grub/menu.lst !! There is a section which defines the default kernel options, e.g. mine looks like this:
Code: Select all
# kopt=root=UUID=8499b6f9-2196-4134-bf75-0599c17658e2 ro
Leave it commented out (that's perfectly OK; the setting *IS* *ACTIVE* nontheless! Also: the documentation says not to uncomment it, so I guess we just leave it like that!) but change that rear part! So if you change /etc/fstab and rid yourself of those UUID entries you also have to do it here. So you'd replace above string "kopt=root=UUID....." with the device name for your " / " mount point. In my case that would be:
Code: Select all
kopt=root=/dev/sda5 ro
Once you've changed this setting you need to activate it. Execute this as "root": update-grub .... It will rewrite your GRUB and make sure all entries in GRUB point to the right " / " partition. Or else those entries might still point to your old UUID= entry ... and that could cause booting to stop unexpectedly. If that happens (e.g. you forgot to edit menu.lst): You can temporarily "edit" the boot entry in GRUB's menu at boot time. Just make sure the menu gets displayed (e.g. by pressing 'ESC' if it's not shown per default) and then select the entry you wish to temporarily edit and then hit "e" (it's all displayed on-screen). Check that kernel line ... if it says "root=UUID....." you can change it to "root=/dev/sda5" (whatever the right setting for your " / " partition is) and then hit the 'ENTER' key to boot your system with the temporarily altered settings. It should work. Once you're in you can still change menu.lst and do the needed changes.


On the shoulders of giants

I of course am not the "inventor" of those backup methods ... Duh. As Einstein once put it: "I am standing on the shoulder of giants". Very huge giants at that. So credits go to the people who taught me UNIX and Linux back in the 90's, to all my bosses and superior managers who kicked my a** if I did stuff wrong (very important learning experience ...), to the people who ridiculed me on IRC back when I was a "n00b" (important learning experiences too ...). Last but not least, this book taught me a lot:

UNIX Backup & Recovery

I highly recommend it despite its age (1999). So nothing here is any "big secret". All the stuff shown here are standard UNIX shell commands that are most likely present out of the box on any UNIX-like operating system. The "big secret" (if it even is one) is to put these commands to work and have them backup our system.


Preparations: What do we use?

First things first: Every backup is only as good as the restore. And my work as UNIX admin has taught me some nasty lessons.

In a professional environment you'd use some form of tapes to write your backups to. Simply because they are very reliable and can hold large amounts of data.

But most home users probably can't afford a 4000$ DLT drive or a 10'000$ tape robot. So what's left?

I personally am not really a fan of making backups to CD's or DVD's. CD's are usually too small and if you span your backups across multiple CD's chances are you might lose one disk of this volume set ... and then your restore won't work anymore and your backup is worth nothing. This has happened to me. DVD's are bigger, yes. But both CD's and DVD's have serious drawbacks when it comes to reliability. Take a DVD+RW disk for example: "Can survive up to 1000 rewrites" they say. Right. What they don't tell you: Those 1000 rewrites are reached way faster than you think. If you want reliability don't count on DVD's ...

So I personally have settled on large external USB harddisks:
- they are cheap if you compare their price per GB with other options.
- these days it's really hard to find a machine that doesn't have USB.
- hence: USB harddisks most likely will work anywhere + anytime.
- reliability: I only turn on those disks when I really really want to use them for backups ... this greately increases their lifespan.
- speed: Let's face it: No DVD or CD could ever hope to beat a harddisk when it comes to read/write speeds.


As for software:

All software that we would need is already there in your OS. The only other thing I would recommend is to have a Linux Live CD ready, either your Mint Installer disk, or Ubuntu Live, or Knoppix, or whatever, for as long as it has:

- USB support
- tar on-board
- gzip on-board

you should already have all the software you'd need. As an extra and "just to be safe" I'd highly recommend the SuperGrub Disk ... Please use Google to find it. It's a live CD that is specialised on getting boot blocks back (hence the name and the relation with the "GRUB" boot loader).


Backup: What do we backup?

In short: Everything that matters. I'm a rather paranoid person and bad experience has taught me to really backup everything + the kitchen sink. It's better to have too much stuff in your backup and not need it than the other way round: Need too much stuff but not have it in any backup. That sucks. And it can seriously ruin your day :twisted:

So really -- what do we backup?

1. the Master Boot Record (MBR) + the partition table
2. we make an exact copy of /etc so we have a copy of all the config files that matter
3. we backup our package selections (can be used for cloning purposes too)
4. we pack all our partitions into tar.gz archives
5. except /home: we make a 1:1 copy of it so we can access all the files if needed
6. Putting it all together



Step 1: The Master Boot Record

For this we need the dd utility. If you want to learn more about dd please read the manual: man dd

In this example I will assume that /dev/hda is the first harddisk (for SATA or SCSI users this would most likely be /dev/sda ... as I said: THINK ... you can't just copy & paste here without thinking!).

We tell dd to read the relevant number of bytes from the first harddisk device (yes, really: the device --- and *not* any partition like /dev/hda1 or /dev/hda2 or anything like that ... ) and put that into a binary file ... if anything goes wrong we could restore the MBR from there (e.g. with the help of a Live CD):
Code: Select all
dd if=/dev/hda of=/tmp/hda-mbr.bin bs=512 count=1
Explanation: dd will read the first 512 bytes (= MBR + partition table) from the first harddisk /dev/hda and place it into the file I specify ... here: /tmp/hda-mbr.bin ... that path and filename could be adjusted to your taste and requirements if needed. Obviously it would be intelligent to have this file e.g. on an external disk or USB stick ... just in case.

How to restore your MBR

If you've destroyed your MBR somehow than this means that your PC won't boot anymore from the harddisks ... so to get it back to work you'd need a Live CD now. So in order to get our MBR back (there are other methods too but I will now focus on this one):
- you'd boot your Live CD
- mount that USB stick or USB harddisk where you put that MBR-backup on to
- open a terminal
- cd into the right directory
- restore the MBR with the "dd" command

Code: Select all
sudo su -
cd /path/to/where/MBR/backup/is/stored
dd if=hda-mbr.bin of=/dev/hda bs=446 count=1
Please note the difference here: We're only restoring 446 bytes ... not the full 512 we backed up earlier. Why? The last 64 bytes contain the partition table. Chances are you don't want to ruin your disk partitions so it's wiser to only restore 446 bytes ... not the full 512.

Please be advised that brainless tampering with the dd command while being "root" can lead to serious data loss and a really ruin your day.

Just think before you hit that enter key. If in doubt: Please come to the IRC channel and/or the forum and ask, OK?

How do we know it worked? dd itself won't give you much info. The only way to tell if it did work is to reboot. Ideally your GRUB menu should reappear again ... :D


Step 2: Copying /etc

That's simple. Just find a good location and then as "root" recursively copy your /etc there ... this has the advantage that if you need to ask for help or if anything goes wrong you have a copy of all your config files. Example:
Code: Select all
sudo su -
cd /where/we/will/store/our/stuff
cp -a -v -u /etc .
Explanation: "cp -a -v -u" copies everything in /etc recursively to the current location (= "." as target) plus it keeps the permissions intact.

How to restore your /etc

Simple: just check what's missing and then copy the single config files back to the /etc of your system. Let's suppose you've ruined your xorg.conf file somehow:
Code: Select all
sudo su -
cd /where/we/placed/the/backup/etc
cd X11
cp xorg.conf /etc/X11/
It's simple, it's stupid ... and it works.


Step 3: Our package selections

1. Backup /etc/apt/sources.list ... or copy it to one of your other systems (in case you want to have the same packages everywhere)

2. On system #1 (the one you want to backup) issue this command:
sudo dpkg --get-selections > packagelist.txt

=> This will write your package selections and the status of all packages into that text file "packagelist.txt". Now copy that file to your other system(s) or put it onto a USB stick or USB harddisk.

How to restore your package selections

1. You need to restore the /etc/apt/sources.list file (see section above about copying /etc recursively ...) and then you need to tell dpkg to read your package list:
sudo dpkg --set-selections <>

=> The system will now take over the package selections that were previously defined on the original or previous system.

2. All you need to do now is to execute the install process:
sudo apt-get dselect-upgrade

=> Result: You now will have the same packages installed like before on "System #1"

"Armed" with these commands you can also clone installations (e.g. produce identical systems with identical software selections) in a couple of minutes. Just make sure you get the /etc/apt/sources.list right (e.g. it should be the same on all systems).


Step 4: Packing your partitions into *.tar.gz archives

Again, you need to think a little and take a look at your /etc/fstab ... it doesn't make sense if you copy & paste my commands here, especially if you don't have the same mountpoints (e.g. separate /usr, /var and /opt partitions ... or not). The sense behind those commands is that you get an archived copy of all your partitions. The advantages you get:

- you could 100% restore a system into the state it had when the archives were made
- you could use the archives to make a 1:1 clone of your system
- you could also choose to only restore selected parts of your system (useful if you e.g. switch distros and e.g. don't need the /usr partition anymore from the previous distro ...)

This example here assumes that you have a " / " root filesystem and separate /boot, /var, /usr and /home filesystems (this needs to be changed according to your system settings):
Code: Select all
sudo su -
cd /path/to/USB/disk
tar --one-file-system -pzcvf BootFS_backup.tar.gz /boot
tar --one-file-system --exclude=/tmp/* --exclude=/dev/* --exclude=/proc/* -pzcvf RootFS_backup.tar.gz /
tar --one-file-system --exclude=/var/tmp/* -pzcvf VarFS_backup.tar.gz /var
tar --one-file-system -pzcvf UsrFS_backup.tar.gz /usr
tar --one-file-system -pzcvf HomeFS_backup.tar.gz /home
The result here would be that all these archive files get written to the path you specified above (here: "/path/to/USB/disk") ... and if that would be your USB disk you could in the future also fully restore your system from there again.

How to restore your partitions via the archives

So let's suppose that due to some serious misfortune your harddisk went to Nirvana -- you were forced to replace it. In such a case you could either re-install everything from scratch or you could restore the archives -- I mean that's precisely what we have them for, right?

So, first step would obviously be to remove the dead disk from the system and put in the new one. I will assume you know how to do it ... if not please Google around. It should be easy to find guides that will tell you how to replace a dead harddisk.

When this is done you now have a shiny new harddisk in your system. And it's totally empty and there is nothing on it .... yet.

So what we need now is to boot the system with a Live CD. The Mint installer disk should suffice, but an Ubuntu, Kubuntu or Knoppix Live CD should do just as fine. What we need to do now is to re-create the partitions as they were on the old disk -- chances are that your Live CD will have some form of partitioning program, either gparted for the GUI or fdisk for the command line. Also: These days chances are that a new harddisk will be bigger than a previous one, so don't worry about the sizes: It's OK if you make the partitions on the new harddisk bigger than they used to be on the old system. They don't have to be absolutely the same size. Just bear in mind that you can't restore e.g. a 4 GB /usr partition if you make the new one only half as big ...

IMPORTANT: What really is important here is that you create the same amount of partitions and in the same order as they used to be on the old harddisk. So if you used to have a /dev/hda5, /dev/hda6, and /dev/hda7 on your old disk then it would be wise to have that too on your new system. If you don't do this then it is imperative that you remember the new disk setup and edit your /etc/fstab before you reboot!

Also: I seriously hope you followed my advice above and got rid of the UUID's in your /etc/fstab ... when it comes to backup and especially when it comes to a restore those UUID's can be a serious pain.

So ... back on track:

- You get rid of the old harddisk, install the new one into your system
- you boot the Live CD
- you partition your new harddisk so that it resembles the old disk's partitioning scheme
- you format each of the new partitions, e.g. sudo mkfs -t ext3 /dev/hdaX
- you attach your USB harddisk (where our backups are) to the Live system
- the live system should have noticed the presence of your USB disk and it should have opened it
- you open a terminal and cd into the right directory on the USB disk so that you can access the archives

Once you got that far i'ts time for some thinking and planning. We are about to restore gigabytes and gigabytes of data ... so it better be the right disk and stuff be in the right order.

First, we need to check: What will our root partition " / " be? ==> Here Step #2 comes in handy. If you did copy your /etc as suggested above you could just open a second terminal and cd into the directory where you copied your /etc into and simply take a look at your old /etc/fstab ...

So let's assume that your " / " filesystem was on /dev/hda5 ... As this will be the " / " filesystem again we should recreate that one first.

Into a new terminal, type this (needs to be adjusted to reflect your own system settings):
Code: Select all
sudo su -
cd /mnt
mkdir new-root
mount /dev/hda5 /mnt/new-root
If everything went right these steps should finish without error.

If you check the contents of your "new-root" chances are that it is empty; there will likely be a "lost+found" directory but that's it.

So as a first step we need to recreate the various mount points again!
Code: Select all
mkdir /boot /usr /opt /home /var
Once that is done we can mount all sub-sequent partitions and attach it to our "new-root". So I will make some assumptions here, e.g. that /dev/hda6 is your /usr, /dev/hda7 is your /var, and so on. This needs to be adjusted, obviously:
Code: Select all
cd /mnt/new-root
mount /dev/hda1 boot
mount /dev/hda6 usr
mount /dev/hda7 var
mount /dev/hda8 opt
mount /dev/hda9 home
With this we have recreated the structures of our harddisk ... so now we just need to fill the disk again!

There should still be an open terminal window where you cd'd to the archives, right? Into that type this command: pwd
This should spit out the current path, e.g. maybe something like /media/USBdisk/backups/archives ... Highlight it with your mouse (so that it gets copied into the clipboard).

In your other terminal window inside the "new-root" directory it's time for unpacking. Let's start with /boot ... being the smallest partition it's an excellent candidate for doing tests as it won't take too long to find out if it worked or not (if you don't have a separate /boot you can skip this ... scroll down where we restore the " / " filesystem).

Type this: tar -pzxvf and now hit your middle mouse button ... It should have inserted the path above from the clipboard into your command line so you don't have to type this path more than unnecessary. All that you need to do now is to add the name of the archive and then hit the enter key. So in the end it should look something like this:
Code: Select all
cd /mnt/new-root
tar -pzxvf /media/USBdisk/where/the/backups/are/BootFS_Backup.tar.gz
If everything worked the contents of your original /boot should have been restored to your current "new-root/boot" directory. If you do a "ls -al /mnt/new-root/boot" you should be able to see the files.

So if this works we can restore the rest too, e.g. the root filesystem " / ":
Code: Select all
cd /mnt/new-root
tar -pzxvf /media/USBdisk/where/the/backups/are/RootFilesystem_Backup.tar.gz
tar -pzxvf /media/USBdisk/where/the/backups/are/UsrFilesystem_Backup.tar.gz
tar -pzxvf /media/USBdisk/where/the/backups/are/VarFilesystem_Backup.tar.gz
tar -pzxvf /media/USBdisk/where/the/backups/are/OptFilesystem_Backup.tar.gz
tar -pzxvf /media/USBdisk/where/the/backups/are/HomeFilesystem_Backup.tar.gz
Please note that if you preferred not to make a tar.gz of your home filesystem and instead used the "cp" command then you'd obviously restore your home folder as described in that section (Step 5: Restoring your /home).

If everything went as it should have you should have restored the content of your partitions to your new disk ... or your new system ... or a clone.

What's still missing is the boot loader. You could try restoring that one via restoring the MBR or with a tool like the Super Grub-Disk (please use Google ... it's a live CD specialised on restoring dead boot loaders).

Once everything is done it's time to umount and reboot the system .... which hopefully will reboot as if nothing had happened.
Code: Select all
cd /mnt/new-root
sync
umount boot
umount usr
umount var
umount opt
cd /mnt
umount new-root
If everything is OK there should be no errors. What could happen is that you get an error message about "mount point busy" so that a partition can't be unmounted. Chances are you still got a terminal open which is accessing something on that partition? Close everything and try again. Even if you still get errors it would probably safe to reboot now. Ideally your system should now come up and boot as if nothing had happened ... If it doesn't it could be that the boot block is still not 100% OK, so it would be a good idea to e.g. use the SuperGrub Disk and have the boot block rewritten again. But after that it should really work (I do the absolutely same thing here and I therefore know that this should work!).


Step 5: Copying /home

In the example above we packed /home into a *.tar.gz archive. That may or may not be a good idea. If you got tons of movies and music files in your /home directory then further compressing /home won't do no good ... it will just be a hell of a slow operation and take forever. In such circumstances it would be better to just make a copy onto an external harddrive as we did with /etc:
Code: Select all
sudo su -
cd /where/we/will/store/our/stuff
cp -a -v -u /home .
... This is basically the same command as above.

How to restore /home

In case something went wrong or if you had to reinstall your PC you can most likely just copy all the contents of /home back onto your new installation. Most stuff like your desktop settings, e-mails, bookmarks etc. should be tip top preserved as if nothing had happened. Example:
Code: Select all
sudo su -
cd /path/to/where/we/backed/up/home
cp -a -v -u yourusername /home/
Please mind the syntax. The position of those slashes "/" is very important. If everything went right and the copy process finishes then all your files should be back in your /home directory and when you login everything should be in its place as if nothing had happened. I did that already several times and I can almost guarantee that if you execute the commands right you have zero data loss ... you can switch distros, uninstall, reinstall, experiment around and if done right you will hopefully never lose your files.


BONUS -- Step 6: Putting it all together

So this is a copy of my own personal backup script ... yes, no joking, I really use this monster and it does its job well. What it does:

- all of the above
- creates sub-directories per date
- which means I can go back to specific dates and get my system the way it was that day (provided I did run a backup then)
- hopefully it's not too hard to adapt it to your needs.
- due to some limitations in my knowledge and/or bash some parts of the script repeat over and over again ... sorry about that, but I haven't yet found a way around that.

Important: The script assumes that you are already "root" and that you are already on your USB drive and in the correct location where you want to store your backup, e.g. somewhere in e.g. /media/USBdrive/backup/wherever ...

Here we go:

Code: Select all
#! /bin/bash

export OSNAME="Mint_3.0"
export TODAYSBACKUPDATE=`date +%d-%b-%Y`
export HOSTNAME=`hostname`
export RELEASE=`uname -r`

mkdir $TODAYSBACKUPDATE
cd $TODAYSBACKUPDATE
dd if=/dev/hda of=$HOSTNAME"_hda-MBR_"$TODAYSBACKUPDATE".backup" bs=512 count=1

cp -a -v -u /etc .
cp -a -v -u /home .
sync

export MOUNTPOINT="BootFS"
export ARCHIVEFILENAME=$HOSTNAME"_"$OSNAME"_Kernel_"$RELEASE"_"$TODAYSBACKUPDATE"_"$MOUNTPOINT".tar.gz"
tar --one-file-system -pzcvf $ARCHIVEFILENAME /boot
sync

export MOUNTPOINT="RootFS"
export ARCHIVEFILENAME=$HOSTNAME"_"$OSNAME"_Kernel_"$RELEASE"_"$TODAYSBACKUPDATE"_"$MOUNTPOINT".tar.gz"
tar --one-file-system --exclude=/tmp/* --exclude=/proc/* --exclude=/dev/* --exclude=/sys/* -pzcvf $ARCHIVEFILENAME /
sync

export MOUNTPOINT="UsrFS"
export ARCHIVEFILENAME=$HOSTNAME"_"$OSNAME"_Kernel_"$RELEASE"_"$TODAYSBACKUPDATE"_"$MOUNTPOINT".tar.gz"
tar --one-file-system -pzcvf $ARCHIVEFILENAME /usr
sync

export MOUNTPOINT="OptFS"
export ARCHIVEFILENAME=$HOSTNAME"_"$OSNAME"_Kernel_"$RELEASE"_"$TODAYSBACKUPDATE"_"$MOUNTPOINT".tar.gz"
tar --one-file-system -pzcvf $ARCHIVEFILENAME /opt
sync

export MOUNTPOINT="VarFS"
export ARCHIVEFILENAME=$HOSTNAME"_"$OSNAME"_Kernel_"$RELEASE"_"$TODAYSBACKUPDATE"_"$MOUNTPOINT".tar.gz"
tar --one-file-system --exclude=/var/tmp/* -pzcvf $ARCHIVEFILENAME /var
sync

dpkg --get-selections > $HOSTNAME"_dpkg-PackageSelection.txt"
sync

unset MOUNTPOINT
unset ARCHIVEFILENAME



That's it from here. Hope this was useful ... :D
.
.
.
.
.
- typos corrected! + section about "menu.lst" and UUID's added.
- bad, bad, bad typo removed. :(

No comments: