IBS Tips -- Pulling a backup

Sometimes, you want to retain a particular backup longer than the usual rotation would imply. On a tape backup system, you simply pull the tape(s) that hold the data you wish to retain. On IBS, simply rename the desired directory to something "other" than the normal YYYY-MM-DD(-ME) format -- I normally use pull-YYYY-MM-DD. However, experience in the corporate world has indicated some kind of hint as to WHY a particular backup has been pulled and how long it should be retained. We used a ticket number -- T123456-pull-2022-08-03, for example. Alternatively, create a file in the backup directory explaining the why and when for pulls. Just do something, you won't remember in a week why you pulled a backup, and in a big enviroment, you might have multiple reasons for multiple pulls.

IBS Tips -- fixing problems

Backup failures

Since each IBS backup is based on the one before it, making sure each backup is "good" is important. If a backup fails half-way through a backup, the next backup would consider ALL the missing files "new" files, and have to copy them over, even though there is an existing copy on the next-to previous backup. This creates a bit of a "Bump" in disk storage requirements.

Unfortunately, a bump like this will persist until all the "pre-bump" backups are rotated out or until virtually all the files are replaced through regular changes. In a system with a lot of month-end backups, it could take many months.

As always, first thing to do is try to figure out why the backup failed.

Moving a backup from one volume to another

Sometimes, you may find you have several backup volumes, but one is getting tight on space, while others are fine. The easiest way to move a backup tree from one storage volume to another is rsync it over, using the -H option so hardlinks are retained.

Migrating to new storage

If your existing storage needs to be replaced due to capacity or aging out, but is still functional, an easy way to migrate would be to bolt on the new storage, create new mount points, copy over your most recent backups to the new volume and a bunch of empty ME and daily backups, and a new symlink pointing to the new storage volume. After that, you can just let the old backups "age out". After your ME rotation has completed, you can just drop your old volumes...done!

The downside to this process is your backups are a bit messy for historic research purposes, since some backups are on one volume, others on another.

The other way is to move one backup at a time from the old storage to the new storage, probably using rsync with the -H option to retain the symlinks. When a backup is moved over, change the symlink in the BUBASE directory. The problem with this process is it is very slow, it can take many days, so generally you want to move a few machines at a time from the old storage to the new storage. You don't want to have a backup run against an incomplete copy here, and you don't want to restart the rsync if you can help it.

Migrating to all new hardware

In this case, I usually prefer to set up the new system, maybe seed it with the most recent backups from the old system, then run them both in parallel for a while, and let the old system age out. And for a while, you got two backups instead of one. However, copying historic backups out of the old system into the new system with an rsync -H will work, too, moving the machines that are backed up to the new system as their old data is copied over.

One complication in moving to new hardware is shuffling the data appropriately. For example, at a company where we deployed this, we replaced a four volume system with a five volume system. In this case, worst-case would be simply copy 1->1, etc, then move individual backups over to the new volume. What we did was to just break up the entire list of servers into five groups, start 'em up, and look for space issues. The report function gives some idea of the individual backup sizes, but unfortunately it really gives only a hint at the size of the overall backup size.

Addiing additional storage

While many OSs and file systems support adding additional space to existing volumes and file systems, I prefer not to have storage span devices, and I don't like to have individual volumes be too huge. So, if you are running tight on space, my preferred solution is to add more storage as a new volume, and move a few existing systems from each "tight on space" volume over to the new storage.

IBS Tips -- getting fancy!

As stated at the beginning, IBS makes USEFUL backups, not just inert lumps you can't do anything with. Here are some examples of what kinds of queries you can answer with IBS.

Backing up the IBS server to tape

You may feel you want to put your IBS backups to tape. This ... should be done carefully. The IBS backup file systems are a mess of hard links, and a lot of commercial backup software handles that somewhere between oddly and wrong. If this is your plan, I'd suggest running the IBS backups, then pushing the most recent backups only to another file system/directory/server, and let your tape backup system handle just that. Otherwise, you may discover restoring data from the IBS backups put to tape becomes very convoluted (i.e., you might have to restore ALL the backups for a particular machine to get a complete copy of the most recent backup). Regardless, do test restores in realistic settings. This is true for ANY backup system.


Holland Consulting home Page
Contact Holland Consulting

since February 20, 2022

Page Copyright 2022, Nick Holland, Holland Consulting.