IBS Installation

Planning the system

OS Options

If you are installing a "large" operating system such as most modern Linux distributions and this machine will be used only for IBS backups, you can generally do the most basic, minimalist install your distribution supports. The features you need are SSH (included on everything these days), rsync, and perhaps a tool to pull new versions of IBS from my website (OpenBSD ftp(1) works great, Linux you might want curl or wget). I found a Rocky Linux "minimal system" install had everything I needed except for rsync, which is an easy install.

A standard OpenBSD install works well. X is not necessary, but easier to just install it, the full OpenBSD install will still be smaller than a minimal Linux install. On OpenBSD, you will have to add rsync.

Partitioning

For various reasons, I strongly suggest a highly "partitioned" disk system for an IBS server. At a minimum, the OS/application and the data should be on two separated chunks of disk. If running in a VM, I'd suggest two logical disks, so that OS upgrades and replacements can be done while still keeping the file system with the data on it intact. Otherwise, the OS and data may well live on one physical array, but still should have clear deliniatons between OS/app and backup data.

You want two directories hanging off root, one for all IBS backup volumes, the physical storage space, and one for $IBSROOT, which is where your system managers will be looking for data. The idea is ALL your backups should be visible in $IBSROOT, even though the individual backups are stored elsewhere. I use these:

/ibs : for $IBSROOT
/v : for the physical volumes.

I suggest these directories be the smallest separate file systems your OS can make on your disk. 1MB is plenty large, for all they will contain is symlinks and mount points. Having these be separate tiny file systems will make sure if you accidently drop real data into them, they will run out of space quickly to inform you of the error.

There are some mount options I would suggest using with your backup partitions:

noatime : atime (access time) serves no purpose here, and using noatime will reduce the amount of disk access your system needs to do. Honestly, this is probably true of most applications; atime is very rarely used.
nosuid : there's no reason to respect the suid bit in a backup, and if you back up a bit of malware from a system, this may help keep it from committing mischief on your backup server. It can still be backed up, it just can't run as suid on the backup.
nodev : again, no reason to have functioning device files in the backup.
noexec : While this is usually a token gesture security teams ofte request, in this case, it might help prevent the accidental execution of a program or script that was backed up.

If there are mount options that are helpful on your platform of choice, go for it. In the case of OpenBSD, I suggest "softdep", as it is a huge performance win on a system with a lot of disk activity -- in the case of the stuff IBS does, you may see a 2x-5x improvement or more in many things.

OpenBSD example

For a system I have here, my disk system consists of two 4TB drives and two 8TB drives. Each pair of drives is in a software RAID mirror. The 4TB drive pair is the OS and the first couple backup volumes, the 8TB drive pair is a second backup-only volume.

# df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/sd4a      554M    207M    320M    39%    /
/dev/sd4g     38.7G   20.0K   36.8G     0%    /tmp
/dev/sd4h      3.9G    1.6G    2.1G    42%    /home
/dev/sd4d      5.8G    1.8G    3.8G    32%    /usr
/dev/sd4e      7.8G    1.9G    5.4G    26%    /var
/dev/sd3o     61.4M    2.0K   58.3M     0%    /bu
/dev/sd3p     61.4M    8.0K   58.3M     0%    /v
/dev/sd4i      2.1T    1.9T    129G    94%    /v/1
/dev/sd5d      7.2T    5.5T    1.3T    81%    /v/3
/dev/sd5j      1.3T    330G    895G    27%    /v/2

I'm using a simplified version of the standard OpenBSD layout -- missing are the X partitions (while X is loaded, it's not used, and I don't see a benefit for separate X partitions here), there's no /usr/local directory. That's a bit risky -- OpenBSD enforces Write^Execute(1) in all code except for programs in the /usr/local directory, but an exception isn't required for IBS and the utilities it needs, and thus, I don't have a /usr/local partion. If you do other things with your IBS server, you will wish you had a /usr/local, but then, I really wouldn't recommend running software like that on a IBS server. There is also no "build from source" partitions here, because if I have reason to build OpenBSD from source, it won't be on this machine.

You may notice /tmp and /var are kinda big -- that's because OpenBSD includes a locate command which takes a lot of /tmp and /var space to rebuild its database on a big IBS system (remember: you might have the same file showing up in 20+ different locations!). It took me several iterations to get this to a good size. Lots of big files aren't as bad as lots of tiny files -- and I have a lot of backups of my mail server with a decade of e-mail on it.

As I always recommend, there's some unallocated space not shown here. In that way, if I later find out that I made something too small or I needed that /usr/local or other partition, I've got space available. In this case, I have over 150G unallocated, pretty much anything I need to change, I can change with that available.

Linux example

In this case, I have a 6G drive, did a minimal install, and partititoned the the disk this way:

[root@ibs-linux ~]# df -h -txfs -text4 -T
Filesystem              Type  Size  Used Avail Use% Mounted on
/dev/mapper/rl-root     xfs   2.0G   67M  2.0G   4% /
/dev/mapper/rl-usr      xfs   5.0G  1.1G  4.0G  21% /usr
/dev/sda2               xfs  1014M  326M  689M  33% /boot
/dev/mapper/rl-var      xfs   5.0G  166M  4.9G   4% /var
/dev/mapper/rl-tmp      xfs   5.0G   69M  5.0G   2% /tmp
/dev/mapper/rl-ibs      ext4  2.8M   14K  2.5M   1% /ibs
/dev/mapper/rl-v        ext4  2.8M   16K  2.5M   1% /v
/dev/mapper/rl-home     xfs   5.0G   68M  5.0G   2% /home
/dev/mapper/ibs_vg-ibs1 xfs   2.0T   15G  2.0T   1% /v/ibs1
/dev/mapper/ibs_vg-ibs2 xfs   2.0T   15G  2.0T   1% /v/ibs2

Installing IBS

At this point, we will assume you have your OS configured. Now for what you actually need to know about IBS installation.

Create root ssh keys

Generate an ssh keypair for the root user. Default with ssh-keygen is RSA, but if your target systems support newer eliptic curve keys, they are easier to copy and paste into other systems.

# ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/root/.ssh/id_ed25519): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_ed25519
Your public key has been saved in /root/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:uHlLOeCTWG50uFGJgnBtE+U5bbT0BlmGgxsc1dtwYNE root@ibs-linux
The key's randomart image is:
+--[ED25519 256]--+
|. ...+o+==*+     |
| o .+.+*+Bo E    |
|  ....=o*.o=     |
|     ..* .. .    |
|      B S        |
|     * O .       |
|    . X =        |
|     . + o       |
|        .        |
+----[SHA256]-----+

Default location and file name is correct, and you don't want to enter a password.

The public key file will look like this:

# cat /root/.ssh/id_ed25519.pub 
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIMJWzOZ7/g/nCEwfTJ04WepP3I9pYj5wOMiPRQZwqcf root@linux-ibs

The last field (root@ibs-linux) is a comment -- make sure it indicates where the key came from.

Assuming your backup server is coming from a fixed IP address, you may wish to restrict the public key to that IP:

from=192.168.1.29" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIMJWzOZ7/g/nCEwfTJ04WepP3I9pYj5wOMiPRQZwqcf root@linux-ibs

Place this line to the ~root/.ssh/authorized_keys file on your systems you are backing up.

install rsync

if not already in place...

RedHat based systems: dnf install rsync
OpenBSD: pkg_add rsync
...etc.

Install IBS scripts

Pick an install directory. This should be a place where user scripts are expected to be found. Some ideas:

/usr/local/sbin
/opt/sbin

Grab the IBS file here, drop it in your chosen location. File should be owned by root and permissions should be 744, 755, or similar. Keep in mind, a non-root user can't successfully do a backup, but they might be able to run a report.

Configuring

IF your system does not have /bin/ksh on it, change the very first line as appropriate. For example, for Linux, it might be: #!/bin/bash, for AIX it should be /usr/bin/ksh.

Make a log directory. The logs could be stored in one of your backup volume directories, or put into an appropriate /var/ directory. Up to you. I put a symlink /bu/z-logs pointing to the IBS log directory (which in my case is /var/log/ibslogs). I find it convenient, but it can also be "wrong", since it isn't a real backup directory.

So...let's assume we will put config files in /etc/ibs and log files will physically be in /var/log/ibs, but will be accessable by symlink in /ibs/z-logs:

# mkdir -p /var/log/ibs /etc/ibs
# ln -s /var/log/ibs /ibs/z-logs

Run

# ibs -template
Template ibs.config file has been generated in current directory.
Populate it, and copy it to /etc/ibs/ibs.config

(btw: if you get something like: bad interpreter, your first line of the script needs to be changed!)

Look at the comments in the file and set things according to your needs. Copy the edited file to /etc/ibs. World-readable should be fine, really not too many secrets on a system like this.

You need a default backup filter template file, /etc/ibs/default.bufilter. Here's mine, which works ok for basic OpenBSD backups:

+ /
- /bin
- /sbin
- /dev
- /lib
- /lib64
- /mnt
- /media
- /proc
- /run
- /sys

The basic premise here is that we want things starting at root, but skipping some directories that we really don't need in the backup. If something is mounted on /mnt, we don't care. You may wish to exclude /usr, but compared to everything else on a typical system, it probably isn't that big, and you very often have important stuff hidden away (for example, administrative scripts in /usr/local!).

rsync filter files are a bit confusing, so there's a whole dedicated page here for that.

Prepping your first backup!

At this point, you are ready to start building backup target directories and do your first test backups.

Install the IBS root public key into the target system's root user's authorized_keys file2.

Verify you can login as root from the IBS system to the backup target. The first time you do so, you will need to verify that you trust the remote system key.

Prep the IBS backup directory by running the -prep option, specifying the directory on a physical storage device you want the backup to end up on -- something like /v/ibs1:

# ibs -p /v/ibs1/newserver

Assuming no errors, you are ready for a test "First Backup!"

# ibs newserver

this will launch a backgrounded backup of "newserver". Repeate the key installation, test login, and ibs -prep for all the rest of your servers.

Setting up for automatic running

If you backup all your systems all the time, I'd suggest creating a "list" file in /etc/ibs, "all.list", which is a list of all systems that ever get a IBS backup.

If you stagger your backups, you can also create individual list files -- for example, I have some backups I run daily, others I run only periodically, so I have a daily.list in addition to all.list. This works nicely so that my IBS reports are typically run against all.list, but daily I run against daily.list.

Now, I have in my root crontab this:

01 4   * * *    /usr/local/sbin/ibs daily.list >/tmp/ibs-daily.out
01 9 */4 * *    /usr/local/sbin/ibs archive  > /tmp/ibs-archive.out
58 6   * * *    /usr/local/sbin/ibs -report all.list  >/tmp/bureport.out 2>&1

The daily job kicks off at 4:01am, the report is printed a little before 7:00am, and the big archive job is run every four days at 9:01am. Note that the way cron works, days are based on 1, not zero, so */4 means 1, 5, 9, etc., so the ME backups are handled pefectly without any extra effort. Not all unixes take the */4 notation, however.

I could have root mail out the report, but for personal reasons, I wanted the mail to come from my personal account, not from root@ (long story). So, I have IBS dump the output into /tmp, then I have a separate cronjob to mail it to me from my own account:

0 7 * * * mail -s "IBS backup report" nick

I'm not sure that was worth the effort.

Footnotes/elaborations

1. OpenBSD Write^Execute, commonly written W^X, is a huge security feature in OpenBSD -- parts of system memory are marked either as data, so it can be written, or it can be marked as excutable, but not both ("^" is a symbol for Exclusive OR, meaning A or B, but not A and B), in which case, an attempt to write results in your program being stopped and tossed out of memory. A common way for malware to control a system is to write desired "program" into what should be data, then execute that "data". This stops that. All of OpenBSD and most add-on packages run fine in an W^X enforced envionment, but some things won't. rsync is the only package required on an OpenBSD IBS server, so a W^X exception (mounting a partition with the "wxallowed" option) isn't required. All third party apps are installed to /usr/local in OpenBSD, so typically an OpenBSD install has a separate partition mounted with the wxallowed flag. This is done by default. My setup breaks that. Do that at your own risk.

IF you put other third party apps on the system, adding a wxallowed partition for /usr/local may be needed, but because of the sensitivity of this system, you don't want to add apps that might introduce vulnerabilities to the system. For example, don't put a web browser on here. Just don't. So if your application doesn't run without a wxallowed partition, maybe it shouldn't be here.

2. Root file authorized_keys location : This will vary from OS to OS. In general, it's in the root user's home directory. On OpenBSD, that's /root/.ssh/authorized_keys, on AIX, it's /.ssh/authorized_keys. For reasons unrelated to IBS, consider putting your keys in a common directory (I have used /etc/ssh/keys) using the AuthorizedKeysFile option in sshd_config.

Incremental Backup System home Page
Contact Holland Consulting

since August 8, 2022