Easy list-of-things handling

why?

Hang with me a moment here. This is a simple problem that is usually solved in a simple way.

You have a task you have to perform across a bunch of systems. So... you write a simple wrapper around your script, something to the effect of:

for S in $(cat server.list); do
    ...some task...
done

And that's stupidly simple, right?

But of course, something goes wrong on a few of the systems, so now you have to re-run that task against those. But your script was set up for a file, and a particular file at that...so you edit the file, put the problem machines in it, re-run. And repeat.

And you could write your script better, but that takes time. So you don't do it. And you have this tiny problem a lot of times. And each time, you think, "If I had done it a little better, it would have saved a lot of time!"

Worse, perhaps...that text file has no provisions for dealing with comments or commented out lines.

listfile

What if instead, you wrote your script to do this:

for S in $(listfile $*); do
    ...some task...
done

where "listfile" is a little script that expands the command line; either as a list of things, or a file that might be sloppily formatted or have commented out lines, blank lines, etc? So, first time you run your process:

$ checkuptime allservers.list

Second time, after you fix a few things, you can run against just the failures by:

$ checkuptime web02 database04

First time, you used a .list file that had a big list of machines in it. Second time, just listed the machines individually.

Difficult, no. But probably not something you are tossing in every script you write. But I did it..and here it is: "listfile": (download a file rather than copy/paste here.)

#!/bin/sh
#
# Copyright (c) 2023,2024 Nick Holland 
#
# Permission to use, copy, modify, and distribute this software for any
# purpose with or without fee is hereby granted, provided that the above
# copyright notice and this permission notice appear in all copies.
#
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

# unrelated to the above license, I'd love to hear from you if you
# use this and find it useful...or if you find flaws or simple improvements.

function helpexit {
    more <<-__ENDHELP

	 Processes a set of parameters, which may be individual items and/or
	 a file name ending with .list, which is then expanded.  Within the
	 *.list file could be other .list files (recursion limited to five 
	 levels, to prevent infinite loops).
	
	 For example: 
	  +------------------------------------------------------------------
	  |# Test for listfile. comments should be ignored
	  |One
	  |    two
	  |# ignore this line
	  |three  # comments to be ignored
	  |four five  # ignore2
	  |              
	  |		
	  |# above lines were several spaces and several tabs
	  |#ignore3
	  |# ignore4
	  |six
	  +------------------------------------------------------------------
	
	  $ listfile zero testfile.list SEVEN
	  zero One two three four five six SEVEN
	
	 if the first parameter is "-l", the space delimiter will be replaceed
	 with a newline (one item per line)
	
	 Here is a simple uptime checker that will take individual machines or 
	 .list files:
	  +------------------------------------------------------------------
	  |#/bin/sh
	  |
	  |LIST="$(listfile $*)"
	  |COUNT=0
	  |
	  |for i in $LIST; do
	  |    COUNT=$(( $COUNT + 1 ))
	  |    echo -e "\n$i -------- $COUNT"
	  |    if ! ssh $i uptime; then
	  |        echo -n "DIDN'T WORK!  (hit enter)"
	  |        read zzz
	  |     fi
	  |done
	  +------------------------------------------------------------------
	
	__ENDHELP
    exit
    }

LAYER=0 # layers of recursion
MAXLAYERS=5 # at some point, we've probably got a loop going.  exit.

if [[ $1 == "-h" ]]; then
    helpexit 
fi

# Set the separatator
if [[ $1 == "-l" ]]; then
    shift
    SEPARATOR="\n"
else
    SEPARATOR=" "
fi

function dotlist {
# $1 is recursion level $2- is the stuff to process
    typeset RL
    RL=$1; shift
    # echo "\n=== $RL $* ==="
    if [[ $RL -gt  $MAXLAYERS ]]; then
	echo "\n==likely loop.  Exiting==" >&2
	exit
    fi
    for E in $*; do
	if [[ $E == *.list && -f $E ]]; then
	    dotlist $(( $RL + 1 )) $(sed "s/#.*//" $E)
	else
	    echo -n "$E$SEPARATOR"
	fi
    done
} #end function dotlist

# do it
dotlist 0 $*

Yeah, I got a lot carried away with the help screen here, but I figured no one was ever going to look at the documentation.

Given no parameters, listfile silently exists. This is sometimes useful.

A recent addition was adding the ability to have list files inside list files. So...you could have five list files, each for different parts of a project, and one other list file that incorporates all of the other five within it. Obviously, this could lead to a problem if a loop was accidentally or intentionally created, where a list file invokes itself, or more subtly, invoked a list which invoked a list which re-invoked one of the earlier lists. So...a recursion limit of five was hard coded in; easy enough to change if you have legitimate reason for deeper. I did it using simple recursion, that's always a questionable choice, but I don't think it is going to bite us here.

Yes, stupidly simple. But download it, look it over, put it on your control systems, and enjoy. Invoking listfile might actually save you a few characters typing in your script, and make it work better, too.

limitations

While the .list files can be very free format, what is NOT handled is spaces within individual items. That was partly deliberately, partly lazy. I wanted this to be valid:

server1 server3 # Bob's servers
server2 server5 # Mary's servers

You may have different needs; if so take my idea and run with it.

Tested on:

OpenBSD
bash on RHEL9-equiv Linux

but at a minimum, should be easily adapted to any ksh/posix/bash like shell on any Unix.

Holland Consulting home page
Contact Holland Consulting

since Aug 4, 2024