diff/cmp for remote files

December 24, 2009

rdiff, rcmp – diff remote files

rdiff and rcmp extends the diff and cmp utilities to remote files.
Each file or directory argument is either a remote filename of the form [[user@]host1:]file, or a local filename.

Note: the scp program is used to retrieve a remote file. Setting up ssh authentication (~/.ssh/authorized_keys on the remote host ) may be required. Google ‘ssh authorized keys’ for info.
Note2: rdiff can be hardlink’d to rcmp (ie: ln rdiff rcmp). The filename is used to determine functionality.

Example usages:
    $ rdiff -b produser@prodhost:/home/prod/bin/xyz.sh xyz.sh
    $ rcmp produser@prodhost:/home/prod/javalib/xyz.jar $HOME/dev/java/lib/xyz.jar
    $ rcmp ruser@rhost:/usr/local/abc:def /usr/local/abc:def

#! /bin/sh
#  Name: 
#  Synopsis:    rdiff [-b] [[user@]host1:]file1 [[user@]host2:]file2
#               rcmp [[user@]host1:]file1 [[user@]host2:]file2
#  Description: Run diff/cmp on remote files.  
#               Note: This script uses scp or rcp to copy remote files.
#                     (~/.ssh/.authorized_keys, or ~/.rhosts may need to be configured)
#               Note2: Simple pattern matching is used to determine if a file is remote or local 
#                      The filename matches a "user@host:" or "host:" prefix pattern.  To force
#                      a match for a local file, specify the full path with the leading '/'.

        if perl -e 'exit !(($ARGV[0] =~ /.*\@.*:\/.*/) || ($ARGV[0] =~ /.*:\/.*/)) &&
                         !($ARGV[0] =~ /\/.*/);' $1; then
                scp "$1" "$2"
                # rcp "$1" "$2"
                ln "$1" "$2";  # This is slight overkill, after all we already
                               # have the file.  But, it simplifies file cleanup...

# Main
case $0 in
  *rdiff) cmd=diff;;
  *rcmp) cmd=cmp;;

while getopts b opt; do
        case $opt in
          b) options="${options} -b";;
shift `expr $OPTIND - 1`

trap "rm -f $tmpfile1 $tmpfile2" 0 15

if getfile "$1" "$tmpfile1" && getfile "$2" "$tmpfile2"; then
        eval $cmd $options $tmpfile1 $tmpfile2

Here’s a very simple utility program that I use to debug how the shell interpreter parses quoted arguments in shell scripts. Shell quoted strings can be one of the trickiest things to get right in a script. White-space characters in filenames and directory names, string concatenation, nested single, double, back-tick quotes are only a few of the complications!

Note: To compile the code    $ cc printargs.c -o printargs

Here’s an example of debugging using printargs:
  $ cp $file $target

This works fine as long as the $file or $target name does not contain any white space characters. But if they do, the command will fail. For example: turn on trace ‘set -x’ and set $file=’Star Trek IV.mp4′. The trace will display ‘+ cp Star Trek IV.mp4 destdir’ which looks right, but actually is wrong. Let’s prefix the cp command with printargs to see why.

$ file=’Star Trek IV.mp4′; target=destdir
$ printargs cp $file $target
+ printargs cp Star Trek IV.mp4 destdir
1 : ‘cp’
2 : ‘Star’
3 : ‘Trek’
4 : ‘IV.mp4’
5 : ‘destdir’

The copy command is actually trying to copy three files {Star, Trek, IV.mp4} to the dest directory. It should be just one file (‘Star Trek IV.mp4’). The fix is to quote both $file and $target ie: cp “$file” “$target”.

$ file=’Star Trek IV.mp4′; target=destdir
$ printargs cp “$file” “$target”
1 : ‘cp’
2 : ‘Star Trek IV.mp4’
3 : ‘destdir’

After debugging, remove the printargs prefix from the command and the script should be good to go.

/* Name:        %I%
 * Synopsis:    printargs -c -i -q commandline
 * Description: printargs displays cmdline arguments.  This is useful for debugging 
 *              shell scripts that escape the space, backslash, quote chars ('"`)
 *              and other wildcard characters (eg: "[*?]").
 *                Options: -c  Echo the command line
 *                         -i  Do not display argument indices
 *                         -q  Do not display single quotes surrounding each argument


main(int argc, char *argv[])
        int opt;
        extern int optind;
        int echo_cmdline = 0; 
        int display_indices = 1; 
        int indx;
        char *quotes = "'";
        char *spaces = "";

        while ((opt = getopt(argc, argv, "ciq")) != EOF) {
                switch(opt) {
                case 'c': 
                        echo_cmdline = 1;
                case 'i':
                        display_indices = 0;
                case 'q':
                        quotes = "";
                        fprintf(stderr, "%s: Invalid option '%c'\n", opt);
                        fprintf(stderr, "Syntax: %s [-ciq] command command_options command_args ..."

        if (echo_cmdline == 1) {
                printf("CmdLine: ");
                for (indx = optind; indx <= argc; ++indx)
                        printf("%s%c", argv[indx], indx < argc ? ' ' : '\n');

        for (indx = optind; indx <= argc; ++indx) {
                if (display_indices == 1)
                        printf("%d : ", indx - optind + 1);
                printf("%s%s%s\n", quotes, argv[indx], quotes);


File Find

October 28, 2009

Here’s a shell script that implements the old Norton Utilities “FileFind” utility. I added a couple of useful additional options.

  • ‘-l’ will display a detailed file listing ala ‘ls -l file’.
  • ‘-c’ and ‘-m’ will display the file using cat and ‘more’ respectively.
  • ‘-e’ option will execute a program on the found file(s).

Example usage:
    $ ff. notes.txt; # find the notes.txt file
    $ ff. -m notes.txt; # display the notes.txt file
    $ ff. -e ‘grep footnote /dev/null’ notes.txt; # grep footnotes from notes.txt files

#! /bin/sh
#  Synopsis:    ff. [-lcm] [-e program] filename ...
#  Description: find files.
#               Options: -l  display directory long listing of file
#                        -c  cat the file
#                        -m  display the file using 'more' (or less)
#                        -e  execute 'program' on the file

while getopts lcme: opt; do
        case "$opt" in
          l) print='-exec ls -l {} \;';;
          c) print='-exec cat {} \;';;
          m) print='-exec less -E {} \;';;
          e) print="-exec $OPTARG {} \;";;
          \?) exit 1;;
shift `expr $OPTIND - 1`

names="-name $1"; shift
for f; do
        names="-name $f -o $names "

eval find . '\(' $names '\)' ${print:--print}

A safe rm

October 27, 2009

Here is a script that implements a Windows style ‘Recycle Bin’ delete function. Similarly like Windows delete, it moves deleted files/directories to a temporary directory (default: $HOME/.tmp). After a file is moved to the tmp directory and has not been accessed for N days, it is permanently deleted the next time the rm command is run (think of this as an automated ’empty recycle bin’ function). It implements all the ‘rm’ options making it an ideal replacement for the /bin/rm command in interactive use.

Example usage:
  $ alias rm=$HOME/bin/rm

#! /bin/sh
#  Synopsis:     rm [-fir] file ...
#  Description:  A Safe rm (move file(s) to a tmp directory for later deletion).  Files in
#                the tmp directory that have not been accessed in N days are permantly deleted 
#                the next time the rm command is run.  
#                Options: -f  (force) Run "/bin/rm" instead of moving files to the tmpdir
#                         -i  interactive mode
#                         -r  move (recursively) a directory tree to the tmp directory
#                Caveat: mv fails if the filename matches a directory name in the $tmpdir directory.

#set -x
while getopts fir opt; do
        case $opt in
        f) force="-f";;
        i) interactive="-i";;
        r) recursive="-r";;
        \?) exit 1;;
shift `expr $OPTIND - 1`

if [ "$force" ]; then                            # Run /bin/rm if '-f' is specified!
	/bin/rm -f $interactive $recursive "$@";  # This is useful in shell functions
	exit $?                                   # when you want the real rm executable

for file; do
        if [ -n "$interactive" ]; then
                echo -n "remove '${file}'? "
                read yn
                if [ "$yn" != "Y" -a "$yn" != "y" ]; then
        if [ -d "$file" ]; then
                if [ -n "$recursive" ]; then
                        mv -f "$file" "$tmpdir" || retcode=2
                        echo "$0: cannot remove '$file': Is a directory"
        bf=`basename "$file"`
        mv -f "$file" "$tmpdir"
        touch "$tmpdir/$bf";  # For post cleanup, $file will be deleted N days from today

# Prolog: Clean up files, permanently delete files after atime days have elapsed
{ /bin/find "$tmpdir" -type f -atime +7 -exec /bin/rm '{}' \;
  /bin/find "$tmpdir" -mindepth 1 -type d -exec rmdir '{}' \; 2>/dev/null 
} &

exit ${retcode:-0}