//////////////////////////////////////////////////////////////////////////////////////
// USB DISK WOES: VOLUME UNMOUNTS DURING COPY, SO STAGGER COPY
// 
// Bryan Smith, Monday March 17 2008
//////////////////////////////////////////////////////////////////////////////////////


I have a FAT32 750GB USB disk with valuable data stored as many, many small files with maximum size of 100MB and an average size substantially lower than this. I started experiencing problems with the disk becoming unmounted in Ubuntu, and had run fsck to repair the disk.

I decided that I wanted to move the data over to a ext3 journal disk. However, while copying, Ubuntu sporadically unmounts and remounts the disk. The call to recursively copy the contents over (using cp -r) then fails quickly, sometimes after running for hours.

I was at a loss on how to best approach the problem, but after a little thought I decided that a shell script to "stagger" copy the data over. (A co-worker suggested the same thing, so this supported my idea.)

I wrote a shell script that recursively copies a directory over to another, waiting a second between each subdirectory is copied. The best part: as a directory fails, it generates a new shell script with the failed copies to try again later.

Most the data has been copied over at the moment. It's admittedly a bit hackish, but a useful solution that highlights one advantage of working in a command line environment.

--------------------------------------------------------------------------------------

#/usr/bin/sh

# Incrementally recursive copy. Used to get data off disk that unmounts
# and remounts sporadically
# Bryan Smith Sat Mar 15

if [ $# != 3 ]; then 
  echo "USAGE: ircp.sh   ";
  echo "Where  can be \"false\" or \"true\", without the quotes"
  
  exit 2 
fi

SOURCE=$1
DEST=$2
USESUDO=$3

# Make sure exists and are directories
BAD_ARGS=0

if [ ! -d $1 ]; then
  echo "ERROR: verify that source $SOURCE exists and is a directory"
  BAD_ARGS=1
fi

if [ ! -d $2 ]; then
  echo "ERROR: verify that destination $DEST exists and is a directory"
  BAD_ARGS=1
fi

if [ $BAD_ARGS != 0 ]; then
  exit 3
fi

# Make sure user provided valid use sudo? variable
if [ $USESUDO = 'true' ]; then
  echo "Using sudo..."
elif [ $USESUDO = 'false' ]; then
  echo "Not using sudo..."
else
  echo "Expecting true or false, found $USESUDO."
  exit 4
fi

echo ""
echo "Incrementally and recursively copying all files from $SOURCE to $DEST"
echo ""

# Clear the fail file log
FAIL_FILE="./ircp.failed-copies"
echo 'Tool started at '`date` > $FAIL_FILE

# Create a shell script that will download failed items
FAIL_SHELL='./ircp.failed.sh'
echo '#/usr/bin/sh' > $FAIL_SHELL
echo '# '`date` >> $FAIL_SHELL

chmod +x $FAIL_SHELL

echo "All failed copies will be recorded at $FAIL_FILE"
echo "  Also, generating shell script with commands to reattempt all failed items at $FAIL_SHELL"

# For each file (directory or normal), recursively copy to dest and sleep
for file in $1/* ; do
  #echo "Copying $file to $DEST"

  if [ $USESUDO = 'true' ]; then
    sudo cp -r $file $DEST
  else
    cp -r $file $DEST
  fi

  # Check whether exitted normally. If not, write failure to failure file
  #echo "Copy exited with value $?"  
  COPY_STATUS=$?
  if [ $COPY_STATUS != 0 ]; then
    # Add entry to fail log
    echo "Failed to copy $file to $DEST" >> $FAIL_FILE
    echo "cp exitted with code $COPY_STATUS" >> $FAIL_FILE
    echo "" >> $FAIL_FILE

    # Add entry to shell script
    echo "echo \"Copying next file/directory $file\"" >> $FAIL_SHELL
    if [ $USESUDO = 'true' ]; then
      echo "sudo cp -r $file $DEST" >> $FAIL_SHELL
    else
      echo "cp -r $file $DEST" >> $FAIL_SHELL
    fi
    echo "sleep 1" >> $FAIL_SHELL
    echo "" >> $FAIL_SHELL 
  fi

  # Sleep one second perchance drive unmounts then remounts
  sleep 1
done