website sync tool for Linux (web site synchroniser with built-in tutorial!)

the shell scripts..

Here are the actual scripts that I use to sync corz.org with its development mirror on my Linux box. Essentially, we log changed local files, and upload only those files. The mechanism for all this is a simple site backup; a mirror of my mirror site! There are many advantages to this technique..

I don't have to keep tabs on which files I've edited; and I tend to edit lots of different site files in one session; the backup script does this for me. Only new and changed files are backed up, thereby indicating which files are new or changed!

You won't be able to use this "as-is", it will need tailoring to your environment. Still, that two minutes will save you hours and hours and hours in the future, so it's worth it. All the main scripts are included in the zip package. If you find any bugs, please report them. ta. okay, here goes..

file: /usr/local/bin/cb

(there's also a non-archiving version included, "cbl", for general update use)

#!/bin/sh
# corz.org mirror backup script.. (run as daily cron)

# this runs on a Linux box, where both the real dev mirror, and the
# backup directory live (though the latter is on a separate physical
# drive - it is mounted in /bax) you could backup to a different
# machine, though remember to ensure the share/network-drive is
# mounted first.

# this script updates $backupdir with all changed files from $devmirror
# all this activity is logged to a file which we parse into sftp commands
# with the "website-sync" php script, running on a web server somewhere.

# © corz.org 2003->

# prefs..

# log to this file (the content we will later convert into sftp commands)
logfile="/home/corz_mirror.log"

# crucially, this must match the $mirror_dir variable in the php script..
devmirror="/home/cor"

# this script will recreate my dev mirror inside here..
backupdir="/bax"

# a gzip of the whole lot will go here..
storedir="/bax/www"

# make things readable..
echo  >> $logfile
echo "corz.org backup script: "`date "+%Y.%m.%d-%H.%M"`.. >> $logfile
echo
echo  >> $logfile
echo "corz.org backup script (full version - with archiving)"
echo
cd $devmirror
cd ..

# tar and gzip the current mirror..
echo "archiving dev mirror.."
tar cfz corz-bax.tar.gz $devmirror/ >> $logfile

# copy the tar.gz to the backup directory with unique dated filename
echo "storing archive backup.."
mv corz-bax.tar.gz  $storedir/`date "+%Y.%m.%d-%H.%M"`_corz.org.bax.tar.gz >> $logfile

# copy over all the newer files (Linux cp rocks! Apple take note.)
echo "copying changed files to mirror.."
cp -d -f -p -r -u -v $devmirror $backupdir >> $logfile

echo
echo >> $logfile
echo "corz.org backup script complete"
echo

# we now have an up-to-date backup
exit 0

# fin

Whenever you want to backup your dev mirror, or sync it with your live site, you run the backup script. Both versions do the exact same backup, updating only new or changed files. The above example also gzips the entire backup and stores it somewhere. Most likely, you will want to set that one as a daily/weekly cron job..


# 5.45am corz.org mirror backup [now weekly, monday]..
45 5 * * 1 /usr/local/bin/cb >> /var/log/cron 2>&1

The 'lite' version (cbl) is for manual use. It simply performs a backup, updating only new or changed files, and makes no zip. When you are looking to do a quick sync, use the lite version. I often use the lite version dozens of times a day. The first time I run it, I see that there are also log entries from the earlier cron backup, and they are simply incorporated into the next upload.

If you use it as often as I do, you may want to rename it to just 'c', or some other single letter that doesn't clash with an existing executable in your $path, so it's quicker to type.

When you run your backup script ('cb', or 'cbl' - or whatever you named them), the output will be captured to a log. The log's contents are then fed into the php web sync tool (which your local Apache will be happy to server up for you at..

http://localhost/website-sync-linux.php

or wherever. The output from the web tool is then pasted into a simple sftp upload script something like this..

NOTE: This section is outdated, everything is sftp these days, and website-sync is setup to output SFTP commands. See below for more details.

file: ~/up

#!/bin/sh

# upload new files to website..

HOST='ftp.mysite.com'
USER='username'
PASSWD='password'

ftp -n $HOST <<FTP_QUIT
quote USER $USER
quote PASS $PASSWD
binary
verbose

  # paste converted ftp commands right here (replacing this line)

quit
FTP_QUIT
exit 0

# fin

Run the "up" script. That's it..

Your wesite and its mirror are one!

Running all this from Windows®..

Although the scripts run on a Linux box, you can easily mastermind the whole show from a Windows workstation (I've had a couple of questions about this, and I'm here anyway, because this is where EditPlus is!). Here are some suggestions on how to get the whole thing down to a 3.62 second operation...

Keep shortcuts in a toolbar, or somewhere handy, to these three important things (which you will use, in order)..

The mirror log (output from the update script is in here)
URL shortcut to a local copy of the website sync tool (you are reading the other half of it).
The "up" shell command on your Linux box
You will need to edit this file, you see, add your ftp commands.

I have shortcuts to all sorts of resources on all sorts of machines, right here in my Windows desktop and toolbars. A handy shortcut can save LOTS of time, and only takes a moment to make and place.

Tools you need:

Text editor (For Windows, I usually recommend EditPlus).
While you are working on the site, it's easier to leave the output log and up script open in your text editor.
Telnet/SSH client (to run a shell on your Linux box - I definitely recommend Putty).
While you are working on the site, it's easier to leave your Linux shell running - it was open anyway, right?
Web Browser (with your local website sync page in on of its tabs).

A few simple steps..
The whole operation takes only a few seconds, on any platform. Here's how..

Login to a shell on your Linux box, run the "lite" backup command; that is; type "cbl<enter>" (no quotes).
Switch to the backup log in your Text editor, (your text editor should have automatically reloaded the file, which now contains the log output we will process) and cut the log output into the clipboard.
Switch to your browser, and paste the output into website sync tool, click 'make ftp commands!'
Switch to your text editor and paste the output into the up script, save that.
Back in your Linux shell, run the up script. You're done.

With practice, you can do the whole thing in under three seconds!

A note about ftp security..

The ftp "up" script (above) is the "simple-and-insecure" version of things. It's easy for anyone to get working, and does the job. But there are least three things you could do to increase the security here..

Firstly, it' not very smart to keep login passwords inside files that can potentially be viewed by anyone. So you could start by keeping the script somewhere inside your home folder, as in the example above. But that's not very convenient, and it's not good organisation, either. Here's a better idea..

Remove the "-n" switch, and all the user/pass commands from the ftp script, and instead allow ftp to use auto-login; with your ~/.netrc file. This is simply a plain text file in your home folder named .netrc. It has single line entries that look something like this..

machine ftp.mydomain.com login MyName password MyP4$$w0Rd

This has obvious security benefits. Do chmod 600 ~/.netrc to make it readable by only you, and then feel free to leave ftp login scripts lying around all over the place!

Automatic SFTP Scripting

Rather than use plain ftp, it's possible you have access to SSH on your web server, and therefore; SFTP. It's not quite so simple to automate SFTP as it is ftp, but it is doable, and this first sftp script, using "expect", works very well, though it's not quite as secure as the second method. Without going into too much detail, you'll need two scripts; a launcher, which will "spawn" an interactive expect session..

file: /usr/local/bin/sftp/up-launch

#!/usr/bin/expect
#
spawn sftp -C -b /usr/local/bin/sftp/up sample@example.com
expect "password:"
send "MyP4$$w0Rd\n";
interact

Which we will script (use a text file to do the interactive parts, instead of us) with a simple list of commands something like this..

file: /usr/local/bin/sftp/up

progress

put "/home/cor/blog/inc/cbparser.php" "/blog/inc/cbparser.php"

put "/home/cor/blog/inc/cbguide.php" "/blog/inc/cbguide.php"

Note the "progress" at the start. Apart from that, it's just a list of regular SFTP commands. For more details, check out the man pages for expect and sftp. Of course, you'll also need these two programs installed for all this to work.

And of course, now you've got passwords inside your sftp scripts! Aarrrghhh!!!!

Better Automatic SFTP Scripting

These days, I prefer to use RSA keys to authenticate with the server. While more work to setup, they are more secure and make automatic scripting a breeze. I won't go into details here. You can Google that.

Once you have your keys setup, you simply do something like this..

file: /usr/local/bin/sftp/up

sftp -b commands.txt MyUserName@MyWebHost

And inside commands.txt, you drop your regular sftp commands (as created by this tool)..

file: /usr/local/bin/sftp/commands.txt

put "/home/corz.org/blog/inc/cbparser.php" "httpdocs/blog/inc/cbparser.php"

put "/home/corz.org/blog/inc/cbguide.php" "httpdocs/blog/inc/cbguide.php"

etc..

Have fun!

;o) Cor