corz.org uses cookies to remember that you've seen this notice explaining that corz.org uses cookies, okay!
<?php // ۞// text { encoding:utf-8 ; bom:no ; linebreaks:unix ; tabs:4sp ; }
require_once $_SERVER['DOCUMENT_ROOT'].'/inc/init.php';//distro
/* v0.9.5 (php4+)
site synchroniser tool
- synchronise a website with its development mirror -
This is a self-contained auto-site-synch-tutorial-cum-conversion-tool thing
In the beginning there was no backup, I would manually go through my whole
site with an ftp client, uploading any updated files to the live website.
Sometimes I'd miss some, and always it was a pain-in-the-ass.
Then came the backup script.
It logs its actions to a file, which *this* tool converts into a set of
valid (s)ftp commands that can be used to synchronise a real live website
with new and updated files from a development mirror at home/wherever.
Don't expect the shell scripts to "just work" for you; they won't. They are
specific to my own setup. you'll find them obvious enough to edit, though.
For all this to work you need a website, a local mirror, and a backup of the
local mirror. Apart from the website, these will need to be on *nix
machines; Linux, OS X, BSD, Unix, whatever, though I definitely recommend
these operating systems for your web server too! while I use multiple
machines for this, it would be no bother to just use one, though obviously
this would be a less secure back-up. A virtual machine will be fine.
The whole sync process is now reduced to a cut&paste and a click, and a
couple of shell commands; one for doing the backup (which creates the
all-important log entries), the other to upload the changed files to the
website via (s)ftp. the backup script part is best run regularly and auto-
matically, a la "cron". What took hours, now takes seconds. The whole point.
The actual backup shell script (available from inside this php script) runs
twice a day as a cron job, (I have a "lite" version, without the archiving,
most useful, and much faster, which I run manually after long sessions) the
other shell command - "up" - I also do manually. note: shell scripts must be
made excecutable before they will run..
chmod +x up
I'll include the shell scripts in the zip, I recommend grabbing that.
I prefer to "up" manually. If the whole thing was automatic (and with php on
the command-line, that's quite doable), there's a very high chance that
"in-progress" scripts would be uploaded to the live website. ouch! Not very
smart. "up" is called at the end of work sessions, when I am fairly certain
the local copy is functioning as expected, without bugs, and so on. And after
I've examined the upload script for unexpected anomalies.
I guess it would be possible to move the whole lot into one php script,
but therein lies a can of worms, so it probably won't happen.
To use..
Copy the contents of the the mirror.log file into this tool's textarea.
click "make sftp commands!"; valid sftp commands are returned, which you
then paste into "up", your sftp upload script (also provided here,
there's a link on the generated page).
In action..
The log file and "up" script are left open in my text editor, always
(usually tabs two and three) and I let it auto-update when files change..
[shell] c (my symlink to the lite backup command)
[desktop] copy log (into clipboard)
[desktop] paste into *this* tool, hit "make sftp commands!"
[desktop] copy output into "up" script, save
[shell] up (my symlink to the upload script)
If you prefer, you can do the whole lot in the shell.
Here's the longer version..
In a shell, I run "c" (as I named my quick backup script; easy to type)
Switch (Alt+Tab) back to the text editor - there's my log - Ctrl+A,
Ctrl+X (select all and cut to clipboard)
Switch to browser (sync tool - tab 1) Ctrl+V, click "make sftp
commands!", click "select all" (or click the textarea), Ctrl+X.
(again, data cut to clipboard, Ctrl+C if you prefer)
Switch back to text editor, Ctrl+S (save blank log), select "up" tab,
Ctrl+V, Ctrl+S (pasted back out, and the file saved)
Switch to shell, "up". (or more usually, hit the up arrow twice to
recall the previous "up" command; so I'm lazy!)
Done.
The whole thing takes only a few seconds once you are up to speed; and
it quickly gets into muscle memory.
I use this very tool every day, sometimes many, many times a day.
have fun!
;o)
(c) copyright corz.org 2004->today
NOTE: Currently the long path handling (recently tagged-on) is buggy. ftp
seems to have a limit to the length of paths. Darn! My work-around for
extremely long paths *usually* works, but occasionally borks the entire
path. Watch for that! If you get the time, feel free to fix this and return
the code to me!
NOTE: sftp appears to have no such limit.
*/
// prefs..
// location of mirror..
$mirror_dir = '/home/sites/corz.org/httpdocs';//distro $mirror_dir = '/home/sites/mysite.com/public_html';
//
// the above path must match the backup script's $devmirror (or similar) variable
// essentially, this path must appear in the left-hand side of your log output, e.g..
// '/home/sites/corz.org/httpdocs/inc/foo/.ht_password' -> '/bax/sam/inc/foo/.ht_password'
// $mirror_dir is stripped out of this, and what's left (/inc/foo.inc) is used to
// construct the sftp command.
// where the backup is..
// we can upload from this directory, too.
$backup_dir = '/bax/mirrors/corz.org';
// location of files on the LIVE server (ftp path from root)
// this is what you see in the remote site directory if you login
// to your ftp manually..
$ftp_web_dir = '/httpdocs'; // Plesk user? ;o)
/* Upload files from which folder?
Normally we just point this at the mirror. However, you may prefer to point it at
your backup folder, which seems sensible, use the full path, like so..
$upload_from = '/bax/mirrors/mysite';
*/
$upload_from = $mirror_dir;
// You will need a local copy of this if you want to suport IE7/8 users..
$HTML5_shiv = '/inc/js/html5.js';
// For regular ftp, long lines (over 190 characters-ish) can be a problem.
// I can split them for you..
$split_long_lines = false;
// for sftp, afaik, it's no longer a problem.
// When using sftp, any error will halt the entire operation. Usually this behaviour is
// undesirable, so we ignore errors and continue. They will still be logged for our
// information. Basically, setting this to true prepends every command with a dash "-".
$ignore_errors = true;
/*
filters..
add any files/file types/paths/strings you want stripped out/ignored/removed/wiped..
Keep/delete what you need/don't..
NOTE: These values are now Case-Sensitive.
*/
$ignore_files = array ( // will still makes dirs called these though //:2do
'/.', // always do invisible files manually // comment out to upload .htaccess files
'example backup', // the info lines in the log
'versions-local.ini', //
'versions-local.php', //
'backup for', //
'licenses/', //
'links.ini', // blog sidebar links
'- Copy', // Windows copies
// 'backup/', // actual backups!
'local-test.p', // local testing files!
'counter', // counters!
'.bak', // backup files
'.out', // debug files
'.lnk', // Windows shortcuts
'.comment', // don't replace online comments!
'.log', // don't replace the logs!
'mail-lists/', // ARRGGHH! don't replace these! (again ;o)
'WindowsSoftware', // SHHH!
'_thm.jpg', // thumber thumbs!
'amp.ini', // local live ampsig
'thumbs.ini', // my live server has GD, thumber can function locally, there.
'thumbs.db', // pesky Windows machines!
'desktop.ini', // pesky Windows machines!
'.DS_Store', // pesky Mac machines!
'._.DS_Store', // pesky Mac machines!
'.desktop', // Pesky Linux Machines!
// Fexxxake! Could all the platforms PLEASE agree on ONE file name for desktop prefs. Sheesh!
// then we could put all settings in one file, and access from any platform, WITH PREFS. THINK DAMMIT!
'1016: Connection', // the puter was sleeping! oh dear.
'Error connecting to ', // ditto
'/log/', // don't replace the logs!
'arc/', // don't upload the archives!
'_arc/', // don't upload the archives!
'php.ini', // again, I prefer to do these manually.
'.user.ini', // ditto.
'sessions/HammerID_', // sorta like php sessions, but better!
'sessions/Count', // guess!
'private/', // local only
'2do', // I leave these extensionless things lying around, well, one. handy.
'tmp_edit.sh', // online file editor temp file
'white-list.txt', // Anti_hammer List
'black-list.txt', // Anti_hammer List
'black-log.txt', // Anti_hammer List
'/test.html', // jus testin'!
'sessions/', // session files'!
'tests/', // jus testin'!
'robots.txt', // online version if different
'chatters.db', // chatters
'.blog', // blogs up manually, if ever!
'backup for samuelz', // My backup script outputs an information line including this string. Remove it.
'[BLANK]', // Blank galleries template.
'/test.php' // jus testin'! [no comma after last entry, of course]
);
// Special Command..
//
// do special commands for these files..
// enter either a path, a file name, or an extension (with the dot)
// and we will perform a "special command" (defined below) on these files
// default: $special_file = array('.blog, mail_footer.txt');
//
$special_file = array(
'mail_footer.txt',
'corz.dic',
'main-links.ini',
'thumbs.ini',
'banned_ips.txt',
'banned_referers.txt',
'schemes.ini',
'original.ini',
'banned_spammer_post_strings.txt',
'banned_agents.txt'
);
// what to do?
// some sftp command, default: $special_command = '-chmod 777';
//
$special_command = '-chmod 777';
//
// NOTE: The special command will not have a dash "-" prepended if you have $ignore_errors
// set to true. If you want your special command to ignore errors, add the dash yourself. e.g.
//$special_command = '-chmod 777'; // this is the default behaviour.
// Make Directories?
//
// items with no extensions are presumed to be directories.
// If you use extensionless files on your web site (WHY? Oh WHY?),
// you might want to set this to false..
//
$make_dirs = true;
// Delete remote files before uploading?
//
// If you set this to true, website sync will put a del/rm command before each
// put command, to delete the file from the server. This is more useful over
// ftp. sftp shouldn't ever have issues simply overwriting the file, but ymmv.
//
$delete_first = false;
// your header here? it's just eye-candy, folks. (use a relative or FULL absolute path)
$site_header = $site_config['header'];//distro $site_header = $_SERVER['DOCUEMNT_ROOT'].'/inc/header.php';
// style sheets, separate paths to multiple style sheets with commas..
$site_styles = $site_config['sync_styles'];//distro $site_styles = '/inc/css/main.css,/inc/css/sync.css';
// and footer.. (use a relative or FULL absolute path)
$site_footer = $site_config['footer'];//distro $site_footer = $_SERVER['DOCUEMNT_ROOT'].'/inc/footer.php';
//
// end regular prefs.
// the (built-in) shell scripts page..
if (isset($_GET['showscript'])) {
do_header();
if (file_exists($site_header)) { include_once $site_header; }
show_script();
die('</body></html>');
}
// Note everything is Apostrophe's now!
$example_string = " example backup script log output
/. some lines will be converted, the rest, stripped..
'/home/sites/corz.org/httpdocs/.ht_password' -> '/bax/cor/.ht_password'
'/home/sites/corz.org/httpdocs/.ht_refs' -> '/bax/cor/.ht_refs'
'/home/sites/corz.org/httpdocs/blog/blogz.blog' -> '/bax/cor/blog/blogz.blog'
'/home/sites/corz.org/httpdocs/blog/config.php' -> '/bax/cor/blog/config.php'
'/home/sites/corz.org/httpdocs/blog/inc/spelling/corz.dic' -> '/bax/cor/blog/inc/spelling/corz.dic'
'/home/sites/corz.org/httpdocs/blog/foobar' -> '/bax/cor/foobar'
'/home/sites/corz.org/httpdocs/blog/foobar/new.php' -> '/bax/cor/foobar/new.php'
'/home/sites/corz.org/httpdocs/dev/debug.out' -> '/bax/cor/dev/debug.out'";
$cmd_prep = '';
if ($ignore_errors) {
$cmd_prep = '-';
}
// display the converter tool..
//
if (isset($_POST['cvt_area'])) { $text = ($_POST['cvt_area']); } else { $text = ''; }
if (get_magic_quotes_gpc()) $text = stripslashes($text);
if (isset($_POST['prep'])) { $converted = str_replace("\n\n", "\n", prep($text)); } else { $converted = $example_string; }
do_header();
if (file_exists($site_header)) { include_once $site_header; }
echo '
<div class="content tool">
<form method="post" action="',$_SERVER['REQUEST_URI'],'">
<div class="form"><br />
<label><strong>enter the raw log data..</strong>
<textarea name="cvt_area" id="cvt-area" rows="20" onclick="this.focus();this.select()" spellcheck="false"
title="If you click, all text is selected (for copying) - if you want to edit, use the keyboard!">',$converted,'</textarea>
</label>
<div class="clear"></div>
<div id="sftp-up">
<input type="submit" name="prep" value="make sftp commands!" />
</div>
<div id="select-all">
<input title="javascript select all function" type="button"
value="select all" onclick="javascript:this.form.cvt_area.focus();this.form.cvt_area.select();" />
</div>
<div class="clear"></div>
<!--site synchroniser tool (c) 2004 -> cor @ corz.org-->
<h2>The linux website sync tool..</h2>
..takes output from a *nix shell command like this..<br />
<br />
<code>
# copy all the files over, newer files only, and log<br />
cp -d -f -p -r -u -v /home/cor /bax >> $logfile
</code><br />
<br />
and converts it into valid sftp commands we can use to update our live web site
with all the new and changed files from its development mirror. i.e synchronize<br />
<br />
if you are a mac user without a linux box kicking around, you will probably prefer
<a href="https://corz.org/machine/source/php/website-sync-mac.php"
title="just got one mac box? no problem...">this version</a>.
</div>
</form>
</div>';
//<div class="toplinks">
// <a href="?showscript=true" title="the shell scripts responsible for producing the log">view the scripts</a>
// •
// <a href="https://corz.org/engine?section=php"
// title="download the zip of the website-sync">download zip</a>
// •
// <a href="https://corz.org/machine/source/php/website-sync-mac.php" title="check out the mac version">mac version</a>
//</div>
toplinks_out(true);
if (file_exists($site_footer)) { include_once $site_footer; }
echo '
<!-- <div id="uplink"><a href="/" title="go to ',$_SERVER['HTTP_HOST'],' proper">.. up to the main site</a></div>
--></body>
</html>';
/*
function do_header() */
function do_header() {
global $HTML5_shiv, $site_styles;
echo '<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="initial-scale=1.0, width=device-width" />
<title>website sync tool for Linux (web site synchroniser with built-in tutorial!)</title>
<meta name="description" content="website synchronizer (synchroniser) tool, converts shell backup log (cp) output to valid sftp commands, comes with all the required shell scripts, and built-in tutorial, sort of" />';
$style_sheets = explode(',', $site_styles);
foreach ($style_sheets as $my_sheet) {
echo '
<link rel="stylesheet" href="',$my_sheet,'" type="text/css" media="screen" />';
}
echo '
</head>
<body>';
}/* end function do_header()
*/
/*
function prep($log)
convert mirror.log (created from corzbax script) into valid scriptable sftp commands...
*/
function prep($log_data) {
global $cmd_prep, $mirror_dir, $ftp_web_dir, $ignore_files, $make_dirs, $upload_from, $special_file, $special_command;
// check for user slashes..
if (substr($ftp_web_dir, 0) == '/') $ftp_web_dir = substr($ftp_web_dir, 1);
if (substr($mirror_dir, -1, 1) == '/') $mirror_dir = substr($mirror_dir, 0, strlen($mirror_dir)-1);
if (substr($upload_from, -1, 1) == '/') $upload_from = substr($upload_from, 0, strlen($upload_from)-1);
$dirs = array();
$special = array();
$log_file = explode("\n",$log_data);
// remove the ignored files..
while (list($key, $value) = each($log_file)) {
$value = trim($value);
if ($value) {
foreach($ignore_files as $remove_dis) {
if (strstr($value, $remove_dis)) {
unset($log_file[$key]);
break;
}
}
}
}
// special commands for certain files..
// best waiting until after we have filtered out the unwanted files
reset($log_file);
while (list($key, $value) = each($log_file)) {
foreach($special_file as $iamspecial) {
if (stristr($value, $iamspecial)) {
array_push($special, trim($log_file[$key]));
}
}
}
$log_file = array_unique($log_file); // remove duplicates
sort($log_file); // sort alphabetically
// folders..
// [we add mkdir commands for these]
$i=0;
reset($log_file);
foreach($log_file as $line) {
// this is a directory (no "." in its name)
if ((!stristr(substr($line,strrpos($line, "/")),".")) and (strlen($line) >= 1)) {
$dirs[$i] = $line;
unset($log_file[$i]);
}
$i++;
}
// we will use this for all the transformations..
$mirror_dir_slashed = preg_quote($mirror_dir, "/");
// the magic..
$i=0;
$commands = array();
foreach($log_file as $line) {
$commands[$i] = preg_replace_callback("/'$mirror_dir_slashed(.*)'(.*)'(.*)'/", "make_commands", $line);
$i++;
}
// need to rsort the dirs here, or else subdirs get created before their parents. That won't work!
rsort($dirs);
// add the mkdir commands to front of the commands list..
if ($make_dirs) {
foreach($dirs as $line) {
$new_dir = preg_replace("/'$mirror_dir_slashed(.*)'(.*)'(.*)'/", "\n${cmd_prep}mkdir \"$ftp_web_dir\\1\"", $line);
array_unshift($commands,"\n$new_dir\n"); // mkdir statements go at the start, of course.
}
}
/*
add the special actions to the end of the commands list..
At the end so that the files we perform actions on, exist. */
foreach($special as $line) {
// [] = is basically the same as array_push()..
$commands[] = preg_replace("/'$mirror_dir_slashed(.*)'(.*)'(.*)'/", "\r$special_command \"$ftp_web_dir\\1\"", $line);
}
return implode("\n",$commands);
}
// convert log output into valid sftp commands
// if the line is too long, split it into cd & put commands..
// (198 character max for regular ftp, inc. both parts & command & quotes)
function make_commands($array) {
global $cmd_prep, $delete_first, $split_long_lines;
if ($delete_first) {
$this_command = "\r-rm \"".$GLOBALS['ftp_web_dir'].$array[1]."\"\r";
} else {
$this_command = "\r";
}
// LONG line..
if ($split_long_lines and strlen($GLOBALS['upload_from'].$array[1].$GLOBALS['ftp_web_dir'].$array[1]) > 190) {
$this_command .= 'cd "'.$GLOBALS['ftp_web_dir'].dirname($array[1])."\"\r".
$cmd_prep.'put -p "'.$GLOBALS['upload_from'].$array[1].'" "'.basename($array[1]).'"';
} else {
// regular command..
$this_command .= $cmd_prep.'put -p "'.$GLOBALS['upload_from'].$array[1].'" "'.$GLOBALS['ftp_web_dir'].$array[1].'"';
}
return $this_command;
}
/*
function show_script() */
function show_script() {
global $site_footer;
echo <<<ERE
<div class="content">
<h2>the shell scripts..</h2>
Here are the actual scripts that I use to sync corz.org with its development mirror on my Linux box.
Essentially, we log changed local files, and upload only those files. The mechanism for all this is a
simple site backup; a mirror of my mirror site! There are many advantages to this technique..<br />
<br />
I don't have to keep tabs on which files I've edited; and I tend to edit lots of
different site files in one session; the backup script does this for me. Only new
and changed files are backed up, thereby indicating which files are new or changed!<br />
<br />
You won't be able to use this "as-is", it will need tailoring to your environment.
Still, that two minutes will save you hours and hours and hours in the future, so
it's worth it. All the main scripts are included in the zip package. If you find
any bugs, please <a href="https://corz.org/corz/contact.php"
title="email me about this script">report</a> them. ta. okay, here goes..<br />
<br />
<code><span class="com-comment">file: /usr/local/bin/cb<br />
<small>(there's also a non-archiving version included, "<a title="though you may
want to rename it to 'c'; faster to type" id="help-rename-shit">cbl</a>", for general update use)</small></span></code><br />
<br />
<pre class="code-space"><span class="textview-comment">#!/bin/sh</span>
<span class="textview-comment"># corz.org mirror backup script.. (run as daily cron)</span>
<span class="textview-comment"># this runs on a Linux box, where both the real dev mirror, and the
# backup directory live (though the latter is on a separate physical
# drive - it is mounted in /bax) you could backup to a different
# machine, though remember to ensure the share/network-drive is
# mounted first.</span>
<span class="textview-comment"># this script updates \$backupdir with all changed files from \$devmirror</span>
<span class="textview-comment"># all this activity is logged to a file which we parse into sftp commands</span>
<span class="textview-comment"># with the "website-sync" php script, running on a web server somewhere.</span>
<span class="textview-comment"># © corz.org 2003-></span>
<span class="textview-comment"># prefs..</span>
<span class="textview-comment"># log to this file <small>(the content we will later convert into sftp commands)</small></span>
logfile="/home/corz_mirror.log"
<span class="textview-comment"># crucially, this must match the \$mirror_dir variable in the php script..</span>
devmirror="/home/cor"
<span class="textview-comment"># this script will recreate my dev mirror inside here..</span>
backupdir="/bax"
<span class="textview-comment"># a gzip of the whole lot will go here..</span>
storedir="/bax/www"
<span class="textview-comment"># make things readable..</span>
echo >> \$logfile
echo "corz.org backup script: "`date "+%Y.%m.%d-%H.%M"`.. >> \$logfile
echo
echo >> \$logfile
echo "corz.org backup script (full version - with archiving)"
echo
cd \$devmirror
cd ..
<span class="textview-comment"># tar and gzip the current mirror..</span>
echo "archiving dev mirror.."
tar cfz corz-bax.tar.gz \$devmirror/ >> \$logfile
<span class="textview-comment"># copy the tar.gz to the backup directory with unique dated filename</span>
echo "storing archive backup.."
mv corz-bax.tar.gz \$storedir/`date "+%Y.%m.%d-%H.%M"`_corz.org.bax.tar.gz >> \$logfile
<span class="textview-comment"># copy over all the newer files (Linux cp rocks! Apple take note.)</span>
echo "copying changed files to mirror.."
cp -d -f -p -r -u -v \$devmirror \$backupdir >> \$logfile
echo
echo >> \$logfile
echo "corz.org backup script complete"
echo
<span class="textview-comment"># we now have an up-to-date backup</span>
exit 0
<span class="textview-comment"># fin</span>
</pre><br />
Whenever you want to backup your dev mirror, or sync it with your live site, you run
the backup script. Both versions do the exact same backup, updating only new or changed
files. The above example also gzips the entire backup and stores it somewhere. Most likely, you will
want to set that one as a daily/weekly cron job..<br />
<pre class="code-space">
<span class="textview-comment"># 5.45am corz.org mirror backup [now weekly, monday]..</span>
45 5 * * 1 /usr/local/bin/cb >> /var/log/cron 2>&1
</pre>
The 'lite' version (cbl) is for manual use. It simply performs a backup, updating only new or changed files,
and makes no zip. When you are looking to do a quick sync, use the lite version. I often
use the lite version dozens of times a day. The first time I run it, I see that there are also log entries
from the earlier cron backup, and they are simply incorporated into the next upload. <br />
<br />
If you use it as often as I do, you may want to rename it to just 'c', or some other single letter that
doesn't clash with an existing executable in your \$path, so it's quicker to type.<br />
<br />
When you run your backup script ('cb', or 'cbl' - or whatever you named them), the output will be
captured to a log. The log's contents are then fed into the php web sync tool (which your local Apache will be
happy to server up for you at..<br />
<br />
<code>http://localhost/website-sync-linux.php</code><br />
<br />
or wherever. The output from the web tool is then pasted into a simple sftp upload script something like this..<br /><br />
<h3>NOTE: This section is outdated, everything is sftp these days, and website-sync is setup to output SFTP commands. See below for more details.</h3>
<div class="clear-tiny"></div>
<code><span class="com-comment">file: <a title="in your home folder" id="help-UNIX-home-folder">~/up</a></span></code><br />
<br />
<pre class="code-space">
<span class="textview-comment">#!/bin/sh</span>
<span class="textview-comment"># upload new files to website..</span>
HOST='ftp.mysite.com'
USER='username'
PASSWD='password'
ftp -n \$HOST <code><<FTP_QUIT
quote USER \$USER
quote PASS \$PASSWD
binary
verbose
<span class="textview-comment"># paste converted ftp commands right here (replacing this line)</span>
quit
FTP_QUIT</code>
exit 0
<span class="textview-comment"># fin</span></pre>
<br />
Run the "up" script. That's it..<br />
<br />
<strong>Your wesite and its mirror are one!</strong><br />
<br />
<h3 id="linux-sync-from-windoze">Running all this from Windows<span class="sup">®</span>..</h3>
Although the scripts run on a Linux box, you can easily mastermind the whole show from a Windows workstation
(I've had a couple of questions about this, and I'm here anyway, because this is where EditPlus is!).
Here are some suggestions on how to get the whole thing down to a
3.62 second operation...<br />
<br />
Keep <strong>shortcuts</strong> in a toolbar, or somewhere handy, to these three important things
(which you will use, in order)..<br />
<ul>
<li>The mirror log <small>(output from the update script is in here)</small></li>
<li>URL shortcut to a local copy of the website sync tool (you are reading the other half of it).</li>
<li>The "up" shell command on your Linux box <br />
<small>You will need to edit this file, you see, add your ftp commands</small>.</li>
</ul>
I have shortcuts to all sorts of resources on all sorts of machines, right here in my Windows desktop and toolbars.
A handy shortcut can save LOTS of time, and only takes a moment to make and place.<br />
<br />
<strong>Tools</strong> you need:<br />
<ul>
<li>Text editor <small>(For Windows, I usually recommend <a href="http://www.editplus.com/" title="Editing text on Windows? Get this."
id="link-editplus">EditPlus</a>).<br />
While you are working on the site, it's easier to leave the output log and <code>up</code> script open in your text editor.</small></li>
<li>Telnet/SSH client <small>(to run a shell on your Linux box - I definitely recommend <a href="http://www.chiark.greenend.org.uk/~sgtatham/putty/"
title="Very professional, fully featured and FREE SSH/Telnet client." id="link-putty">Putty</a>).<br />
While you are working on the site, it's easier to leave your Linux shell running - it was open anyway, right?</small></li>
<li><a href="http://www.spreadfirefox.com/?q=affiliates&id=126442&t=187"
title="Firefox. I just can't think in these other browsers"
id="Firefox-link" target="_blank" rel="noopener noreferrer">Web Browser</a>
<small>(with your local website sync page in on of its tabs).</small></li>
</ul><br />
<h3 id="simple-sync">A few simple steps..<br />
<small><small>The whole operation takes only a few seconds, on <em>any</em> platform. Here's how..</small></small></h3>
<ul>
<li>Login to a shell on your Linux box, run the "lite" backup command; that is; type "cbl<enter>" <small>(no quotes)</small>.</li>
<li>Switch to the backup log in your Text editor, <small>(your text editor should have automatically reloaded the file,
which now contains the log output we will process)</small> and cut the log output into the clipboard.</li>
<li>Switch to your browser, and paste the output into website sync tool, click 'make ftp commands!'</li>
<li>Switch to your text editor and paste the output into the <code>up</code> script, save that.</li>
<li>Back in your Linux shell, run the <code>up</code> script. You're done.<br />
<p>With practice, you can do the whole thing in under three seconds!</li>
</ul>
<br />
<h3 id="ftp-security">A note about ftp security..</h3>
The ftp "up" script (above) is the "simple-and-insecure" version of things. It's easy for anyone to get
working, and does the job. But there are least three things you could do to increase the security here..<br />
<br />
Firstly, it' not very smart to keep login passwords inside files that can potentially be viewed by anyone.
So you could start by keeping the script somewhere inside your home folder, as in the example above.
But that's not very convenient, and it's not good organisation, either. Here's a better idea..<br />
<br />
Remove the "-n" switch, and all the user/pass commands from the ftp script, and instead allow ftp to use
auto-login; with your <code>~/.netrc</code> file. This is simply a plain text file in your home folder named
<code>.netrc</code>. It has single line entries that look something like this..<br />
<br />
<code>machine ftp.mydomain.com login MyName password MyP4\$\$w0Rd</code><br />
<br />
This has obvious security benefits. Do <code>chmod 600 ~/.netrc</code> to make it readable by only you, and then
feel free to leave ftp login scripts lying around all over the place!<br />
<br />
<h4 id="automatic-sftp-script">Automatic SFTP Scripting</h4>
Rather than use plain ftp, it's possible you have access to SSH on your web
server, and therefore; SFTP. It's not quite so simple to automate SFTP as it
is ftp, but it is doable, and this first sftp script, using "expect", works
very well, though it's not quite as secure as the second method. Without
going into too much detail, you'll need two scripts; a launcher,
which will "spawn" an interactive expect session..<br />
<br />
<code><span class="com-comment">file: /usr/local/bin/sftp/up-launch</span></code><br />
<br />
<pre class="code-space"><span class="textview-comment">#!/usr/bin/expect</span>
<span class="textview-comment">#</span>
spawn sftp -C -b /usr/local/bin/sftp/up sample@example.com
expect "password:"
send "MyP4\$\$w0Rd\\n";
interact</pre><br />
Which we will script (use a text file to do the interactive parts, instead of us) with a simple list of
commands something like this..<br />
<br />
<code><span class="com-comment">file: /usr/local/bin/sftp/up</span></code><br />
<br />
<pre class="code-space">progress
put "/home/cor/blog/inc/cbparser.php" "/blog/inc/cbparser.php"
put "/home/cor/blog/inc/cbguide.php" "/blog/inc/cbguide.php"</pre><br />
Note the "progress" at the start. Apart from that, it's just a list of regular SFTP commands. For more
details, check out the man pages for <a href="http://www.linuxcommand.org/man_pages/expect1.html"
title="I hope they don't mind me hot-linking to their man pages, my host dunt have it installed!"
id="expect-man-page-link">expect</a> and <a href="http://node1.yo-linux.com/cgi-bin/man2html?cgi_command=sftp"
title="This is a nifty man page viewer - looks familiar!" id="sftp-man-page-link">sftp</a>.
Of course, you'll also need these two programs installed for all this to work. <br />
<br />
And of course, now you've got passwords inside your sftp scripts! <a title="chmod is your friend"
id="help-chmod-that-thang">Aarrrghhh!!!!</a><br />
<br />
<h4 id="automatic-sftp-script-better">Better Automatic SFTP Scripting</h4>
These days, I prefer to use RSA keys to authenticate with the server. While
more work to setup, they are more secure and make automatic scripting a
breeze. I won't go into details here. You can Google that.<br />
<br />
Once you have your keys setup, you simply do something like this..<br />
<br />
<code><span class="com-comment">file: /usr/local/bin/sftp/up</span></code><br />
<br />
<pre class="code-space">sftp -b commands.txt MyUserName@MyWebHost</pre><br />
<br />
And inside commands.txt, you drop your regular sftp commands (as created by this tool)..<br />
<br />
<code><span class="com-comment">file: /usr/local/bin/sftp/commands.txt</span></code><br />
<br />
<pre class="code-space">put "/home/corz.org/blog/inc/cbparser.php" "httpdocs/blog/inc/cbparser.php"
put "/home/corz.org/blog/inc/cbguide.php" "httpdocs/blog/inc/cbguide.php"
etc..</pre><br />
Have fun!<br />
<br />
ERE;
echo '
;o) <a href="https://corz.org/corz/contact.php" title="" id="link-external-corz-sontact" target="_blank" rel="noopener noreferrer">Cor</a><br />
<br />
</div>
<div class="space-small"></div>';
toplinks_out();
if (file_exists($site_footer)) { include_once $site_footer; }
}/* end function show_script()
*/
// my_function..
function toplinks_out($at_tool=false) {
$href_add = '';
$alink_str = 'test conversion tool';
if ($at_tool) {
$href_add = '?showscript=true';
$alink_str = 'view the scripts';
}
echo '<span class="toplinks"><a href="',$_SERVER['PHP_SELF'],$href_add,'" title="',$alink_str,' page">',$alink_str,'</a> • <a href="https://corz.org/engine?section=php&download=website-sync-linux.zip" title="download the zip of the website-sync Linux Version">download the zip</a> • <a href="website-sync-mac.php" title="check out the mac version">mac version</a></span>';
}
/*
version info..
0.9.2
More additions for sftp scripting use.
Added ability to ignore sftp errors (a good idea)
Improved documentation.
Embedded styles.
0.9
+ The magic "create ftp command" regex has been moved into a callback
function so that I could get inside the string..
In daily usage, I have found that occasionally, the ftp commands are so
long that they don't action, but instead give an "input line too long"
error, which to spot, you would need to be watching the console
carefully. As the "delete" command probably actioned just fine, you now
have a file missing from your site.
What the sync script now does is split long commands (where the total
length will be over 198 characters, or thereabouts - as deduced from
tests) into two commands, one "cd" into the directory, and then a
shorter "put" command, using only the file name. In practice, this is a
good solution.
Upgraded to sftp - saves me/you having to edit the script, because
everyone uses sftp these days, right. Added another info section about
sftp, this time pointing to shared keys.
~ cp no longer uses wonky ` quote. Switched everything to apostrophe. "'"
0.8.5
* fixed a bug in the special command processing.
0.8.4
* moved the ignore files pref to the top, with the other prefs
(AT LAST! I hear you say!)
+ Added option to not make directories, if you need that.
0.8.3
* vastly improved built-in documentation/tutorial side of the script
0.8.2
* made the "special file" into an array of files, so you can specify
multiple extensions, file names, or whole paths.
0.8.1
* improved css and link code
0.8
* New XHTML version of the tool
* Upgraded the tutorial, included loads more info.
* Removed confusing Mac-centric code from tutorial.
0.1 - 0.7
no version info kept
*/
?>