Version 2.5
Copyright ©1997-2001 Emmett Gray. All rights reserved.
Fetch is a registered trademark of Dartmouth College.
Helpful hint: use the "Find" command in your browser to look for a particular word or phrase in this manual.
Introduction
1. Installation
A. Quick Start
B. Detailed Installation Instructions
C. PPP Note
2. Settings
A. The "Settings" file
i. Settings file entries
ii. Other related issues
iii. Excluding files
iv. Managing multiple sites
B. FetchProfile
3. Methods
4. Operation
A. General
B. FetchNewLocal:
i. What It Does
ii. The "To Upload" file
C. FetchUp
i. What It Does
ii. The changesFile
Truncating the changesFile
iii. Interrupted Uploads: FetchUp Auto-Correction
iv. The messageFile
v. The "Activity Log" file
vi. Simultaneous Uploads
vii. Adding to an upload in progress
D. FetchChanges
E. FetchDown & The "To Download" file
F. FetchTimeStamp, FetchMe
5. The Utilities
A. FetchCompare
B. FetchContinual
Unattended operation
C. FetchDelete
D. FetchHarvest
Unattended operation
E. FetchMeNot
F. FetchStructure
6. Miscellaneous
A. "Secret" Files/Areas: Multiple Fetch-O-Matic Suites
B. Stopping Execution
C. Error Messages
"Apple event timed out"
"Couldn't find..."
"Error number..."
"Local changesFile has data not present in the remote file..."
"Not connected"
"Out of memory"
"<<script>> doesn't understand the <<event xxxx>> message"
"That file or directory is unavailable or nonexistent"
"The application Standard Additions has unexpectedly quit..."
"This file may be nonexistent"
"Tried to create temporary file..."
"Unable to connect to remote server"
"Unexpected server response"
"Unix or DOS Linefeeds..."
"Varable logFile not defined"
"You must install Remote Access"
Other (strange-looking) AppleScript errors
D. Modem connection problems and crashes with OS 9.1
E. Memory Allocation
F. Fetch Preferences
7. Frequently Asked Questions
8. Obtaining Software
9. Registration and Payment
10. Legal Notice
Appendix A: List Files settings
Appendix B: Required Scripting Additions
Appendix C: Contents of Fetch-O-Matic Folders
Appendix D: Revision History, Updating Notes
Appendix E: Credits and Thanks
Fetch-O-Matic is a suite of applications for automated website maintenance via FTP. Fetch-O-Matic allows web authors to freely create and update files locally without needing to keep track of their work, and then automatically catalog the alterations and update the website, depositing files in the proper locations (and creating new directories if necessary). The suite also includes applications for comparing and synchronizing the remote site with local mirrors, and an application to "harvest" files delivered to a remote directory at a specified interval or on a schedule. Fetch-O-Matic is fully compatible with Macromedia Dreamweaver, Adobe GoLive, and similar site-building tools.
Fetch-O-Matic differs from other available synchronization and maintenance schemes in that it does not require any local machine to have a complete site copy. For example, one might decide to grab a few pages and change them (and/or add some new files) from a PowerBook. Later, the site copy on the primary machine can be quickly brought up-to-date with just two clicks - without having to remember what was changed or open the PowerBook.
Fetch-O-Matic is easy to use. Novice web authors in search of a simple way to update a personal web page should not be intimidated by its power. Experienced hands will find that previously dreaded operations such as multi-file search and replace, with the subsequent headache of tracking down and uploading scattered altered files, are rendered painless. Transfer errors such as uploading/downloading to the wrong directory are a thing of the past. If several people are maintaining a site they are protected from treading on each other's updates.
Fetch-O-Matic was conceived and developed to fill a need in servicing a large site (http://www.filmscouts.com). Uploads and downloads of thousands of files at once have been performed with ease. But even on a site with just a few pages, Fetch-O-Matic will ease the updating chores.
Fetch-O-Matic requires Dartmouth's FTP client "Fetch", AppleScript, the scripting addition Jon's Commands, and, optionally, Alessandro Levi Montalcini's "List Files" (providing a significant speed improvement for large sites). It runs on any Mac OS machine with reasonable (8mb+) RAM running System 7, 8, or 9. It runs in the background. Fetch-O-Matic is shareware with a single-user license fee of $20. There is a free 30-day trial period.
1. You must have Fetch installed, must have a copy of Jon's
Commands, and, optionally, List Files. The AppleScript extension and
Apple's standard Scripting
Additions are also required. Also required is a text editor or
word processor of your choice. (Microsoft Word is problematical. Use
BBEdit, Tex-Edit, Smile, or even SimpleText instead).
2. Put Jon's Commands, Password, setDate, and perhaps a PPP component
(see PPP note) in your Scripting Additions
folder. (It is advisable to reboot after this step; although
scripting additions are not loaded at startup, there is an
AppleScript cache and new additions can be missed). Under System 8.0
and later, Scripting Additions are automatically placed in the proper
location when dropped on a closed System folder.
3. Open, configure, and save the "Fetch-O-Matic Settings" file;
instructions are in that file.
4. Run "FetchProfile" and do a "Full" test (if using List Files, also
select the option to set List Files prefs, then follow the presented
instructions).
5. If FetchProfile's result file ("F-O-M Profile...") reports only a
single problem - a missing "<sitename> filelist" file, normal
at this point - you are ready to use Fetch-O-Matic (unless
using OS 9.1 as noted in the next paragraph).
OS 9.1 "Variable not defined" bug and workaround. There is an acknowledged bug in OS 9.1 with AppleScript versions 1.5.5 and 1.6 (and possibly later versions). To avoid it, do as follows: before you first run any of the Fetch-O-Matic applications (other than FetchProfile), launch Fetch, your chosen text editor, and, if using it, List Files. Otherwise, you will get a "Where is..." dialog asking you to locate these helper applications, and after that, the Fetch-O-Matic application will quit with the error message "Variable logFile not defined." If the helper applications are already running, Fetch-O-Matic applications will recognize them and this dialog and error will not occur. This workaround is required only on first run; thereafter, the helper application locations will be remembered. This situation is due to a bug which AppleScript version 1.6 (released late March 2001) claims to have fixed, but in fact did not.
The Fetch-O-Matic folder may be placed anywhere on any drive, and this folder may be renamed as desired (useful if you run more than one copy of Fetch-O-Matic for multiple sites). However, you should not change the names or structure of the sub-folders, applications, or text files (doing so will cause execution errors). You may of course create aliases to folders and their contents and rename the aliases as you like. A complete list of Fetch-O-Matic files appears in Appendix C.
The AppleScript extension (1.1 or later) must be installed, along with the standard Apple scripting additions, in order to use Fetch-O-Matic. (AppleScript and the additions are included with Apple system software 7.5 and later. The latest version can be found at http://www.apple.com/applescript). Fetch-O-Matic is fully compatible with all known versions of Mac OS Systems 7, 8, and 9 and should run on any Mac OS hardware.
You will need to get three additional widely available files before you can use Fetch-O-Matic: a) "Fetch" 2.7 or later (3.0.3 is current as of this writing; beta 4.0b7 works and so should the release version 4), by Jim Matthews ($25.00 shareware); b) "Jon's Commands" 1.8 or later (2.1 is current), by Jon Pugh (free for private use; donations accepted. Note: type "jons", not "Jon's", when searching for this file); and c) "List Files" 2.6 or later, by Alessandro Levi Montalcini ($10.00 registration fee). (Having and using "List Files" is not required, but it speeds things up and is recommended). All three can be found on the Fetch-O-Matic download page (http://www.filmscouts.com/software/fomdown.asp), Info-Mac (http://hyperarchive.lcs.mit.edu/HyperArchive/SearchForm.html), Shareware.com (http://shareware.cnet.com/), and many other Mac http and ftp software sites.
After getting these files, if you are using System 7.x then put Jon's Commands, PassWord, setDate, and perhaps MacPPP Control or FreePPP Control (see PPP Note) in the Scripting Additions folder in the Extensions folder in your System folder; under System 8.x and later, just drag these to the closed System folder and they will be correctly installed (the Scripting Additions folder was "promoted" to System folder root level under System 8). It is advisable to reboot after this step; although scripting additions are not loaded at startup, there is an AppleScript cache and new additions can be missed.
Install Fetch and List Files anywhere on any drive. Fetch should be configured normally and you should make sure you can log on and transfer files manually before running Fetch-O-Matic. Three significant Fetch preferences, to overwrite files with conflicting names, disable any name changes, and ignore files beginning with a period (i.e, the current or parent directory, if your server shows it) are set automatically by Fetch-O-Matic; your prior preferences for these options are saved on launch and then restored when Fetch-O-Matic applications quit. You won't need to create any bookmarks or enter passwords since the signon process is handled by Fetch-O-Matic. Secure FTP access (via SKey and other one-time password schemes) is automatically supported directly by Fetch 3.0 and later, and thus compatible with Fetch-O-Matic as well. See more information below under Fetch Preferences.
If you have an ISDN, DSL, T1, cable modem, or other such always-on connection to the Internet, set the "usePPP" setting to "N". Additionally, Fetch-O-Matic can be used with applications such as SurfDoubler and IPNetRouter (although it will not close connections when used with them); in these circumstances "usePPP" should also be set to "N". The following PPP information is not relevant when using the "N" setting.
If you are using Open Transport TCP/IP with the PPP control panel or the more recent Remote Access control panel, you should already have either the "PPP Commands" or the "Remote Access Commands" Scripting Addition in your Scripting Additions folder. If so, you may select the "O" option on the "usePPP" line in the Settings file and enable Fetch-O-Matic to close a connection (hang up) on terminating. The terminating behavior is set in the "closeLink" line and is described in the Settings file. (Whether you have these Scripting Additions or not, Fetch-O-Matic can make connections automatically if "auto-connect" is enabled in the PPP or Remote Access control panel). If running an older version of the OS and you do not have either of these scripting additions, get a more recent version of OT/PPP/Remote Access. This is available for download from official Apple software sites, and is included in System 8.1 and later.
The FreePPP Control Scripting Addition offers the same functionality as PPP Commands/Remote Access commands and works with FreePPP 2.6 and later, under both Open Transport and the older MacTCP. If using this PPP software, set the "usePPP" line in the Settings file to "F" and install this addition. (The folder "Test & Doc - FreePPP Ctrl" contains the original documentation and a test application for this Scripting Addition). FreePPP 2.6 is free, is compatible with all system versions through 8.6 (and maybe with 9.x, but has not been tested therewith), and offers some advantages over Apple's implementation. For more information visit the official FreePPP page.
MacPPP Control is similar, but works only with MacTCP and not with Open Transport, and works only with versions of MacPPP and FreePPP up to and including version 2.5v2, but not beyond. (MacPPP 2.5 can be found on the various official Apple software sites). If using this generation of PPP software, set the "usePPP" line in the Settings file to "M" and install this addition.
In the "Fetch-O-Matic Auxiliary" folder there is a text file named "Fetch-O-Matic Settings". This file must reside in the Auxiliary folder, which must remain within the main Fetch-O-Matic folder, and you may not change the names of the Auxiliary folder or the Settings file.
The Settings file is where data shared by all the Fetch-O-Matic applications is stored. Open this file with a text editor or word processor and enter the required information, replacing the dummy values, and then save the file. Instructions for entering the information are contained in the Settings file itself. Once you have done this, run the FetchProfile application, which will check your settings and create necessary files.
The password setting has two lines, one for a password-saving preference and one for your FTP logon password. There are three options, with convenience inversely proportional to security: if you elect to enter your password here your security level is zero, but you won't be bothered by any dialog boxes. The middle ground, selecting a letter other than "N" for the passWordSave option, will store your password (encrypted) and you will be asked only once for it by each Fetch-O-Matic application that needs it; however, anyone with access to your machine can log on - although this won't enable them to know your password. The only secure option is to set the passWordSave option to "N" and enter nothing (other than spaces) in the passWord line. But then you will then be queried for your password every time you run an application that needs it. Further instructions (i.e. regarding changing your password if stored) are in the Settings file.
There are two more text files which must exist, referred to in the Settings file (and in the following discussion) as the "changesFile" and "messageFile". These may be renamed and/or moved, but must exist WITHIN YOUR SITE on both local and remote machines. (You may use the default names for these files as they appear in the Settings file, which will place them at the root level of your site). The FetchProfile application will automatically create these files (if they don't already exist). These files are used by Fetch-O-Matic to keep a record of what activity has taken place on the remote host. The changesFile is written to automatically. The messageFile is just a note pad which you can use as you see fit, and isn't consulted about action to take, but if the remote copy has changed you are notified and the updated version is downloaded and opened - an automatic alert.
The "autoRemove" option will allow both FetchUp and FetchDown to automatically update the To Upload and To Download files by removing successfully transferred files. This is a fail-safe option; no file is removed until its transfer has been successfully completed. For uploads, the To Upload file is only altered after the completion of a cycle of uploads; for downloads, the To Download file is re-written after every single download (this difference has to do with the fact that uploads need also to be entered in the changesFile, while downloads do not). If autoRemove is enabled, it is also possible to add items to the To Upload file during an upload.
The "reconnect" setting permits you to enable or disable an attempt by Fetch-O-Matic to automatically resume an interrupted transfer session, due to a dropped connection or similar error. If reconnection is successful, any file partially uploaded or downloaded will be resumed from the beginning, and the session will then proceed with remaining files in the queue. Although most users will want to allow this, the option to disable is provided since there may be circumstances, such as downloading huge files, for which reconnection is not desired, since it will conflict with Fetch's potential to (manually) resume an interrupted download of a large file. See the Fetch Help topic "Resume Download" for more information on Fetch's resumption capabilities.
The "ignoreChanges" setting permits you to ignore the remote changesFile. However, doing this will disable significant features of Fetch-O-Matic, specifically, those that relate to site maintenance from more than one location. The only advantage to doing this is that you'll save a little time uploading and downloading the changesFile, which otherwise gets transferred frequently. You should do this ONLY if a) no one else is uploading files, and b) you upload from one location only. (If you set ignoreChanges to "Y" before running FetchProfile in a full test, it will not place these files on the remote site as it would otherwise do - if they don't already exist there).
Dual-server settings: Fetch-O-Matic can transparently cope with a dual-server environment (where, for example, streaming video is served from a separate machine which appears to be a single location to browsers, but which actually has its own IP address, logon, and password). On your local machine, directories actually on the secondary server remain in their logical locations as if on a single machine. You need merely enter the settings parameters for such a machine just as was done for the primary machine. The altDirHead setting specifies the start of the names of any directories on your local mirror whose contents are destined for the alternate server. The altDirHead can be as short as a single character or it can encompass a path of any length. Any directory (and its subdirectories) whose local path (i.e. within the local root) commences with altDirHead will relate to the alternate server instead of the primary one. For example: in the root of your local mirror there are 3 directories named video, video1, video2. If you set altDirHead to video then anything in all of these directories will be directed to the alternate server. If you set altDirHead to video1, only the contents of video1 will be so directed. This works of course for downloads as well as for uploads, and in the applications FetchCompare and FetchStructure. (The altPassWord storage is handled according to the preference set in the passWordSave line for the primary server).
Fetch-O-Matic sets only the Fetch preferences it requires; see Fetch Preferences below.
The text files "Activity Log", "To Download", and "To Upload", must be in the main Fetch-O-Matic folder and may not be renamed. These are empty files to start with, but they must exist. The similarly empty "To Delete Local", "To Delete Remote", and other text files in the Utilities folder must also exist and not be moved or renamed.
In the Auxiliary folder, there will appear a "<sitename> filelist" file created by either FetchNewLocal or List Files (if used), whose name will be determined by the name of your local site copy (the "localHome" setting). (If using List Files, you must configure it to save its output file in the Auxiliary folder. This should be done by running the FetchProfile application, which simplifies the setup of List Files). This file (and some others) won't exist until you run the application which creates it. When you run FetchProfile before running FetchNewLocal (as you should do), it will fail to find this file, which doesn't yet exist. This error is normal.
In order for the FetchNewLocal application to work with List Files, you must first set List Files up with the correct preferences. The FetchProfile application will take care of this for you, and will prompt for the destination folder of the output file, and that file's creator application (text editor or word processor). This output file must be saved in the "Fetch-O-Matic Auxiliary" folder. To set this option, select "File Saving... "under the "Options" menu in List Files and navigate to the correct destination. There is also an appendix at the end of this document with a series of illustrations of the proper settings for the various List Files options. Your set up should be as illustrated, and also have a checkmark next to the menu item "Skip Save Warnings" under the "Options" menu.
The file "List Files Prefs", in the Auxiliary folder, should not be moved. Each Fetch-O-Matic folder maintains its own copy of the List Files preferences file so as to allow you to use List Files for other things, including running other copies of Fetch-O-Matic for different sites. These preferences are swapped out with the one in the System folder's preferences folder each time FetchNewLocal or FetchCompare runs. If you are already using List Files, as long as you use FetchProfile to set the preferences for Fetch-O-Matic, your previous preferences will be maintained. If you have not used List Files before but now wish to use it independently of Fetch-O-Matic, you can set up any other preferences you like and these will be preserved by Fetch-O-Matic.
There are three selection methods for excluding files from transfer: exclusion by directory, file extension, and filename. Exclusion is useful if Fetch-O-Matic is used in conjunction with Macromedia Dreamweaver, Adobe GoLive, and similar site-building tools which add directories to your site. The three exclusion methods may be used concurrently and they permit overlap of criteria. The mechanism tests each file against all the exclusion criteria and won't add any files that fail a test to the list of files to transfer. (Because the "To Upload" and "To Download" lists are not examined for exclusions thereafter, it is possible to manually enter an otherwise excluded file in these lists so as to have it transferred).
The "noUploadFolders" setting permits the identification of one or more folders in your local drive whose contents, including any subfolders and their contents, will be ignored. No contents of skipped directories will be added to the "To Upload" list. You may create folder "wild card" entries by entering a partial folder name instead of a full path, which places any folders sharing the specified stem onto the excluded list. Detailed instructions and examples are found in the Settings file. This setting applies to FetchNewLocal and FetchContinual.
The "noDownloadFolders" setting works just like "noUploadFolders" , but in the opposite direction. This setting is used only by the application FetchHarvest.
The "ignoreFiles" setting directs all Fetch-O-Matic applications to ignore files with certain extensions (typically such as .dir, .cst, .lck, .psd, .pla, .plx, .snd and so on). No files with the designated extensions will be added to a transfer list. Multiple extensions may also be used in the exclusion list and are considered as one. For example, if ".sit.hqx" is excluded, a file with extension "zip.hqx" would not be excluded, but if ".hqx" is excluded, all files whose final extension is .hqx will be excluded, and both ".sit.hqx" and ".zip.hqx" are in that category.
Finally, individual files may be excluded in either direction. This is handled not by the Settings File but by placing the names of such files in the "Never Upload" or "Never Download" lists, two text files which are found in the Fetch-O-Matic Utilities folder. Use a text application to edit these files. The easiest and least error-prone way to enter filenames is to copy and paste items from the "To Upload" or "To Download" lists (or from the Activity Log) after you discover them there. This will ensure that the correct format is used in these "Never" files. (Entries should use colon delimiters. No path should be specified for files at the site root level; specify a path only if deeper. All entries will be excluded whether or not there are blank lines before them in the "Never" file in which they appear). Files so excluded will still be transferred if appearing in the To Upload or To Download lists; the exclusion occurs only at the point of creating the To Upload or To Download lists.
If you are managing multiple sites you must keep a Fetch-O-Matic folder for each site. Each Fetch-O-Matic folder must contain the applications (not aliases) and a Settings file, and must of course relate to a separate complete set of Fetch-O-Matic text files as well. (You may of course use aliases to any of the Fetch-O-Matic applications or files). For managing separate areas of a single site, see the section "Secret Files/Areas".
FetchProfile eases the setup process by checking your settings and creating needed files. It does a dry run with the configuration in your Settings file, and looks for problems such as missing files, applications, and Scripting Additions. It will create the local and remote changesFile and messageFile if they don't already exist. It also leads you through the setup of List Files preferences, if you are using List Files.
You should run this application after entering your settings and before you attempt to run any other Fetch-O-Matic application. At this time it will report a missing "<sitename> filelist" file, which is normal. After you have run FetchNewLocal, run FetchProfile again, and you should get a report stating "No problems found".
You should also run FetchProfile again after installing List Files if you do that at a later time. You should run this application whenever you want to check your current settings, such as when running for the first time from another machine, or if some problem develops later on.
FetchProfile offers options on launch. It then proceeds with the selected actions and creates a file with title "F-O-M Profile <date>" in the Fetch-O-Matic Utilities folder. This will show what it did and what it found. This result file is not used subsequently and may be discarded.
In its principal mode of operation, Fetch-O-Matic functions by automatically generating lists of files requiring action, and storing these lists in the "To Upload" and "To Download" files. Whether uploading or downloading, Fetch-O-Matic operates by duplicating whatever file structures it sees in the paths of the items that appear in these files. It will create any elements of that path that don't already exist. So, for example, on your local machine you could decide to create a new folder and put in it a bunch of other files and folders, to any depth you like. All of this structure will be transferred to the site, and will also be downloaded intact to any other machine running Fetch-O-Matic. When building your site, you may simply forget about structural transfer issues and put things together any way you want, creating files and folders at will all over your local site copy.
Because Fetch-O-Matic operates by consulting a central logfile (the changesFile) which contains only paths and filenames, no machine involved in site work needs to have a complete copy of the site. The only requirement for every machine doing work is that the DIRECTORY STRUCTURE encompassing every FILE being worked on is complete, and that copies of the changesFile and the messageFile exist in their proper locations (and are current) when work is begun. In other words, all the paths to files must be exactly as they are (or will be) be on the main site; every level in the path from the file in question to the root of the main site must be present on the copy. Items - files and folders - outside this direct line to the root may be ignored or dispensed with in the site copies as desired. If you are starting work from a new location that has no site copy, at the very least you need to create a local root directory, have a Settings file with correct information for that machine, and run FetchProfile to create the local changesFile and messageFile. If you then create files from scratch on this machine, you must also create locally their enclosing directory structure, if any. You may also run FetchChanges at this point, which, since there are no files in your local root, will put every recently uploaded file into your To Download List, which you may then edit as needed. Finally, if you wish to create a (more) complete local mirror, you may run FetchCompare, which will create a To Download list which represents the entire remote site.
Because Fetch-O-Matic overwrites files without warning, you must be sure that the relevant local and remote structures match. The author recommends testing Fetch-O-Matic by creating a root-level dummy "test" folder on your local machine with several levels and dummy files and directories, and verifying operations with that until you are comfortable with the results. Create test files, run FetchNewLocal, run FetchUp. Then remove the local files, remove the added lines from your local changesFile (as if you were a different machine with no knowledge of those files), run FetchChanges and then FetchDown. Everything you had removed should reappear. If this doesn't happen then there is an error in your configuration of Fetch-O-Matic's settings.
If you create empty directories on your local machine, including nested ones, they will not be uploaded. Fetch-O-Matic only deals with files. It actually doesn't upload or download directories at all; rather, when it begins to process a file, it first looks for parent directories and only then, if they don't already exist, creates as many of them as needed to contain the files inside them. Thus there must be at least one file in a directory (or one of its subdirectories) for Fetch-O-Matic to recognize a need for that directory and do something about it.
Once set up, Fetch-O-Matic is easy to use. Work on your site, run FetchNewLocal (and/or FetchMe, see below), peek at the "To Upload" file and alter it if you want, run FetchUp. You're done. If others (and/or yourself from another machine) are involved, just run FetchChanges (and FetchDown if required) before you start work. Run FetchUp without uploading files - or perhaps just uploading the messageFile - to post a notice informing others you're currently working.
All Fetch-O-Matic applications run in the background, and will allow you to perform other Internet activities like web browsing and email, and you can even open another simultaneous FTP connection with Fetch (but only to a different server - if you open a second Fetch window to your own server, Fetch will get confused because the windows will have the same name, so this is not advisable). However, some of the applications, in particular FetchCompare, are not so polite and don't background as well as others.
None of the Fetch-O-Matic applications have menus or allow user interaction, other than responses to dialogs. All of them (except FetchContinual and FetchHarvest)will quit running automatically when their actions are complete. Most of them conclude with an informative dialog which will wait indefinitely for your attention, then cause the application to quit when OK'd. Error messages will also appear in dialog boxes and will cause the application to quit when OK'd. Error messages are also written to the Activity Log.
Fetch-O-Matic was written in AppleScript. The beauty of AppleScript is that existing applications can be integrated, alleviating software creators from wasteful reinvention. The applications "Fetch" and "List Files" are used by Fetch-O-Matic in this way to do the tasks they were designed for, and which they do superbly.
Before you attempt to run any Fetch-O-Matic applications, you must configure the Settings file, see Settings, above. You should then run the application FetchProfile which will check your settings, create needed files, perform a test log-on, set List File's preferences (if you are using List Files), and finally, produce a report.
The first time you run each Fetch-O-Matic application, it may ask you to locate the applications it needs to communicate with: Fetch, your text application, and List Files (if used). When asked to locate an application, you will be presented with a standard file dialog box. Navigate to the folder where the application is installed and select it. If the name does not match exactly the name as specified in the Settings file, the Fetch-O-Matic application will still run, but you will be asked again the next time you run it to locate the application whose name did not match. If you are repeatedly asked to locate an application, it means that the names do not match exactly. Unlike most Mac file actions, this match is case sensitive, so exact capitalization is required.
After configuring your Settings file, run this application first. FetchNewLocal determines, by examining modification dates, what files have changed and need to be uploaded. As an option it employs the application "List Files" to compile a local site file list. "List Files" is significantly faster than the internal mechanism.
The first time you run FetchNewLocal it will show a dummy date. Tell it to "Set Time Only", which will cause it to quit (and create the file "FetchNewLocal Log" in the Auxiliary folder). Then run it again. The displayed time will now be the time when you clicked "Set Time", in YYYY/MM/DD HH:MM format. As a test, you should now change this to some time prior to when you last updated a file in your local site copy, and allow it to continue to run.
When FetchNewLocal concludes its run it should have identified your recently altered file(s) and listed them in the "To Upload" file, and opened this file. If this did not happen then there is an error in your configuration. Make sure the entries in your Settings file are correct. (At this point, you will also find a complete local site file list in the Auxiliary folder, which you might wish to examine at some point. This file is overwritten each time you run FetchNewLocal, so if you wish to preserve it, you must remove it from the Auxiliary folder; if not found there, it will be created anew each time).
If there is more than one local machine building your site, you will be downloading files created on those other machines. Every time you download files, you should run FetchNewLocal in Set Time mode right afterwards, so that these files will not be seen as new the next time you run FetchNewLocal.
Whenever you do any work on your local site copy, run FetchNewLocal to generate a list of anything that has changed since the last time it was run. This list of changed files is automatically placed at the top of your "To Upload" file, with a blank line separating it from anything already there. Nothing is removed from this file by FetchNewLocal; it only adds to what is already there; thus you can run it many times, building up a larger list of files to upload with each run.
FetchNewLocal resets its internal time stamp every time it runs, and shows you this time when you boot it up. You may change this time if you want to look further back (for example if you think you might have missed something), or less far back (if you have been running FetchMe instead, see below).
In order to not have FetchNewLocal skip files such as graphics which have been simply moved into the local site rather than modified, the drag-and-drop application FetchTimeStamp is provided to update modification dates so that such files will be included in the To Upload list.
The "To Upload" file is created in reverse alphabetical order by directory; files will be uploaded in the order they appear in it. Files listed after a blank line are ignored. On the site for which Fetch-O-Matic was originally developed, most graphics files happen to be stored in directories whose names start with "z" (as a way of keeping them grouped), and since one will want to upload graphics before HTML pages (so they won't be missing when called), this reverse order was useful, and has persisted. In any case, if you are building a "live" site you should always check the "To Upload" file to see that things are going to be uploaded in a logical order, and make alterations as required.
Both FetchNewLocal and FetchMe write to the "To Upload" file. You may run either application at any time, and you may also enter filenames in the "To Upload" file by typing or pasting. Doing so might be useful in situation where a set of files needs to be re-uploaded on a regular basis. In this case, just keep a list of these files permanently in your "To Upload" file, stored after a blank line. Then copy and paste the list to the top of the "To Upload" file whenever needed.
The "autoRemove" option in the Settings file selects whether or not you want to have successfully uploaded files removed automatically from the "To Upload" file by FetchUp. This is a timesaving convenience and is enabled by default, but if desired you may disable it. The changesFile will always have a list of the files which you attempted to upload in the event you want to know what they were, and both the Activity Log and the changesFile will show you what actually got uploaded.
You may specify files to exclude from the "To Upload" file. Files can be excluded by directory, extension, or filename.
FetchUp uses the "To Upload" file to determine what to upload. Before running FetchUp, you may manually reorganize the "To Upload" file as you see fit in terms of upload order and postponement. FetchUp will ignore anything after a blank line, so to postpone, just insert a blank line before the files you want to skip. A blank line as the first line will result in nothing uploaded at all. If you have run FetchNewLocal multiple times without uploading, there will be blank lines in it (representing each time FetchNewLocal was run), which you will want to remove.
FetchUp has two main functions: it posts notices about activity and it uploads files. The notices it posts are contained in the changesFile, which is the core of the multi-user aspect of the system. Assuming there's not a conflict (see about the changesFile below), the first thing FetchUp will do is to post a notice that "you" (as defined in your Settings file) are doing something. If you are uploading files, it will post a list of files being attempted. If you are not uploading files, it will simply announce that you are "at work"; this is to avoid the conflict of others working on the same files at the same time. You can make use of the messageFile to announce what (areas or) files you are tackling and tell others to keep out, or just tell them to contact you. Anyone blindly attempting an upload while your "at work" notice is up will be stopped, as described below. This does not outlaw simultaneous uploads, but forces everyone to examine the situation for possible conflicts.
FetchUp always posts the date, an operator's name, the uploads being attempted, and the successful uploads. The optional posted parameters are: "at work" or "off" (in the case of nothing currently being uploaded), and "off when finished" or "still working when finished" (posted during an upload in progress). The options are set by default as follows: if nothing is being uploaded, an "at work" message is posted. If something is being uploaded, an "off when finished" is posted during the upload, and "off" is posted when complete.
All possible options can be accessed by holding down the command key during launch. You must hold down the command key until you hear a beep; unfortunately, AppleScript cannot recognize a key press until well into execution, so it may be several seconds before you hear the acknowledging beep. You will then see a dialog box. You may change your name (a temporary change valid for the current run only); post an "off" notice with nothing being uploaded (canceling a prior "at work" notice); post "still working when finished" during an upload and "at work" on completion; or simply cancel execution. The slow response of the command key is actually a benefit here - you have a second to press the key if you launch FetchUp by mistake and decide you want to cancel.
A second dialog will appear which asks if you want to override your setting for PPP sign-off (i.e. hang-up) behavior on completion of the upload. This dialog can also be accessed by command-launch of FetchChanges and FetchDown.
The changesFile on the remote machine is always assumed to be the latest version of that file. So, before FetchUp does anything at all, it downloads the remote changesFile and compares it to your local copy. If the first line of the remote version is not the same as yours, FetchUp assumes that remote activity has occurred and stops what it was about to do, and instead operates as if it were FetchChanges (see below) - it moves your local changesFile to the trash with suffix .old, and adds any new files it finds for downloading to your "To Download" file.
(Note: if you are the only person maintaining your website, and work exclusively from a single location, you may choose to ignore the remote changesFile, by setting "ignoreChanges" in the Settings File to "Y". The local changesFile will still contain a record of activity. The only advantage to doing this is a speed improvement due to not having to upload and download this file. If you subsequently wish to re-enable the multi-user/multi-location features of Fetch-O-Matic, you will need to delete everything from your local changesFile first, or upload that file manually; otherwise an error will occur. If the remote file is ignored, all of the statements referring to it will of course no longer apply).
If your local changesFile is an empty file, as will be the case when first starting or when operating from a new location, the entire contents of the remote changesFile will be added to your "To Download" file. However, if your local file is not empty AND its first line is NOT found inside the remote file, as it should be, then FetchUp assumes that some kind of serious error has occurred and quits. This situation is one which you will have to resolve manually before proceeding further. If your local changesFile is corrupted, use the downloaded remote version as your new local copy and simply delete the lines in it which contain activity you are missing. Then run FetchChanges to get back in sync. If you are operating from a new location but don't want the entire remote changesFile added to your downloadList, then download the remote file manually, and delete lines as above.
Otherwise, if all is well, your current activity - a "working" notice and/or upload attempt - is written to the changesFile and it is uploaded. If there are files to upload they are then uploaded. On completion of uploads, the changesFile is rewritten and uploaded again, and FetchUp quits with a dialog box announcing that it was successful. It also alerts you to any other activity that may have occurred during your upload - see below under "Simultaneous Uploads." (If you have disabled the autoRemove feature, you'll have to manually clean out the uploaded files from your "To Upload" file).
It is possible, and sometimes necessary, to manually write to the changesFile, then manually upload it. This topic is discussed under the "Activity Log" file heading below. The main reason to do this is to erase some action which occurred by mistake, such as uploading files unintentionally. So as not to propagate this error further, you will want to remove all mention of the erroneous files from the changesFile. The only way to do this is by hand, since if you alter your local changesFile and then try to run FetchUp, it will see your file as the older one (not matching the site copy) and move it to the trash. The correct procedure is to simply delete the erroneous upload from the local changesFile and then upload that file manually. If your entire upload session was erroneous, you can simply delete the whole event from the changesFile (restoring it to its status before you did anything). Finally, attend manually to whatever you have to do to put the site back in order.
It will also be necessary to do a manual correction and upload after a broken upload if the automatic recovery procedure of FetchUp (see below) should fail for some reason, such as a corrupt local changesFile.
Truncating the changesFile: the changesFile will grow indefinitely. At some point you will want to reduce its size. You may do this at any time, simply by truncating (from bottom upwards, of course) your local copy, then running FetchUp (since the first lines still match, your new changesFile will be seen as up-to-date and will replace the remote one). You may want to preserve the old information for your records, saving the excised section into a new file. The only thing to watch out for when truncating is the possibility that you will be removing a portion which someone else may not have yet seen, so be sure at least to post a notice in the messageFile that you have truncated the changesFile as of a certain date. Anyone needing a list of older uploaded files can then get it from you.
The settings file entry "alertSize" will enable reminders when the file grows beyond a particular size.
In the event of a crash, broken connection, or other disturbance of the upload, FetchUp has an automatic recovery procedure. You should just run it again, without altering any of the Fetch-O-Matic text files. It will recognize that an upload was interrupted, and it won't upload anything on this run. First it will check if other activity has taken place on the site. (If someone else has done something in the meantime, that information is handled exactly as under normal circumstances, so you don't need to worry that any other information will be erased or skipped by the auto-correction procedure). Then it will write the successful portion of your upload to the changesFile and re-upload that file, merely putting the changesFile back into sync with what has actually occurred. Then it will quit, allowing you to decide when to resume the interrupted upload. The changesFile will now have the notation "Auto-Correction of this file" above the added files.
This feature is independent of the "reconnect" feature and setting, which attempts to continue an interrupted upload or download session, and which, if successful, will not require FetchUp auto-correction. The reconnect process is transparent and will generate no error message or cause itself to be noticed if successful.
To use the messageFile, just write whatever message you want into it. Although it must exist, its use is optional. In order to upload it, its name must be in your "To Upload" file. This will happen automatically if you run FetchNewLocal or drop the messageFile on FetchMe (see below), or, once you are comfortable operating Fetch-O-Matic, you may simply type its name (and path if any) into the "To Upload" file.
The messageFile is the only file besides the changesFile which is automatically downloaded (rather than being added to the "To Download" file) by FetchUp, FetchChanges, and FetchDown. If you have written something to this file and then run FetchChanges or FetchUp and the remote messageFile has been changed, your local messageFile will be moved to the trash - simply because it is not the same as the remote version, even though it may in fact be newer. (This is done as a safeguard that no messages posted on the remote machine are ever skipped or mistakenly overwritten). If this happens, just open the trash and drag your messageFile (now with suffix ".old") back out, recover your text, and paste it into the new messageFile.
Strategies for use of the messageFile depend on how many people are involved. One method is to always put new messages at the top of the file so as to maintain a message history, and then periodically truncate the file by mutual consent, as with the changesFile. On the other hand, if just two people are involved, messages can be erased when read by the second person. Another idea would be to have people post their initials in the file acknowledging that they've seen a particular message; once everyone has seen it, it can be deleted. If only one person is doing site maintenance, the file may still be useful as a central "to do" list for pending work.
This file is a record of local activity. It is written to after each individual file transfer in either direction. Error messages are also logged here. Its primary purpose is verification of transfers and disaster recovery - if you get dumped off or have a crash, the log will be the guide as to where and how to resume. FetchUp reads the "Activity Log" file when it sees that a transfer had been interrupted, and FetchUp should be able to automatically recover, but not every circumstance can be foreseen and it is conceivable that it will be necessary to manually copy the "Activity Log" file's list of successfully uploaded files to the changesFile and manually upload that file.
If you must copy files from the "Activity Log" file to the changesFile, you need precision only in the file names, which you should copy and paste exactly as they appear in the "Activity Log" file. You don't need to worry about other details, like the date, time, attempted files, your name, and so on. Such material is posted with an informational purpose alone and is NOT used by the automatic process to create the "To Download" file list, or as a basis for determining what has changed on the site. The fact that the changesFile is DIFFERENT from local copies is sufficient to trigger the automatic list assembly. When reading the changesFile, the file-gathering process looks only for lines without commas to build a file list. All of the informational lines, including attempted uploads, have commas. All such lines (and blank lines) are ignored, except for that one line which exactly matches the first line of the changesFile on the local machine, which represents the demarcation line for new material. (If there is no such line, unless the local changesFile is blank, Fetch-O-Matic will refuse to execute further since things are out of whack and some manual intervention is required).
The "Activity Log" file, like the changesFile, will grow indefinitely. As with the changesFile, you may truncate it at any time. The "alertSize" setting determines reminders as with the changesFile.
Another person working on the site, either by mutual agreement or by simply looking at the files currently being uploaded as listed in the current changesFile, may decide to simultaneously upload files. This is not a problem. If he runs FetchChanges first, putting his changesFile in sync, then his FetchUp won't complain about your upload currently in progress. However, you won't be aware of his new activity, so to handle this situation, FetchUp looks again at the changesFile whenever it concludes an upload, and if it sees that someone else has logged on in the meantime, it runs as a modified version of FetchChanges (see below). If the other upload has concluded, FetchUp will build a "To Download" file list just like FetchChanges; otherwise, it will show an alert that someone else is now at work, and your changesFile will list the files in progress from his end just as if you had not been on at all. Since the changesFiles is in reverse chronological order, the other person's upload will appear below the successful list from your end, which is more recent because it concluded after he began.
It's possible to add files to an upload in progress by running FetchNewLocal or dropping files on FetchMe while FetchUp is running. If you want FetchUp to run continuously (checking for new files at the end of each batch and uploading whatever it finds) then the autoRemove option must be enabled - and of course FetchNewLocal or FetchMe must finish their work before FetchUp finishes uploading. Otherwise the new files will still be added to the To Upload file, but FetchUp will quit after the current batch.
FetchNewLocal changes its normal behavior if FetchUp is running when it writes its result: it won't add the usual blank line at the end of the new files. This was done to enable continuous operation without having to edit the To Upload file.
The way the process works is that when FetchUp is finished with an upload batch it re-reads the To Upload file and, if that has changed in the meantime, it figures out what was added and it starts in on a new round of uploads, just as if it was started anew. This can happen as often as you like, so that, for example, if you have a huge update you are doing and want to get the stuff going up while you are working, just keep dragging to FetchMe or running FetchNewLocal while you work. (The safeguards and notifications about others simultaneously working on the site are always in place and will function as expected).
IMPORTANT: Do not manually remove anything from, or add to the bottom of, the list currently in progress: FetchUp needs to re-read this list when it finishes, or in the event of an upload error. So, if you edit the To Upload file during an upload, be sure that such edits occur only within the newly added files which appear before the current batch in progress, or else after the first blank line which denotes the end of the current batch. If you mistakenly add a blank line before the current batch, FetchUp will display an error message and quit rather than uploading any additional files. (For recovery, see note below).
Nevertheless, do not be afraid to re-order the new additions or to postpone newly added files by moving them after a blank line below the current batch; anything after a blank line in the To Upload file is always ignored. Just exercise caution not to disturb the current batch list. The end of the current batch is the first blank line in the file. If you are not sure where the beginning of the current batch is, open the local changesFile. The first "Attempting:" paragraph is the entire current batch.
You must finish editing and save the To Upload file before FetchUp quits, and should use an editor such as BBEdit which won't claim as busy an open file, because if FetchUp happens to quit while this file is still open in your editor, FetchUp will attempt to read /write the file "behind your back", and this will cause an error if you use an editor like MS Word which does claim the file. If FetchUp should happen to quit while you are editing and re-saving the file, if it "beats you to the punch", then you also have a little problem: the file you are looking at will no longer be the current one. If you then save it, the uploads just completed are still in it, and they will reoccur unless you remove them manually. Consult the changesFile or the Activity Log to see what to remove.
If you run FetchMe or FetchNewLocal without the autoRemove option enabled, no damage will occur; FetchUp will simply quit without re-reading the To Upload file.
Note: If you edit the To Upload file during a FetchUp run and get an error message about improper blank lines or edits, the successfully uploaded files up to that point will not have been removed from the To Upload file. You should then read the changesFile to verify that the intended files were indeed uploaded, and remove them from the To Upload file by hand. The reason for this (the fact that this situation produces a fatal error) has to do with the upload-failure auto-correction process, which depends on a re-reading of the To Upload file for certainty as to what went wrong. If the conditions before the failure cannot be re-established, there are too many uncertainties for the process to be reliable.
FetchChanges snoops the remote changesFile for activity and adds items to your "To Download" file if there have been any uploads from another location. To determine activity, FetchChanges downloads the remote changesFile and compares its first line to that of your local version (see changesFile, above). If a difference is found, it knows that remote activity from another location has occurred, and a further comparison is made of the files' contents. If it finds that uploads have been made from elsewhere, it adds all such files to your "To Download" file, in the same way that FetchNewLocal adds to your "To Upload" file. It doesn't erase anything from this file, and adds at the top of the file. You can run FetchChanges multiple times without downloading anything and the list will accumulate in the "To Download" file, with a blank line inserted after each FetchChanges run.
After downloading the changesFile and comparing it to your local copy, if there is a difference your old changesFile is moved to the trash with extension ".old" after the name, and the remote file replaces it. If there is no difference, the downloaded file is deleted instead (without visiting the trash). If FetchChanges detects that the messageFile has been changed it downloads the new messageFile (and puts the old one in the Trash with extension ".old"). After opening any new files, FetchChanges quits with an informational message.
If the command key is pressed at (or shortly after) launch, a dialog will appear which asks if you want to override your setting for hang-up behavior on completion of the run.
FetchDown uses the "To Download" file to do downloads in just the same manner as FetchUp uses the "To Upload" file for uploads. It ignores files after a blank line, and it writes to the "Activity Log" file after each successful file download. The main difference from FetchUp's procedure is that FetchDown does not touch the changesFile, since it is only updating your local machine, which will in fact just be catching up to the remote site (and to your current copy of the changesFile).
Interrupted downloads, due to a crash or broken connection, are handled slightly differently than in the case of uploads. The reason for this is that broken downloads only affect your local machine and no one else. During the download process, each file is logged, one by one, in the "Activity Log" file after successful arrival. Additionally (if you have not disabled auto-removal), each file is also removed one by one from the "To Download" file. Therefore, if your download is interrupted, all you have to do is run FetchDown again and it will simply pick up from where it left off. (However, if you have disabled auto-removal, you'll have to compare the "Activity Log" file with the "To Download" file and remove the successful transfers manually from the "To Download" file).
If no files are queued for downloading in the "To Download" file, FetchDown quits without logging on and informs you that it was given nothing to do.
After logging on, if FetchDown sees that remote activity has taken place, the download is aborted, and it switches modes to operate as FetchChanges: any new files are added to your downloadList, and you are alerted. You may then readjust your downloadList and rerun FetchDown. FetchDown also checks the site again at the conclusion of the downloads and will again write anything new to the downloadList and alert you if someone else was at work during your download.
If the command key is pressed at (or shortly after) launch, a dialog will appear which asks if you want to override your setting for hang-up behavior on completion of the download.
These two additional applications work by drag-and-drop only (they won't do anything if double-clicked). Both identify and operate on every file in any possible combination of dropped items (files and/or folders - including all nested items down to the last file in the last folder). This means that you may select, by shift-clicking, any desired assortment of files and/or folders, and drag and drop them all at once onto either of these applications; any included files will be handled as if dropped one at a time, file by file.
FetchTimeStamp will set the modification date of all received files to the current time, useful for acquired or moved material which would otherwise be skipped by FetchNewLocal.
FetchMe will add all received files to the "To Upload" file directly, without having to run FetchNewLocal, and without consulting modification dates. It does not add any blank lines to the "To Upload" file, so it may be run multiple times in succession to put together a single unbroken list. Its purpose is for performing immediate uploads or doing small jobs for which running FetchNewLocal is "overkill"- for example when only a few files are worked on, or when all updated files are in a single directory, or when augmenting an upload in progress with an additional file or two. However, be warned: the author has occasionally made mistakes using FetchMe - forgetting, for example, to upload a graphic associated with a new HTML page. Running FetchNewLocal, although more time-consuming, is always a safer procedure.
The Fetch-O-Matic Utilities reside in a subfolder in the Fetch-O-Matic folder. They may not be moved from this folder, nor may the associated text files be renamed or moved. There are seven utilities: FetchCompare, FetchContinual, FetchDelete, FetchHarvest, FetchMeNot, FetchProfile, and FetchStructure. There are also six permanent text files: "To Delete Local", "To Delete Remote", "Unique In Local", "Unique In Remote", "Never Upload", and "Never Download". The contents of some of these text files changes when you run their associated applications, so they should be copied rather than moved if you want to preserve them in a particular state. Then there are the "F-O-M Profile" text files appear when you run FetchProfile, and the "Site Structure <date>" text files which are produced by FetchStructure. These files are not re-used and may be trashed. Finally, FetchCompare will access - or create if missing - the text file "remote home filelist", which is a complete list of every file on the remote site; this is also a permanent file, but it lives in the Auxiliary folder (since that's the home of the local file list).
FetchCompare compares your local site mirror with the remote site. It first generates a list ("remote home filelist" in the Auxiliary folder) of the contents of the remote site by directory, similar (see note*) to the local list generated by FetchNewLocal/List Files, and then writes the files "Unique In Local" and "Unique In Remote", which contain the differences (these files are all re-written every time you run FetchCompare). If you then find any files in either of the Unique lists which ought to be in both locations, simply copy and paste them into either the To Upload or To Download files as appropriate (i.e., unique local files go into the To Upload file and unique remote files go into the To Download file). There is another thing you might want to do about unique files: remove them. To do this, use the application FetchDelete.
In order to do the comparison, you must of course have a local site filelist. If you have a current one you can save time by using it instead of making a new one. When you start FetchCompare, a dialog box presents the options.
File modification dates are not used in the comparison, because they are misleading. For example, if you download something from the site it will have a modification date later than the copy on the site, because the Mac assigns a new date upon arrival. Conversely, if you upload something the date shown by the server is also the arrival date and will thus be later than your local modification date. So, there is no way that FetchCompare can tell you whether your copy of a particular file is identical to the remote version or not. Nevertheless, all dates are collected and appear in the Remote Site Filelist, and you can sort this list (and the local list also) with a text editor or word processor to see everything listed in date order - that's the reason Fetch-O-Matic uses the YYYY/MM/DD date format.
FetchCompare is quite slow, but behaves itself in the background reasonably well, although not as well as other Fetch-O-Matic applications. (Speed is improved with OS 8.5 or better, but processor hogging is worse). Its slowness is a result of a) having to open every directory on the remote site; b) the need to interpret and rewrite the information it obtains; and c) the comparison it makes between the local and remote sites. On a site with thousands of files, a complete run could take several hours. To minimize online time, constructing the remote filelist is done in two steps; first the information is collected, then it is reformatted off-line. (If your online session is interrupted, the partial filelist will be in a different format than a complete list. This is not cause for concern; the format is not changed until the list is complete). On a large site, it's advised to run FetchCompare at bedtime - although it may be briefly entertaining to watch it flail away wading through the remote site. If your site is very large, you may need to increase FetchCompare's memory partition; see Memory Allocation.
Because FetchCompare is slow, some options are provided on launch. One option is a choice to only get the remote list and postpone the comparison, or to only do the comparison if you already have the list. The other option is the capability to resume gathering a remote list from somewhere in the middle, if the list creation was interrupted. If this happens, examine the Remote Site Filelist. You will notice that it is in reverse order, bottom to top, with a line with a date followed by a line with a filename. Find out at which item (file or directory) of the root directory it was cut short, then determine the number (up from the bottom of the list) of the incomplete item as it would appear in a Fetch window showing your site root, and enter that number. Your list will be resumed from that point. The mechanism is intelligent and will find the exact point at which the break occurred, so it won't make double entries; it will also quit and tell you if you entered a starting file which doesn't show up on the partial list, with the subsequent danger that you will have missed files. If you attempt to run FetchCompare in compare-only mode and your remote list is incomplete or nonexistent, you will also get an informative error message.
If you are running a dual-server setup, FetchCompare will behave appropriately, looking on the second server for the relevant files. It looks first at the primary server, then the secondary one. If you wish to resume a partial remote list which was interrupted while on the second server, enter the start number as if the root directory on the secondary server were a continuation of the primary root directory (such that the entered number will be larger than the number of files in the primary root).
FetchCompare does not recognize the existence of empty folders, and neither does FetchMeNot. However, FetchStructure will locate any empty folders on both local and remote sites.
*Note: There is one difference between Mac and UNIX file systems which you may encounter. On UNIX systems, filenames and directory names which contain numbers are alphabetized with the number interpreted as a number: MyFile10 comes after MyFile9. But on the Mac, unfortunately, the opposite occurs: MyFile10 comes before MyFile9, because the digits of the numbers are treated individually as characters and not as part of a number, and 1 comes before 9. What this means is that if you open the remote list and the local list and compare them side by side, then if you have filenames or directory names with multi-digit numbers in them, things will not line up as you expect. Because of this quirk, FetchCompare must do its work by exhaustive, not merely alphabetic, comparison, which is one reason it runs slowly).
FetchContinual further automates identifying and uploading new files by periodically scanning the local site for new or updated material and then uploading what it has found. In effect, it is FetchNewLocal and FetchUp rolled into one continuously running application.
Rather than duplicating the instructions for use of those applications, this section will point out only the additional or altered functions. Please do not attempt to use FetchContinual until you are comfortable with manual operation of FetchNewLocal and FetchUp.
Unlike every other Fetch-O-Matic application except FetchHarvest, FetchContinual doesn't quit until you tell it to. Other than that, some changes to dialogs, and the fact that it will not add blank lines to the To Upload file and will ignore the "autoRemove" setting (for obvious reasons - it must remove previously uploaded files to avoid redundant uploads), virtually all of the instructions regarding FetchNewLocal and FetchUp are valid. For example, you may choose to run FetchContinual on only a portion of your site, in the manner described in the section "Secret" Files/Areas. FetchContinual may optionally also be run from a scheduling application (see Unattended Operation).
When launched, FetchContinual will ask you for a scan interval, in minutes, with a default as for the previous run. There is no maximum interval - for an interval of a week, you'd enter 10080 (i.e., 60*24*7). The timer starts at the conclusion of the current cycle - for example, if you set the interval to 10 minutes and then have an upload which takes an hour, the 10 minute count begins at the end of the hour.
As with FetchNewLocal, on launch FetchContinual will ask for a cutoff date, with the default as the last time it checked for new files, and won't upload anything older than that time. As each scan begins, the cutoff time is reset for the subsequent run as of that moment, so no files will fall through the cracks if they are added during a scan (although it is possible that a particular file could get seen twice and uploaded twice since file modification dates are truncated to the minute; it was thought better to err on the side of caution).
After the opening dialogs, FetchContinual will perform its first scan right away, and if new files are found, the upload will begin immediately. If nothing is found, no action is taken - Fetch is not activated and the remote site is not accessed. This cycle will then continue indefinitely at the specified interval until the application is quit, or remote activity or an error is encountered. Errors and all uploads are written to the Activity Log.
FetchContinual reads and writes to the remote changesFile just like FetchUp. This means that it is possible to monitor FetchContinual's actions from another location. Furthermore, if you want to make FetchContinual stop running from another location, all you have to do is alter the remote changesFile. And if FetchContinual detects remote activity, it will put new remote files in the To Download file, just like FetchUp does.
FetchContinual can be used for more than updating websites. It can be used to monitor any local folder and deliver content arriving there to another location via FTP. Possible uses would be to watch a mailbox folder, a digital camera folder, or a folder receiving automatically generated data of any kind. Whenever a file within its view is new or newly modified, it will be delivered.
FetchContinual is pretty well-behaved in the background and will allow you to do almost anything else simultaneously on the machine - except run List Files or use Fetch to access the same site as you are automating. Be aware that if you are doing other work via PPP, FetchContinual does observe the closeLink preference in the Settings file and could disconnect you if you have it set to log off after an upload. If you hold down the command key when you launch FetchContinual, you will be presented with the option to override this setting for the current run.
You can run FetchNewLocal and FetchContinual interchangeably, with consistent behavior; the only difference is in the degree of automation. Each of these two applications, when launched, will check the last time run of the other one, so that if you are using both of them, redundant uploads won't occur. The FetchNewLocal Log file in the Auxiliary folder is where this time is stored.
There are a series of settings in the Settings File dealing with running FetchContinual in unattended mode. This mode bypasses the dialogs which normally accompany the launching of FetchContinual, and substitutes for them a group of settings which specify the desired behavior. Instructions for each setting are given in the settings file. The purpose of this mode is to be able to run FetchContinual by using a scheduling application such as cron or MacAT. Using this mode you can also place an alias of FetchContinual in the Startup Items folder of a machine subject to unattended reboots. Unattended launch would not succeed without this mode because the startup dialogs would wait indefinitely for a response. In unattended operation mode, FetchContinual will quit after a single scan if the scan interval is set to zero (the correct setting when using a scheduling application), or it can be set to run indefinitely at a desired interval (if launching from a Startup Items alias and no scheduling application is used).
FetchDelete removes files (and/or directories) from either, or both, the local and remote site copies. When deleting files locally, they are moved to the trash, and if you make a mistake you can open the trash and use the "Put Away" menu command in the Finder to restore them. However, deletion via FTP is not undoable, so you must be extremely cautious with this option. FetchDelete provides ample opportunity to confirm and cancel before deleting anything, so don't be afraid to run it on that account.
The operation of FetchDelete is similar to FetchUp and FetchDown. It reads the To Delete files, ignoring anything after a blank line, and will remove deleted items from them if autoRemove is turned on, and it will log everything it did in the Activity Log.
Paste any items you want to expunge into these files from the Unique lists, or, alternatively, you may add to these lists (both of them at once) by dragging and dropping local files and directories on to FetchMeNot, see below. You may also manually add to these lists. As with the To Upload and To Download files, pathnames in these files are based on the localHome directory as the root, and must use colon delimiters.
When you run FetchDelete, you are first presented with a dialog box to select which site(s) (local and/or remote) you want to remove files from. As a precaution, before doing anything, the list(s) in question will be opened by your text application for confirmation. You may then edit them, but if so you must save the changes before continuing. You will then need to select FetchDelete from the Application Menu to proceed. Finally, two more dialog boxes confirm your intentions and allow escape before execution continues.
FetchDelete can remove entire directories. Removing directories removes all files and subdirectories in them as well. Be very, very careful here - you could delete a whole lot of website with a single line. To delete a directory, it must have a trailing delimiter. Just find a path that has the name of that directory in it, and eliminate everything after that directory except the colon. (You may then also erase from the deletion list any files that were in that directory, but this is not necessary. If files and folders are found not to exist, such as after deletion of their parent directory, they will be skipped without complaint).
FetchDelete does not write automatically to the changesFile. This is because the deletions will not involve any further FTP activity. If you have removed items from the remote site, if there are other site mirrors then you should alert their owners of the deletions by using the messageFile. You should paste a list of deletions into this file from your Activity Log, and then the others using Fetch-O-Matic can copy this list to their To Delete Local file and use FetchDelete to remove these items from their local machines.
Unlike every other Fetch-O-Matic application except FetchContinual, FetchHarvest runs continuously, monitoring a remote site and downloading anything new or modified which appears there. It bases its actions on file modification dates of items on the remote site. Its actions don't affect the remote site so it doesn't read or write to the changesFile. Its intended use is in situations where unlogged uploads are to be collected on a local machine, for example in news gathering or other report collection situations, or when several people are maintaining a site and some are running Windows or UNIX and can't use Fetch-O-Matic. It has two modes of operation: it can check at a specified time interval, or it can run daily at specified times of day. It may optionally also be run from a scheduling application (see Unattended Operation).
As with FetchDown, FetchHarvest will mirror any remote structure on the local machine. Thus it is possible for operatives in the field to upload an entire folder hierarchy which will be downloaded intact. Because it may be necessary in some circumstances to prevent over-writing of local files by downloads with the same filenames, there is an additional feature. If the setting "timeStampFolders" is set to "Y", each time FetchHarvest checks the remote site, if there is anything new, a new local folder will be created (in the localHome folder) with a name derived from the date and time of day, and all downloads from the current session will be put in that folder, effectively adding a level to the folder hierarchy. This is the mode to use for news-gathering or web-cam monitoring, etc. when the intention is to preserve every new remote item, whatever its name or location.
Each time FetchHarvest cycles it rewrites the remote home filelist (as does FetchCompare), and if there are new files it writes them to the To Download file. It then proceeds to download those files. It will ignore the "autoRemove" setting (for obvious reasons - it must remove previously downloaded files to avoid redundant downloads),
Because the process of scanning a remote site is not speedy, it is best to set up FetchHarvest to examine the minimum remote directory structure required for it to perform correctly. Users may wish to run FetchHarvest as a separate suite with its own settings file, addressing a sub-directory of a site, to limit its scope. See "Multiple Fetch-O-Matic Suites".
When launched, as with FetchNewLocal and FetchContinual, FetchHarvest will ask for a cutoff date, with the default as the last time it checked for new files, and won't download anything older than that time. FetchHarvest will then ask you for a scan interval, in minutes, with a default as for the previous run. There is no maximum interval - for an interval of a week, you'd enter 10080 (i.e., 60*24*7). The timer starts at the conclusion of the current cycle - for example, if you set the interval to 10 minutes and then have a download which takes an hour, the 10 minute count begins at the end of the hour. Entering a time interval of zero will perform a single cycle and then the application will quit. If you wish FetchHarvest to operate in this mode, enter the interval and click "Use this interval".
If you have entered times of day in the "harvestTimes" setting in the settings file, you may click the button "Use Fixed Times" instead. FetchHarvest will then ignore the interval setting and run daily at the specified times. It will always perform an immediate check no matter what time of day it is currently. This is done so that the operator can verify that the application is working properly. The next check will occur at the next specified time (potentially not until the following day, depending on the settings).
As each scan begins, the cutoff time is reset for the subsequent run as of that moment, so no files will fall through the cracks if they are added during a scan (although it is possible that a particular file could get seen twice and downloaded twice since file modification dates are truncated to the minute; it was thought better to err on the side of caution).
After the opening dialogs, FetchHarvest will perform its first scan right away, and if new files are found, the download will begin immediately. If nothing is found, no action is taken. This cycle will then continue indefinitely at the specified interval or times of day, until the application is quit or a fatal error is encountered. Errors and all downloads are written to the Activity Log. Connection errors are as a rule not fatal; they will be written to the Activity Log and operation will attempt to continue.
The noDownloadFolders setting works exactly like the noUploadFolders setting, but in the opposite direction. Anything withing the scope of this list of remote directories will be ignored by FetchHarvest. All the other exclusions will be active as well.
FetchHarvest is pretty well-behaved in the background and will allow you to do almost anything else simultaneously on the machine - except to use Fetch to access the same site as you are automating. Be aware that if you are doing other work via PPP, FetchHarvest does observe the closeLink preference in the Settings file and could disconnect you if you have it set to log off after an FTP session. If you hold down the command key when you launch FetchHarvest, you will be presented with the option to override this setting for the current run.
FetchHarvest has three phases in a cycle: 1, log on and get information; 2, examine the information; and 3, download new files, if any. The onWhileDigesting setting determines whether or not FetchHarvest will log off during phase 2, which could last quite a while if it is looking at many, many files. If that is the case it may be desired to minimize the online time by setting onWhileDigesting to "N". If so, and if there are files to download, FetchHarvest will then reconnect to perform phase 3; if there's nothing to download, it will stay disconnected.
FetchHarvest provides the option to delete remote files after download. This option is enabled by the setting "harvestCleanup" in the settings file. When set to "Y", files will be deleted from the remote host after they have been transferred to the local machine. The operation is fail-safe; no file is deleted until after a download is complete. The deletions occur on a one-by-one basis after each file is transferred, so that in the event a transfer session is interrupted, the remote directory will contain only those files not yet transferred. Use this option with caution: remote files cannote be recovered after deletion.
There are a series of settings in the Settings File dealing with running FetchHarvest in unattended mode. This mode bypasses the dialogs which normally accompany the launching of FetchHarvest, and substitutes for them a group of settings which specify the desired behavior. Instructions for each setting are given in the settings file. The purpose of this mode is to be able to run FetchHarvest by using a scheduling application such as cron or MacAT. Using this mode you can also place an alias of FetchHarvest in the Startup Items folder of a machine subject to unattended reboots. Unattended launch would not succeed without this mode because the startup dialogs would wait indefinitely for a response. In unattended operation mode, FetchHarvest will quit after a single scan if the scan interval is set to zero (the correct setting when using a scheduling application), or it can be set to run indefinitely (if launching from a Startup Items alias and no scheduling application is used).
FetchMeNot functions very much like FetchMe; it's a drop-only application which adds all dropped items to the top of both "To Delete" lists. These items can then be deleted from the local and remote sites by FetchDelete. Dropping a folder of items produces a list of files in it, not just the name of the folder, even though it may be desired to delete the entire folder. But it was thought safer to exhibit the entire contents so that you will be reminded of exactly what will be deleted. As noted in the discussion of FetchDelete, you may then truncate the list to show only the parent folder, but this is not necessary.
Unlike FetchMe (which ignores empty folders), FetchMeNot does allow you to drop empty folders and have them added to the "To Delete" lists.
This is a utility to ascertain the directory structure on your local and remote machines. It writes what it finds to a file named "Site Structure <date>" which it will create in the Fetch-O-Matic Utilities folder. You have the option to look at either or both the remote and local sites. FetchStructure examines every directory and identifies any empty ones, which it lists separately at the top of its output file, followed by all the others.
Remote investigations can be resumed as with FetchCompare. You can look at the output file to see where to resume. However, this resumption will overwrite an existing "Site Structure" file rather than attempting to add to it, and the new file will contain only the newly examined portion. If you want to preserve the partial previous information, move or rename the "Site Structure" file before resuming.
If you are running a dual-server setup, FetchStructure will behave appropriately, looking on the second server for the relevant directories. It looks first at the primary server, then the secondary one. If you wish to resume a partial remote list which was interrupted while on the second server, enter the start number as if the root directory on the secondary server were a continuation of the primary root directory (such that the entered number will be larger than the number of files in the primary root).
Since Fetch-O-Matic is only aware of files which get listed in its text files, it is possible to have a website with files and/or areas which are independent of Fetch-O-Matic and ignored by it. This may be desired if a particular person or group is responsible for a website portion which stands pretty well on its own, or at least has links to and from another area which don't need to be updated frequently or at all (because they only point to index pages, for example). Not only is this possible and practical, but in fact you can run an entirely separate Fetch-O-Matic suite for this other portion - or several such. All that is required is that the changesFile and messageFile have separate names and/or locations for each "secret" Fetch-O-Matic suite, and that all files in question are not updated by unauthorized persons "from the other side" and thus don't ever appear in their changesFile. If you set ignoreChanges to "Y", then there will be no evidence on the site of the additional Fetch-O-Matic suite(s) since it will not be necessary to have a remote changesFile or messageFile at all.
To stop execution in progress of any Fetch-O-Matic application, you may use command-period; this keystroke won't always work instantly and on rare occasions may produce strange error messages (which can safely be ignored). How quickly command-period functions depends on which application is running and what it's doing. A more brutal and instantaneous approach is to type command-option-escape, the universal Mac application "kill" command. In order for either keystroke to work, the Fetch-O-Matic application will first need to be brought to the front. Use the Application menu to do this, then type the keystroke. Killing a running application with command-option-escape may cause instability in your system. You should save any open files and reboot; even though everything may seem healthy enough, you are courting a system crash if you continue, particularly if you try to run the killed application again.
The file transfer situation is a little more complex because killing a Fetch-O-Matic application will not tell Fetch to interrupt a particular file's download or upload in progress; it will only prevent the next pending activity from occurring. If you wish to interrupt an ongoing upload or download, you have to also bring Fetch to the front and tell it to stop. If you don't kill the Fetch-O-Matic application first, if you have the "reconnect" setting enabled, Fetch-O-Matic will re-attempt the transfer, since it interprets your manual intervention as a connection error. (As for List Files, to stop it you have to bring it to the front also, then cancel).
Stopped execution, whether by design or accident, can produce unwanted side effects. See the "Couldn't Find..." error topic.
Fetch-O-Matic puts up various messages, and puts its error messages both on screen and into the Activity Log file. Should you crash before you have had a chance to read an error message, you can still find out what it was by reading the Activity Log.
Unless the Fetch-O-Matic application is front-most when it reports an error or finishes its run, its error message or informational message will not appear on screen until you select the application from the Application menu. That menu will blink and there will be a diamond next to the applications' name in the menu, indicating that it wants your attention. The messages will wait indefinitely for your attention, until they are read and dismissed.
This means that Fetch-O-Matic had to wait too long for something to happen and gave up. This is most likely to occur if a connection was dropped while Fetch-O-Matic was trying to talk to the server (such as negotiating a log-on, but not actually uploading or downloading, which will produce other error messages). This will also occur if Fetch drops a connection but that fact is not correctly reported back to Fetch-O-Matic, more likely to occur with MacPPP/FreePPP than with Open Transport, but always a possibility. This will also occur if you run an application that has an opening dialog to which you don't respond. Eventually, with no activity, Fetch-O-Matic quits.
If some expected file is not where it belongs or its name gets accidentally changed, you will see this message, which will identify the missing file. You will have to make a manual correction of this situation (except in the case of a List Files Prefs file missing from the System's Preferences folder, which can be corrected by running FetchProfile, see second paragraph below).
There are a few circumstances, involving either a crash or quitting a Fetch-O-Matic application while it's running, which will result in this situation. In normal operation, certain files (the changesFile, messageFile, and List Files Prefs) are renamed with temporary names as they are compared or swapped with another version. Should that process be interrupted, there will be a file with either ".old" or ".temp" after the name, with or without the original still in evidence. To recover the changesFile, in general it is safe to merely trash the newer one, if it exists, and then remove the ".old" designation from the other one, restoring the situation to what it was before the interrupted application was run, and then run FetchChanges, which will then insure that you have the latest version. This could possibly result in doubled entries in your To Download file, but that is better than missing something.
In the case of a missing List Files prefs file, the best solution is to run FetchProfile and reset List Files prefs; as long as there is still a "List Files Prefs" file in the Auxiliary folder (whether or not there is a ".temp" file there as well), FetchProfile will recover automatically, preserving the original List Files Prefs file that had been in the Preferences folder before this situation arose.
You may encounter error messages with no description, just a number like "-1055". These may have a number of sources, but usually indicate a network or TCP/IP error, such as a link that went dead during an upload or download.
"Your local changesFile has data not present in the remote file, indicating a prior problem."
When a Fetch-O-Matic application looks at the remote changesFile to determine if activity has taken place, the test it does is that it first reads the local changesFile and stores the first line of that file. It then reads the remote file and looks for that line in the remote file. If there has been no remote activity, the first lines will match and the test is done. If there was remote activity, since the changesFile is written to from the top, the corresponding line in the remote file will have been pushed down with new activity above it. The new activity is everything above this corresponding line. (If you have an empty local file, that is a special case and there is no error - the entire remote file appears as new. If that file is empty too, as when first running Fetch-O-Matic, there is also no error).
If you get this error, it is probably due to a prior failure in uploading the remote changesFile, or other corruption thereof. Whatever the cause, Fetch-O-Matic can have no way to automatically recover in this situation, since it depends entirely on the validity of the remote changesFile. You must then examine your local file and the remote file and try to figure out what went wrong.
If you are collaborating with others in maintaining a website, you will need to contact them because there could be missing data - they might not be aware that the remote file is not up-to-date or accurate, since their local changesFiles may in fact not have any data not present in the remote file.
To correct this situation, first determine what the proper contents of the changesFile ought to be, correct your local file if required, and manually upload that file, replacing the remote version.
The most likely error messages you will see are these, generated by Fetch. The first two are self-explanatory. A common cause is that you have switched to an always-on internet connection (DSL, cable modem, etc.) but have not changed the "usePPP" setting to "N". The "Unexpected response" message means that an upload or download or remote directory change was attempted and failed because an expected file or directory wasn't found . This could be just a connection glitch or other momentary anomaly, so just try again. Another possible cause is that you had two Fetch windows open to your server, changed their order while Fetch-O-Matic was trying to use one of them, and confused Fetch-O-Matic as to which one it was using. When you open multiple connections to a single server, the windows have the same name, and Fetch-O-Matic can't tell the difference between them. It's best not to mess around in this way, although it's perfectly OK to open a new Fetch window to a different server while Fetch-O-Matic is in action. If you absolutely MUST open a new window to your server while Fetch-O-Matic is running, do so while Fetch-O-Matic is in the middle of uploading or downloading a very large file, so that it will not attempt any new action simultaneously with your other activity. Better yet, use a different FTP application such as Anarchie or NetFinder or a browser, which will cause no conflict when run concurrently.
See under Memory Allocation, below.
"<<script>> doesn't understand the <<event xxxx>> message"
This is the result of a missing Scripting Addition (or, under OS 9.1, a side effect of disabling the Security Library extension which must be enabled to make Standard Additions work correctly). Scripting Additions are extensions which add capabilities to AppleScript. A number of Scripting Additions are supplied by Apple as part of the basic AppleScript installation, and some of these are required by Fetch-O-Matic. Fetch-O-Matic also requires the third-party scripting additions discussed under "Installation". There is a list of official Apple Scripting Additions appended to this guide.
This message may be seen in the event of a lost connection to the FTP server. Fetch is not able in this situation to know what has happened, so this can be seen even if you were in the middle of a download when the connection was lost. Therefore, just ignore this message and try again.
"The application Standard Additions has unexpectedly quit..." (Scripting Addition conflict)
This will occur on occasion under System 8.5 or later if you have duplicate Scripting Additions. Most of the Apple additions used in earlier system versions (through 8.1) have been replaced by the single Standard Additions, introduced in 8.5. If you have copied over the obsolete additions from an earlier system folder, you will have a conflict which will produce this or another error. Under 8.5 and later systems, your Scripting Additions folder should have only those Apple additions installed by the installer, plus any non-Apple additions you use such as those required by Fetch-O-Matic.
You will encounter this message if you have tried to upload or download a deleted file, or if there is a mistake in the filename in the "To Upload" or "To Download" file. You may also (erroneously) see this message if Fetch drops a connection during a session. If the file exists, then just resume your upload or download session.
You'll see this message if you have a changesFile or messageFile with the ".old" suffix which you recovered from the trash and put back on your site copy, or if you had a crash while downloading (thus not cleaning up). Fetch-O-Matic will abort when it sees this situation, since it indicates a prior problem, and you will have to intervene and decide what to do. The "old" file is your local copy. If this is the one you want (see about "data not present..."), move the newer file to the trash and remove ".old" from the local file's name.
"Unix or DOS Linefeeds in changesFile"
If you do all file transfers as raw binary when using Fetch-O-Matic, you may get this message, particularly if someone has used another platform to edit this file. As long as it is only edited on a Mac then you won't see this message, unless you have set this file up to be in some other format (as possible with BBEdit and other editors).
Fetch-O-Matic must convert the changesFile to Mac text format, otherwise it can't recognize the items in it properly. You have the option for Fetch-O-Matic to warn you every time it encounters this situation, or to skip warnings in the future. The only side-effect of encountering these linefeeds is that Fetch-O-Matic will run a little slower due to having to perform the conversion.
"Variable logFile not defined" - see OS 9.1 note in the Quick Start section.
This message, or one like it, will appear in the event that your setting for usePPP does not correspond to the software you are using (i.e., you are using Remote Access or PPP but your usePPP setting is set to "F", etc.). Adjust the usePPP setting to reflect what is actually in use.
These error messages look different from the usual Fetch-O-Matic ones; instead of "OK" you'll just see a "Cancel" button. These messages may or may not make sense. They may be caused by insufficient memory (see Memory Allocation). Some may appear if you interrupt a running Fetch-O-Matic application with command-period, and these can be ignored. Other AppleScript errors may result from some temporary system instability, such as after a forced quit. You may get this type of error if you have not installed a required Scripting Addition. Note that these messages probably won't give you a clue to what's actually the problem. If you get such a message, reboot and run FetchProfile and have it check your installation.
If things have been working and you suddenly start seeing messages of this sort, suspect that some portion of Fetch-O-Matic or AppleScript is corrupt, or that you have introduced an extension or other conflict. The first thing to suspect would be the Settings file. If this has been corrupted, there's no telling what might happen. Next, did you install any new extensions, control panels, scripting additions, or change your system in any way? If problems persist and you're sure that your installation of AppleScript or Fetch-O-Matic - or an extension conflict - is not at fault, please contact the author describing the situation in detail.
There is an acknowledged bug in OS 9.1 with AppleScript versions 1.5.5 and 1.6 (and possibly later versions). Portions of Remote Access scripting are broken, affecting scripted dial-up connections. Fetch-O-Matic 2.5 works around this issue, and the workaround has performed well in testing, but it's possible that some users might still see occasional erratic behavior. Symptoms can include failure to dial and type 2 crashes of a Fetch-O-Matic application. (The workaround involves multiple FTP logon attempts which occur before and during the establishment of a PPP link, so when you see that behavior it is normal). If you experience repeated failure to dial, the Fetch-O-Matic application in question has become corrupt and must be replaced. If you have this trouble, there are two ways to avoid the issue completely: 1) Set usePPP to "N" and connect and disconnect manually. 2) Use FreePPP (http://www.rockstar.com/ppp.shtml) instead of Remote Access. FreePPP 2.6.2 is free, compatible (except for the inessential control strip module) with OS versions 8.0 thru 9.1, stable, fast, and is the recommended solution.
For most Fetch-O-Matic applications, the "factory" memory allocation has been set to 2 to 4mb. This is adequate for most situations. However, this will not be enough to handle a big website so and you should allocate additional RAM; use around 6mb for a site with 2,000 to 5,000 files, and add another mb for each additional 2,000 files.
You will get an "Out of Memory" error message, or possibly some other spurious error message, if you have an insufficient allocation. Use the "Get Info" command in the Finder to allocate additional RAM to each Fetch-O-Matic application that asks for it. If you have encountered an "Out of Memory" message, AppleScript may have been left in an unstable state and cause a crash next time you run any AppleScript application, whether or not you increased its memory partition. To be on the safe side, you might want to reboot - or at least, save all open documents.
List Files also needs a lot of RAM for a big site. Unfortunately, List Files is not polite when it runs out of RAM; it simply walks off the job - no crash, no error message - writing an incomplete list. FetchNewLocal has no way to know that the list is short and files will be missed if this happens. Give List Files 5mb for a site of 5,000 files, and add an additional mb for each additional 2,000 files.
It is assumed that the user is familiar with Fetch for uploads and downloads and will set Fetch's preferences appropriately. Due the the many options of Fetch and many possible environments, Fetch-O-Matic sets, at log-on, only the following preferences: 1) the sign-on dialog is disabled (otherwise it would interrupt automatic operations); 2) the cache is ignored so that remote file lists are always refreshed; 3) name changing (encoding file names) is turned off; 4) overwriting files with conflicting names is turned on; and 5) filenames beginning with a period will be ignored (if this were not done, directories such as "." will cause endless looping in Fetch-O-Matic because it will think that's a different directory, which it isn't). All of these preferences are restored to their previous state at log-off.
A user reported that having MacBinary enabled was the source of some trouble, but this subject has not been fully investigated. There would be no reason to use MacBinary for maintaining an HTTP site, but there is no reason in theory why MacBinary should be incompatible with Fetch-O-Matic.
Q. Every time I run FetchUp it asks me for the location of Fetch,
this is driving me nuts. What can I do about this?
A. In your Settings file, you must enter the complete, exact
case-sensitive name for the application as it exists on your hard
drive, as in "Fetch 3.0.3" or whatever you've renamed it to. The name
is not important, but it must match character-for-character the name
in the Settings file.
Q. I get an error "<<script>> doesn't understand the
<<event xxxx>> message" when I try to run FetchProfile,
and then it quits. What's wrong?
A. You may be missing an AppleScript Scripting Addition. Read the
section on this topic.
Otherwise, If you have just installed all the required additions and
get this message, then reboot - your AppleScript cache needs to be
updated.
Q. I download huge files and sometimes get dumped off in the
middle of a transfer. Fetch has capabilities to resume partial
downloads. What about Fetch-O-Matic?
A. Fetch author Dave Matthews has resumed development, and this
capability is likely to be incorporated in a future release. Version
3.0.3 of Fetch does not permit scripted resumption. See reconnect
for more information on interruptions.
Q. I ran FetchCompare, it saw it crawl through my site, but then
it seemed to be doing nothing forever, so I did a force quit. Why
doesn't it work?
A. On large sites (3,000+ files), FetchCompare takes a really, really
long time (a couple of hours) to compare the remote site to your
local copy. Due to subtle differences between UNIX and Mac file
ordering, the comparison has to be exhaustive, and since the
remote-site list creation is slow anyway, no effort was made to speed
this up. Run it overnight.
Q. All those little applications are confusing. Why don't you wrap
the whole package up in a single application with a menu
interface?
A. I kind of like it this way, actually. It's already like a menu in
List view. Or you could make some aliases and put them in your
launcher or whatever... There aren't enough rainy days for me to do
this.
Q. I'd like to run FetchNewLocal or FetchMe while I'm doing an
upload and have it add files to the upload session in progress. Can I
do this?
A. Yes, this capability has been implemented; see Adding
to an Upload in Progress.
Q. I'd like to have a continuously watched local folder and have
anything dumped in there uploaded automatically. Can Fetch-O-Matic do
this?
A. Yes, this capability has been implemented; see FetchContinual.
Q. I'd like to have a continuously watched folder on the remote
site and have anything dumped in there downloaded automatically. Can
Fetch-O-Matic do this?
A. Yes, this capability has been implemented; see FetchHarvest.
Q. I have a problem I can't seem to solve. If I write to you, will
I get an answer?
A. Yes. Please include 1) the results of a run of FetchProfile; 2)
any relevant error messages; 3) a copy of your Settings file (minus
password, if present); and 4) a copy of the FTP session transcript
("Fetch Transcript" under Fetch's "Windows" menu). This will expedite
a solution. I like user reports because that's how bugs (and errors
in this guide) get eliminated. Send mail to egray@kagi.com.
Q. I have a great idea for a new feature, and also I found a bug.
If I write you, will you do anything about it?
A. Quite likely. Almost all of the features which have been added
since the initial release of Fetch-O-Matic have come about due to
user requests. Bug reports are always welcome too and acted upon.
Send mail to egray@kagi.com.
The Fetch-O-Matic home page is http://www.filmscouts.com/software/fom.asp. This page will always have the latest version of Fetch-O-Matic and links to other required software.
In general, the easiest way to find shareware and get updates is to use the search capabilities of the two most user-friendly Mac shareware sources: the Info-Mac HyperArchive at the Massachusetts Institute of Technology (http://hyperarchive.lcs.mit.edu/HyperArchive/SearchForm.html); and Shareware.com (http://shareware.cnet.com/).
Info-Mac is updated almost daily and maintains a huge archive of software and has abstracts which you can read before deciding to download.
Shareware.com is a search engine and also shows abstracts of the software it finds. One can sometimes find things here which elude a search of Info-Mac. A great feature is that the abstracts are summarized in the search results, which doesn't happen at Info-Mac. If you find what you're looking for, you can choose a download location near you, which may be a good idea if you are not in North America. Shareware.com also publishes "reliability" ratings for the various download sites.
You should always be able to find the latest versions of Fetch-O-Matic, Fetch, Jon's Commands, and List Files at either of these locations. (As noted earlier, type "jons" when searching for "Jon's Commands").
Finally, http://www.apple.com/applescript
is the place to go for AppleScript itself and scripting information,
scripting software, and links to scripting sites.
Fetch-O-Matic is distributed as shareware. The version you have is the complete package; there is no "lite" or "demo" version, nor will this version expire sometime in the future. You may run each application 30 times as a trial. If you continue to use Fetch-O-Matic after that, you are obligated to pay the registration fee.
Fetch-O-Matic has the following pricing:
1 to 10 single user licenses: $20 per user
11+ single user licenses: $15 per user
A Site License costs $400 (roughly equal to 27 users) and covers all locations for your organization within a 160 kilometer radius of your site (100 miles). One advantage of a Site License is that you do not need to keep track of how many people at your site are using the software.
A World-Wide License costs $1600 and it covers all locations for your organization on the planet earth.
Paying for Fetch-O-Matic is fairly simple. Open the Register program that is in the folder with this file. Enter your name, your e-mail address, and the number of single user licenses you desire to purchase (or Site or World-Wide licenses). Save or Copy or Print the data from the Register program, and send the data and payment to Kagi, a clearing house which handles all payment processing for Fetch-O-Matic.
If paying with Credit Card or First Virtual, you can e-mail or fax the data to Kagi. Their e-mail address is sales@kagi.com and their fax number is +1 510 652-6589. You can either Copy the data from Register and paste it into the body of an e-mail message or you can Save the data to a file and you can attach that file to an e-mail message. There is no need to compress the data file, it's already pretty small. If you have a fax modem, just Print the data to the Kagi fax number.
Payments sent via e-mail are processed within 3 to 4 days. You will receive an e-mail acknowledgment when it is processed. Payments sent via fax take up to 10 days and if you provide a correct Internet e-mail address you will receive an e-mail acknowledgment.
If you are paying with Cash or USD Check you should print the data using the Register application and send it to the address shown on the form, which is:
Kagi
1442-A Walnut Street #392-9A
Berkeley, California 94709-1405
USA
You can pay with a wide variety of cash from different countries, but at present, if you pay via check, it must be a check drawn in US Dollars. Kagi cannot accept checks in other currencies; the conversion rate for non-USD checks is around USD 15.00 per check so this is not practical.
If you have a purchasing department, you can enter all the data into the Register program and then select Invoice as your payment method. Print three copies of the form and send it to your accounts payable people. You might want to highlight the line that mentions that they must include a copy of the form with their payment.
Kagi cannot invoice your company. You need to generate the invoice and handle all the paperwork on your end.
Please do not fax or e-mail payment forms that indicate Cash, Check or Invoice as the payment method but don't contain the payment. (Last time we checked, there was still no technology to transfer physical objects via fax or e-mail, and without the payment the form cannot be processed).
Payments sent via postal mail take time to reach Kagi and then up to 10 days for processing. Again, if you include a correct e-mail address, you will hear from Kagi when the form is processed.
Fetch-O-Matic has a shareware registration reminder dialog which appears after 30 runs. When you pay, you will receive a code to enter in the Fetch-O-Matic Settings file to turn off this reminder. If you do not have an e-mail address, please enter your complete postal address - and please remember that we do not know what country you live in, so please enter that into the postal address also.
If you do not have an e-mail address you should consider selecting the Postcard Receipt so that Kagi can inform you of your registration code. Kagi transmits the registration codes via e-mail and paid postcard receipt only.
Fetch-O-Matic is pretty robust and doesn't make mistakes on its own, but it is a potent tool. Improper configuration or other operator error can result in lost data due to the fact that Fetch-O-Matic overwrites existing files. Before using Fetch-O-Matic, and at all times in any event, you should have a backup of your website.
USE AT YOUR OWN RISK. BY USING THESE APPLICATIONS, THE USER ACKNOWLEDGES THAT THE AUTHOR AND DISTRIBUTOR WILL ACCEPT NO RESPONSIBILITY OR LIABILITY FOR LOST DATA, LOST TIME, MENTAL ANGUISH, ILL HEALTH, OR ANY OTHER CONSEQUENCES RESULTING FROM SUCH USE. NO GUARANTEE OF FUNCTIONALITY OR APPLICABILITY FOR A PARTICULAR SITUATION IS GIVEN OR IMPLIED. WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE EXPRESSLY DISCLAIMED. CONSEQUENTIAL AND INCIDENTAL DAMAGES ARE EXPRESSLY EXCLUDED.
The following screens are accessed from the Options menu in List Files. They are shown here as they should be configured for use with Fetch-O-Matic. (Note: some selections are not relevant, but it's probably a waste of time to try to figure out which ones you can ignore). The final screen, "File Saving Options", has 3 options you should set: the destination folder for List Files' output file, which must be the Fetch-O-Matic Auxiliary folder, the file's suffix (which should be as shown), and the file's creator (shown here as ttxt, the code for SimpleText). If you are using BBEdit, its creator code is R*ch; Tex-Edit Plus uses TBB6, but in any case you can click the "Select" button to choose your text application.
<
OS 7.0 -> 8.1: the following components should have been installed in your Scripting Additions folder by the System Installer or AppleScript installer (if you are missing any of these, then you will need to reinstall AppleScript): Beep, Current Date, Display Dialog, File Commands, Numerics, Read/Write Commands, Run Script, String Commands.
OS 8.5 and later: All of the above were rolled into the single "Standard Additions", which replaces them. Note: you should remove all of the obsolete Apple additions (which you may have copied over from a prior System folder) or you may have a Scripting Addition conflict problem.
This is not a complete list of official additions; these are just the ones required by Fetch-O-Matic. It is recommended that you leave all the official additions in place.
The suite contains the following files: In the topmost folder, the applications FetchChanges, FetchDown, FetchMe, FetchNewLocal, FetchTimeStamp, and FetchUp; the text files Activity Log, To Download, To Upload, and Fetch-O-Matic README; and the folders "Fetch-O-Matic Auxiliary" and "Fetch-O-Matic Utilities".
The "Fetch-O-Matic Auxiliary" folder contains the Register application, the text files "Fetch-O-Matic Settings" and "remote site filelist"; this User Guide with accompanying folder "User Guide Illustrations", a "List Files Prefs" file, a folder "Stuff for Scriptg. Addns." containing the AppleScript scripting additions FreePPP Control (by Sebastian Kloska), MacPPP Control (by Mark Alldritt), PassWord (by Donald Olson), and setDate (by Mikael Flensborg); and the folder "Test & Doc - FreePPP Ctrl" with a test application and ReadMe file re FreePPP Control. Two additional files will appear in this folder: a <sitename> filelist file created by FetchNewLocal, which is a catalog of your local mirror, and a file "FetchNewLocal Log" which stores the time of last run for both FetchNewLocal and FetchContinual.
The "Fetch-O-Matic Utilities" folder contains the applications FetchCompare, FetchContinual, FetchDelete, FetchHarvest, FetchMeNot, FetchProfile, and FetchStructure, and the text files To Delete Local, To Delete Remote, Unique In Local, Unique In Remote, Never Upload, and Never Download. Files named "F-O-M Profile <date>", created by FetchProfile, and "Site Structure <date>", created by FetchStructure, will appear in this folder as well. These last two types are not re-used and may be trashed.
Note about updates for users of previous versions:
There is a new feature in v. 2.5 which adds a new setting to the Settings file ("harvestCleanup"). Rather than retyping your settings, you may cut and paste between these files. The recommended procedure, after backing up your current settings, is to copy only the new setting and description from the just-downloaded file into your old Settings file, and then option-drag that file to the new Auxiliary folder, replacing the just-downloaded one. You should similarly also option-drag your existing "To Upload", "To Download", "Activity Log" (and any other text files you want to keep) to their proper corresponding locations in the new folders. When you're all done, drag _everything_ in the new folders back to the original folders, overwriting what's there. FetchProfile should then be run as a check that everything is OK. The advantage to taking this rather convoluted approach is that any aliases you have created to Fetch-O-Matic applications or files will still function correctly; if you have no such aliases, then you needn't bother with all of this.
New settings in v. 2.4: noUploadFolders, noDownloadFolders. (The setting ignoreFolder is no longer used and should be removed, but will do no harm). New files, "Never Upload" and "Never Download", were added to the Utilites folder.
New settings in v. 2.2 and later: altWord, altLogOnName, altRemoteHost, altRemoteHome, altDirHead, timeStampFolders, harvestTimes, ignoreChanges, ignoreFiles, closeFTP, onWhileDigesting; (added in v. 2.03: reconnect, ignoreFolder).
-------------------------------------------------------
V. 2.5 (April 2001) adds the option to delete remote files after download by the FetchHarvest module, and provides improved compatability with OS 9.1.
V. 2.4 (December 29, 2000) offered a more flexible exclusion mechanism for files, allowing separate Never Upload and Never Download lists of specific files to ignore for either uploading or downloading. A similar flexibility was also provided for directories, with separate noUploadFolders and noDownloadFolders lists for excluding folders and their contents. Unattended operation of FetchContinual and FetchHarvest was implemented via a dialog-suppression option, such that these applications may be launched by a scheduling application or by aliases placed in the Startup Items folder of a machine subject to unattended reboots.
V 2.3 (December 11, 1999) is Y2K compliant (now using 4-digit year for file modification dates), accommodates the changed file ordering in Jon's Commands 2.1 (while maintaining compatibility with earlier versions of Jon's Commands), and fixes an obscure bug (thanks, Mickey).
V. 2.2 (August 1, 1999) introduced a new module, FetchHarvest, which provides continuous (by interval or at set times of day) scanning for and downloading of new files and folders appearing (un-announced) in a remote "drop" directory. "Harvested" files may optionally be directed to new folders named by date and time of download. Introduced also was the capability for transparently handling dual hosts (for use when, for example, streaming video is served from a separate machine with the same logical but different actual IP address). Also, a problem with applications sometimes choking on huge upload chunks (many thousands of files at once) was worked around. It is now possible to ignore the remote changesFile, a time saver if there is only one person maintaining a site. FetchContinual and FetchNewLocal now interact to pass the last time either was run to the other, preventing redundant uploads when used interchangeably. New settings permit skipping files with certain extensions, and leaving the FTP connection open at the end of a session. Certain requisite Fetch preferences are now set automatically at log-on and reset to their prior values at log-off.
V. 2.1 (April 7, 1998) introduced a new module: the FetchContinual application, which provides continuous scanning for and uploading of new and altered files.
V. 2.0.4 (March 4, 1998, beta only) introduced a new feature, the ability to add files to an upload in progress. Worked around a bug in Jon's Commands 2.0.3 which affected FetchProfile and, rarely, FetchNewLocal (causing a Finder crash).
V. 2.0.3 (December 28, 1997) fixed all remaining bugs reported in the initial months of release; added two new features, "reconnect" and "ignoreFolder" (now discontinued, replaced by "noUploadFolders" and "noDownloadFolders").
V. 2.0.2 (Oct. 26, 1997) fixed a potential problem with non-U.S. and customized date formats (set via the Date and Time control panel), and a small bug which occurred under OS 8 if List Files was already running before launching FetchNewLocal. Also, some error messages were rewritten to be more informative, and there were minor improvements to the User Guide.
V. 2.0.1 (Oct 23, 1997) fixed a bug in v.2.0 which affected certain users, rendering Fetch-O-Matic inoperable on their machines.
Version 2.0: First public release, October 19, 1997
Special thanks to Jeremy Posner.
Thanks to Mark Alldritt for allowing free distribution of the
MacPPP Control Scripting Addition.
Thanks to Mikael Flensborg for permission to distribute the setDate
Scripting Addition.
Thanks to Sebastian Kloska for permission to distribute the FreePPP
Control Scripting Addition.
Thanks to Jim Matthews, author of Fetch, for making it so complete,
solid, and scriptable.
Thanks to Alessandro Levi Montalcini for List Files and other great
shareware.
Thanks to Jon Pugh for Jon's Commands and many other contributions to
the Mac community.
Thanks to Apple Computer for providing the scripting environment to
make all this possible.
Thanks to all the generous contributors to the Mac Scripting Systems
list, an invaluable resource.
Thanks to my two cats, One-Many and Too-Many, who often reminded me to eat.