Simple Mirror Utility
---------------------

This utility performs a configurable wget based mirror:

'wget --continue --no-clobber --no-host-directories --no-parent --recursive --no-verbose --accept <accepted list of files> --directory-prefix=/where/stuff/will/be/downloaded repo_url/repo_name --append-output=log_file'

Installation:
-------------

git clone https://[email protected]/vramiro/easy-wget-mirror.git
change config as you need (check next section)

run manually:

./easy-wget-mirror ./conf/easy-wget-mirror.conf

run with crontab

# each day at 00:01 or whatever
0 1 * * *      /complete/path/to/main.py /complete/path/to/config

Note: be sure that the user will have *chown* rights for the user in the configuration

Configuration:
--------------

-Default values

[easy-wget-mirror]
#mirror.active.list= comma separated values list with section names
mirror.active.list=mirrorA, mirrorB

#plugin.active.list= comma separated values list with plugins module names (they should be located at plugins folder)
plugin.active.list= pluginA, pluginB

#wget.path= path to wget executable
wget.path=/opt/local/bin/wget

- Then, for each mirror name in mirror.active.list you should provide a section as follows:

[repo_name]
url= repo base url
repositories= file with the repositories list, one repo per line. Saved in conf/
files.pattern= mirror accepted file patters (*.some_extension)
destination= mirror downloads destination folder
logs= mirror logs destination folder

- Each plugin could have special configs. by now:

[RemoveFiles]
#patterns= a list of separated comma values with a valid python regular expresion
patterns = '^@.*','*.html$'

[ChangeOwner]
#user= user owner for chmod the downloaded files
user=_www
#group= group owner for chmod the downloaded files
group=_www

[RunDate]
#format= a valid python datetime format
format=%Y/%m/%d %H:%M:%S

[UpdateReposList]
#update.repos.list= a list of comma separated entries. Each entry is mirrorName=>url
update.repos.list=mirrorA=>http://foo/bar,mirrorB=>http://fooo/baar


Author:
-------

Victor Ramiro
<[email protected]>
http://victor.cl