Mirror by ftp
It is a good idea to read the mirroring Scientific Linux helpful tips before proceeding.
ftp is one of the prefered ways that we would like people to mirror Scientific Linux. We have found that while wget is quite good, lftp has many features that make it preferable.
Some of the functions that I like is the ability to use a script, parallel downloads, and being able to put the processes into the background easily. Below are two sample config files that demonstrate some of different ways to do the mirroring.
Mirror in the background (no output)
lftp -f download.background
This will mirror both 5x and 5rolling at the same time, in the background. You will not get any output from this. The only way to check it is by doing a 'ps' and seeing if lftp is still running. This is good if you are mirroring the first time and just want it to get everything.
set ftp:list-options -a set cmd:fail-exit true open ftp.scientificlinux.org:/linux/scientific/4x lcd /export/linux/scientific/5rolling mirror --delete --exclude sites/Fermi --exclude errata/debuginfo --exclude errata/obsolete & open ftp.scientificlinux.org:/linux/scientific/5x lcd /export/linux/scientific/5x mirror --delete --exclude sites/Fermi --exclude archive/debuginfo --exclude archive/obsolete & quit
Mirror one at a time (nice output)
lftp -f download.verbose > /tmp/lftp.scientific.output
This will mirror both 5x and 5rolling. It will not do this in the background, but it will save all the output into a file. Having verbose set to 1, will just output the changes, but nothing else. You can have more output by changing the verbose= to a higher number in download.verbose. (verbose=4 is the highest)
This method is great for cron scripts, so that you can see what get's changed.
set ftp:list-options -a set cmd:fail-exit true open ftp.scientificlinux.org:/linux/scientific/4x lcd /export/linux/scientific/5x mirror --delete --exclude sites/Fermi --exclude errata/debuginfo --exclude errata/obsolete --verbose=1 open ftp.scientificlinux.org:/linux/scientific/5x lcd /export/linux/scientific/5rolling mirror --delete --exclude sites/Fermi --exclude archive/debuginfo --exclude archive/obsolete --verbose=1 quit
The line "set cmd:fail-exit true" causes lftp to exit upon a failure of a command. Without this line lftp just goes on to the next line. This can have unwanted effects.
One scenerio would be if your /export/linux/scientific partition was remotely mounted, and one day that mount failed. When your nightly cron job ran, and it tried to do "lcd /export/linux/scientific/5x" and failed, lftp would be at the path "/". It would then skip to the next command which begins "mirror --delete ". If you are wondering if lftp really would replace your entire operating system with a mirror of Scientific Linux, the answer is yes.
If you do not want to put the "set cmd:fail-exit true" in each of your scripts, you can put it in your lftp.conf file, found at /etc/lftp.conf in Scientific Linux.
Many thanks go to Justin Anderson for finding this solution.
Last modified 2011-10-18 04:02 PM