Skip navigation

For those that haven’t had the chance to use it, wget is a powerful gnu application that allow to retrieve content non interactively from the web, which makes it ideal to an array of sys-admin tasks. It gets a bit hard to tame so much power and you need to craft the outcome by carefully picking inside the command options. Something practical, like downloading the entire or partial content from inside a folder and its sub-folder, for example, requires a couple of parameters to be passed so not to get confused by the unintuitive way the program goes about retrieving upper-level directories along with the desired stuff.
When you specify the recursive option (-r) to address a site path like http://somesite.com/folder/subfolder/ you would reasonably expect to get every content from that point down the directory tree (much in the fashion that cp -rf would do it, coping all in the hierarchy below) but given the weird behavior of fetching other directories above, the extra option ‘–no-parent’ proves essential.

Along with those two I normally include a couple of extra parameters:

:: ‘-nv; (or ‘–no-verbose‘) that turns off verbose output, without being completely quiet, which means that errors messages and basic information still get printed;

:: ‘-nH‘ (‘or –no-host-directories‘) that disable the generation of host-prefixed directories. By default, invoking wget with ‘-r http://somesite.org’ will create a structure of directories beginning with http://somesite.org

The hypothetical example then will look something like:

wget -r -np -nv -nH  http://www.addressedSite.info/coolJavascriptLib/
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: