Some Apache servers do activate the DirectoryIndex so you can easily go through the directories listing over HTTP and download some files. It’ s often used for giving possibility to people to download multiple releases for a package/software.
But in some cases, it can be useful to download the content of a full tree view.

This can be done by using a simple tool available by default on Linux (can be downloaded on Windows/Mac): wget

Let’s take an example!

Without proxy

We want to download all the files on https://pkg.mywebsite.eu/releases/ which contains all the latest releases of our preferred soft.

wget -r -np -nH --cut-dirs=1 -l 15 -R index.html https://pkg.mywebsite.eu/releases/

Here, we are using many options to say to wget what we want to do:

• -r : Turn on recursive retrieving (max 5)
• -np : Do not ever ascend to the parent directory when retrieving recursively
• -nH –cut-dirs=1 : Disable generation of host-prefixed directories, and remove 1 level (here, remove “pkg.mywebsite.eu” and create a “releases” dir)
• -l 15 : Change default depth for downloading to 15 levels (default is 5)
• -R index.html : Specify comma-separated lists of file name suffixes or patterns to reject

With proxy

Let’s now add a complexity, this server is only available through a proxy to protect the content. The proxy is available on your localhost (127.0.0.1) on port 3128 (default port for Squid Proxy):

wget -r -np -nH --cut-dirs=1 -l 15 -R index.html  -e use_proxy=yes -e http_proxy=127.0.0.1:3128 https://pkg.mywebsite.eu/releases/

Here, we just added 2 options to be able to send exactly the same request but using proxy:

• -e use_proxy=yes : Execute command to enable proxy
• -e http_proxy=127.0.0.1:3128 : Execute command to give URL to use for HTTP proxy