Wget - Looking for a spidering solution

Wget is a Unix program that recursively downloads a site. Now, there's
a Win32 version of Wget!

Wget doesn't have a nice frontend as Teleport or Blackwidow: it just
does what it has to do, and it does it well.

You can download from HTTP or FTP and you can specify level depths,
filetypes, domains.
It supports background operation (it runs completely unattended), HTTP
cookies, resume of canceled downloads, etc.

Plus, its free and open source :)

I think it's definitly what you're looking for.

The main page of the project is:

http://www.gnu.org/software/wget/wget.html

The main page of the Windows port is

http://space.tin.it/computer/hherold/

You can download it from here:

ftp://sunsite.dk/projects/wget/windows/
ftp://sunsite.dk/projects/wget/windows/wget20020612b.zip

Also there's a windows frontend (it uses the same program, wget but it
has a nice frontend)

http://www.jensroesner.de/wgetgui/

You can see a screenshot here
http://www.jensroesner.de/wgetgui/#screen



For all answers: click here