What is wget command?
The ability to download content from the world wide web (the internet) and store it locally on your system is an important feature to have. The wget(Web get) command line utility is a freely available package which can be used to retrieve files from the web using HTTP, HTTPS, and FTP protocols. In addition to the mentioned protocols, wget can also be used to download files through HTTP proxies. This is similar to cURL command which we covered recently.
One very useful feature of wget is that it is non-interactive and therefore can be easily used in scripts and cron jobs.
This allows wget to be used to download files as part of triggering a specific action or retrieve files at a specific point in time.
It is also worth noting that wget is designed to work with unreliable network connections and has the ability to resume downloads interrupted due to network disruptions.
We can use wget to download files recursively and also set the number of times wget will try to resume an interrupted download.
In this article, we show you some useful examples implementing the features of the wget command we’ve mentioned thus far.
Before proceeding with the examples, let’s first verify that the command is available on the system.
[root@linuxnix ~]# which wget /usr/bin/wget [root@linuxnix ~]# rpm -qf /usr/bin/wget wget-1.12-10.el6.x86_64 [root@linuxnix ~]#
If the command is not available, you may install it.
Install wget in RedHat/CentOS based systems
yum install wget
Install wget in Debian/Ubuntu based systems
apt-get install wget
You may also download the package from the URL http://ftp.gnu.org/gnu/wget/.
Now, let’s move on to the examples.
Example 1: Download file via HTTP
For our first example, we’ll download the Centos DVD via the HTTP protocol.
[root@linuxnix ~]# wget http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso --2017-11-20 17:03:18-- http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso Resolving centos.mirror.net.in... 103.240.105.131 Connecting to centos.mirror.net.in|103.240.105.131|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 3972005888 (3.7G) [application/octet-stream] Saving to: “CentOS-6.9-x86_64-bin-DVD1.iso” 0% [ ] 190,117 14.6K/s eta 3d 9h
How wget command works?
The wget command first resolves the hostname mentioned in the URL to its corresponding IP address. After that, a connection on port 80 is established. An HTTP request is sent and if the response to the request is 200 indicating success, then the size of the file to be downloaded is calculated and the file download is started.
As soon as the download starts wget creates a file with the same name as the file being downloaded in the current working directory.
[root@linuxnix ~]# ls -ltrh CentOS-6.9-x86_64-bin-DVD1.iso -rw-r--r--. 1 root root 373K Nov 20 17:04 CentOS-6.9-x86_64-bin-DVD1.iso
You may monitor the size of this file. It should increase as the download progresses.
Example 2: Download a file with different name from the source
In this example, we will download the same file but this time we will save it with a different name on our local system.
We use the -O option to specify the output file name.
[root@linuxnix ~]# wget -O CentOS-6_9.iso http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso --2017-11-20 17:13:43-- http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso Resolving centos.mirror.net.in... 103.240.105.131 Connecting to centos.mirror.net.in|103.240.105.131|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 3972005888 (3.7G) [application/octet-stream] Saving to: “CentOS-6_9.iso” 0% [ ] 1,158,917 46.5K/s eta 11h 50m
As soon the download begins the file CentOS-6_9.iso is created in the current working directory of my system from where I executed the wget command and the size of the file increases as the download progresses.
[root@linuxnix ~]# ls -ltrh CentOS-6_9.iso -rw-r--r--. 1 root root 956K Nov 20 17:14 CentOS-6_9.iso
Example 3: Download a file in the background
In the examples, we’ve seen so far, when we execute the wget command we do not get our prompt back until the download is completed.
We could run type & after the command and put the process in the background.
But it’s useful to know that we can use -b option with the wget command to download the file in the background and get our command prompt back immediately after executing the command.
[root@linuxnix ~]# wget -b http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso Continuing in background, pid 3496. Output will be written to “wget-log”.
As you may notice from the above output that when used with the -b option the wget command is pushed to the background.
We get our command prompt back immediately and the PID of the process running wget is displayed along with the name of a log file which we may query to see the progress of the download.
Here’s a snippet from the log file wget-log:
[root@linuxnix ~]# tail -f wget-log Saving to: “CentOS-6.9-x86_64-bin-DVD1.iso” 0K .......... .......... .......... .......... .......... 0% 33.3K 32h20m 50K .......... .......... .......... .......... .......... 0% 41.4K 29h11m 100K .......... .......... .......... .......... .......... 0% 20.3K 37h7m 150K .......... .......... .......... .......... .......... 0% 46.7K 33h36m 200K .......... .......... .......... .......... .......... 0% 35.8K 32h54m 250K .......... .......... .......... .......... .......... 0% 5.98K 2d9h 300K .......... .......... .......... .......... .......... 0% 38.1K 2d5h 350K .......... .......... .......... .......... .......... 0% 24.9K 2d4h 400K .......... .......... .......... .......... .......... 0% 21.2K 2d3h
Example 4: Resume an interrupted download
I had terminated the download we started in one of our previous examples by pressing ctrl+c. We can use the -c option to resume this download from where we left off.
This holds true provided we have not removed the part of the file which had been downloaded before it got interrupted.
If we’ve removed the file part then the download even with -c option will be treated as a fresh download.
[root@linuxnix ~]# wget -c http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso --2017-11-20 17:28:16-- http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso Resolving centos.mirror.net.in... 103.240.105.131 Connecting to centos.mirror.net.in|103.240.105.131|:80... connected. HTTP request sent, awaiting response... 206 Partial Content Length: 3972005888 (3.7G), 3971332771 (3.7G) remaining [application/octet-stream] Saving to: “CentOS-6.9-x86_64-bin-DVD1.iso” 0% [ ] 1,284,570 15.2K/s eta 35h 4m
Example 5: Change destination location
Up till now the files that we have downloaded were stored in the same location as the current directory from where we were executing the wget command. If needed we may modify the destination path location for the file download using the –directory-prefix option as shown below.
[root@linuxnix ~]# wget --directory-prefix=/tmp http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso --2017-11-20 17:37:51-- http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso Resolving centos.mirror.net.in... 103.240.105.131 Connecting to centos.mirror.net.in|103.240.105.131|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 3972005888 (3.7G) [application/octet-stream] Saving to: “/tmp/CentOS-6.9-x86_64-bin-DVD1.iso” 0% [ ] 5,317 1.15K/s eta 67d 10h
The above command will download the file CentOS-6.9-x86_64-bin-DVD1.iso in the /tmp directory.
Example 6: We can force wget to be quite when doing the work with quite mode(-q)
In the examples we’ve seen so far, wget printed verbose output regarding the HTTP connection being established and the progress of the download.
If we do not want to view this information then we can use the -q option to run the command in quite mode.
[root@linuxnix ~]# wget -q http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso
If you are downloading a large file you should certainly avoid using quite mode.
Example 7: Download multiple files
We can specify more than one URL to download files from while using the wget command.
Here is an example where we use the wget command to fetch the wget package from the GNU website via HTTP as well as FTP protocols.
[root@linuxnix ~]# wget http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz ftp://ftp.gnu.org/gnu/wget/wget-1.7.1.tar.gz --2017-11-20 17:51:31-- http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz Resolving ftp.gnu.org... 208.118.235.20, 2001:4830:134:3::b Connecting to ftp.gnu.org|208.118.235.20|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 446966 (436K) [application/x-gzip] Saving to: “wget-1.5.3.tar.gz” 100%[==========================================================================================>] 446,966 86.8K/s in 5.0s 2017-11-20 17:51:51 (86.8 KB/s) - “wget-1.5.3.tar.gz” saved [446966/446966] --2017-11-20 17:51:51-- ftp://ftp.gnu.org/gnu/wget/wget-1.7.1.tar.gz => “wget-1.7.1.tar.gz” Connecting to ftp.gnu.org|208.118.235.20|:21... connected. Logging in as anonymous ... Logged in! ==> SYST ... done. ==> PWD ... done. ==> TYPE I ... done. ==> CWD (1) /gnu/wget ... done. ==> SIZE wget-1.7.1.tar.gz ... 1035518 ==> PASV ... done. ==> RETR wget-1.7.1.tar.gz ... done. Length: 1035518 (1011K) (unauthoritative) 100%[==========================================================================================>] 1,035,518 60.3K/s in 14s 2017-11-20 17:52:13 (70.5 KB/s) - “wget-1.7.1.tar.gz” saved [1035518] FINISHED --2017-11-20 17:52:13-- Downloaded: 2 files, 1.4M in 19s (74.7 KB/s)
I would like to point out here that the downloads are serial and not parallel i.e. one file download completes after which the second download starts.
Both files are not downloaded together.
Example 8: Download files from URL list contained in a text file
We can put the list of URLs we’d like to download files from within a text file and then feed that file to the wget command using the -i option.
We’ll use the two URLs we used in our previous example to demonstrate this.
I’ve added the URLs to a file named url.txt.
[root@linuxnix ~]# cat url.txt http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz ftp://ftp.gnu.org/gnu/wget/wget-1.7.1.tar.gz [root@linuxnix ~]#
Now we’ll feed this file to the wget command with the -i option as shown below:
[root@linuxnix ~]# wget -i /root/url.txt --2017-11-20 18:01:38-- http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz Resolving ftp.gnu.org... 208.118.235.20, 2001:4830:134:3::b Connecting to ftp.gnu.org|208.118.235.20|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 446966 (436K) [application/x-gzip] Saving to: “wget-1.5.3.tar.gz” 100%[==========================================================================================>] 446,966 81.2K/s in 5.4s 2017-11-20 18:01:51 (81.2 KB/s) - “wget-1.5.3.tar.gz” saved [446966/446966] --2017-11-20 18:01:51-- ftp://ftp.gnu.org/gnu/wget/wget-1.7.1.tar.gz => “wget-1.7.1.tar.gz” Connecting to ftp.gnu.org|208.118.235.20|:21... connected. Logging in as anonymous ... Logged in! ==> SYST ... done. ==> PWD ... done. ==> TYPE I ... done. ==> CWD (1) /gnu/wget ... done. ==> SIZE wget-1.7.1.tar.gz ... 1035518 ==> PASV ... done. ==> RETR wget-1.7.1.tar.gz ... done. Length: 1035518 (1011K) (unauthoritative) 100%[==========================================================================================>] 1,035,518 62.9K/s in 17s 2017-11-20 18:02:16 (60.1 KB/s) - “wget-1.7.1.tar.gz” saved [1035518] FINISHED --2017-11-20 18:02:16-- Downloaded: 2 files, 1.4M in 22s (65.2 KB/s) [root@linuxnix ~]#
Example 9: Downloading files from sites requiring authentication
A very useful feature of the wget command is that it allows us to download files from password protected sites.
In this example, we’ll download a file via FTP from my FTP server by providing credentials for my username.
[root@linuxnix ~]# wget --user=sahil --password=$@F1_p@$$w0rd ftp://192.168.188.129/test.txt --2017-11-20 18:53:58-- ftp://192.168.188.129/test.txt => “test.txt” Connecting to 192.168.188.129:21... connected. Logging in as sahil ... Logged in! ==> SYST ... done. ==> PWD ... done. ==> TYPE I ... done. ==> CWD not needed. ==> SIZE test.txt ... 20 ==> PASV ... done. ==> RETR test.txt ... done. Length: 20 (unauthoritative) 100%[==========================================================================================>] 20 --.-K/s in 0s 2017-11-20 18:53:58 (58.5 KB/s) - “test.txt” saved [20] [root@linuxnix ~]# ls -l test.txt -rw-r--r--. 1 root root 20 Nov 20 18:53 test.txt
This is awesome but if you are downloading a big file then anyone could view your password when listing processes in the ps -ef command.
A safer way is to use the –ask-password option instead of –password so that wget will prompt you for the password.
[root@linuxnix ~]# wget --user=sahil --ask-password ftp://192.168.188.129/test.txt Password for user “sahil”: --2017-11-20 18:56:59-- ftp://192.168.188.129/test.txt => “test.txt.1” Connecting to 192.168.188.129:21... connected. Logging in as sahil ... Logged in! ==> SYST ... done. ==> PWD ... done. ==> TYPE I ... done. ==> CWD not needed. ==> SIZE test.txt ... 20 ==> PASV ... done. ==> RETR test.txt ... done. Length: 20 (unauthoritative) 100%[==========================================================================================>] 20 --.-K/s in 0s 2017-11-20 18:56:59 (75.5 KB/s) - “test.txt.1” saved [20]
Example 10: Limit download speed
While using wget we could also instruct it to limit the download speed if we are on a slow connection by using the –limit-rate=100k option.
[root@linuxnix ~]# wget --limit-rate=100k -o wget_logs.txt http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz [root@linuxnix ~]#
In this example, we’ve also used the -o option to redirect the output of the wget command to a file named wget_logs.txt.
The contents of the file are shown below:
[root@linuxnix ~]# cat wget_logs.txt --2017-11-20 18:59:59-- http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz Resolving ftp.gnu.org... 208.118.235.20, 2001:4830:134:3::b Connecting to ftp.gnu.org|208.118.235.20|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 446966 (436K) [application/x-gzip] Saving to: “wget-1.5.3.tar.gz.1” 0K .......... .......... .......... .......... .......... 11% 48.1K 8s 50K .......... .......... .......... .......... .......... 22% 84.4K 5s 100K .......... .......... .......... .......... .......... 34% 25.4K 7s 150K .......... .......... .......... .......... .......... 45% 40.2K 6s 200K .......... .......... .......... .......... .......... 57% 81.0K 4s 250K .......... .......... .......... .......... .......... 68% 92.7K 3s 300K .......... .......... .......... .......... .......... 80% 85.1K 2s 350K .......... .......... .......... .......... .......... 91% 156K 1s 400K .......... .......... .......... ...... 100% 76.4K=7.4s 2017-11-20 19:00:12 (59.1 KB/s) - “wget-1.5.3.tar.gz.1” saved [446966/446966]
Example 11: Increase retry attempts while downloading files.
By default, the wget command retires a download 20 times to download a file successfully. We can increase this count by using the –tries option.
[root@linuxnix ~]# wget --tries=101 http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso --2017-11-20 19:05:18-- http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso Resolving centos.mirror.net.in... 103.240.105.131
In the above example, I increased the retry limit from the default 20 to 101.
Example 12:
In the final example for this article, we’ll show how you can use the wget command to download a webpage and save it to a file.
[root@linuxnix ~]# wget https://www.linuxnix.com/7-mpstat-command-examples-in-linux/ --2017-11-20 19:28:35-- https://www.linuxnix.com/7-mpstat-command-examples-in-linux/ Resolving www.linuxnix.com... 104.28.4.110, 104.28.5.110 Connecting to www.linuxnix.com|104.28.4.110|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: “index.html” [ <=> ] 100,658 151K/s in 0.6s 2017-11-20 19:28:38 (151 KB/s) - “index.html” saved [100658]
The above command saved the web page https://www.linuxnix.com/7-mpstat-command-examples-in-linux/ to a file named index.html.
Don’t be confused by the file name.
If you open the file you’ll find that it has the content of the actual web page and is by no means a conventional index.html file.
Conclusion:
In this article, we explored different methods to retrieve content using the wget command.
We hope that you’ve found this information useful and we encourage you to try out these features on your own as well.
Sahil Suri
Latest posts by Sahil Suri (see all)
- Google Cloud basics: Activate Cloud Shell - May 19, 2021
- Create persistent swap partition on Azure Linux VM - May 18, 2021
- DNF, YUM and RPM package manager comparison - May 17, 2021
- Introduction to the aptitude package manager for Ubuntu - March 26, 2021
- zypper package management tool examples for managing packages on SUSE Linux - March 26, 2021
