What is curl command?
The curl command is a powerful command line tool used to transfer data to or from a server.
It can use any one of the supported protocols (HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP or FILE) for data transfer. In many aspects curl is similar to the wget command but curl is more feature rich as compared to wget. The cURL stands for “See URL“.
The curl command is generally available by default on UNIX systems.
Let’s verify that we have curl installed.
[root@linuxnix ~]# which curl /usr/bin/curl [root@linuxnix ~]# rpm -qf /usr/bin/curl curl-7.19.7-52.el6.x86_64
In case your system does not have curl installed then you can install it manually.
If you are using a RedHat based system then type:
yum install curl
If you are using a Debian based system then type:
apt-get install curl
In this article, we’ll go through some examples to understand some of the features curl provides.
Example 1: Save a web page.
[root@linuxnix ~]# curl https://www.linuxnix.com/ >> linuxnix.html % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 206k 0 206k 0 0 29952 0 --:--:-- 0:00:07 --:--:-- 179k [root@linuxnix ~]#
The above example will save the content of the web page https://www.linuxnix.com/ to a file named linuxnix.html.
Example 2: Specify output file name in curl command
By using the -o option you may specify the name of the output file in the curl command itself instead of using output redirection as we did in the previous example.
[root@linuxnix ~]# curl -o sahil.html https://www.linuxnix.com/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 206k 0 206k 0 0 27231 0 --:--:-- 0:00:07 --:--:-- 133k [root@linuxnix ~]# ls -ltrh sahil.html -rw-r--r--. 1 root root 207K Nov 20 20:32 sahil.html
The above command saves the content of the web page to a file named sahil.html.
Example 3: Save multiple files with a single curl command
We can specify multiple URLs to save content from and in a separate file using a single curl command line statement as shown below:
[root@linuxnix ~]# curl -o sahil.html https://www.linuxnix.com/7-mpstat-command-examples-in-linux/ -o sahil2.html https://www.linuxnix.com/awk-scripting-print-header-line-along-with-pattern-match/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 94230 0 94230 0 0 41826 0 --:--:-- 0:00:02 --:--:-- 85353 100 79251 0 79251 0 0 84024 0 --:--:-- --:--:-- --:--:-- 487k [root@linuxnix ~]# ls -l *.html -rw-r--r--. 1 root root 79251 Nov 20 20:39 sahil2.html -rw-r--r--. 1 root root 94230 Nov 20 20:39 sahil.html
Example 4: Do not display progress bar
We can use the -s option with the curl command if we do not want it to display(silent) the default progress bar.
Here is an example.
[root@linuxnix ~]# curl -s -o sahil.html https://www.linuxnix.com/ [root@linuxnix ~]#
Example 5: Follow HTTP location headers
By default, the curl command does not follow location headers.
When a requested web page is moved, then an HTTP Location header will be sent as a response and it will mention the present location of the web page.
For example, when someone types google.com in the browser from India, it will be automatically redirected to ‘google.co.in’.
This is done based on the HTTP Location header as shown below.
[root@linuxnix ~]# curl google.com <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"> <TITLE>302 Moved</TITLE></HEAD><BODY> <H1>302 Moved</H1> The document has moved <A HREF="http://www.google.co.in/?gfe_rd=cr&dcr=0&ei=4KMSWtCLPPScX-rqvoAF">here</A>. </BODY></HTML> [root@linuxnix ~]#
We can instruct curl to follow the redirection by using -L option, as shown below.
[root@linuxnix ~]# curl -L google.com
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en-IN">
<head><meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
<meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop="image">
<title>Google</title><script>(function(){window.google={----output truncated for brevityNow curl will print the web page in HTML format.
Example 6: Fetch and display the HTTP-header only
To just verify that a website URL is accessible and to print it’s HTTP header, we use the -I option with the curl command.
[root@linuxnix ~]# curl -I https://www.linuxnix.com HTTP/1.1 200 OK Date: Mon, 20 Nov 2017 09:55:19 GMT Content-Type: text/html; charset=UTF-8 Connection: keep-alive Set-Cookie: __cfduid=dd037dc84d9c6b26fc56c8e37c1be10f91511171718; expires=Tue, 20-Nov-18 09:55:18 GMT; path=/; domain=.linuxnix.com; HttpOnly Strict-Transport-Security: max-age=15552000; includeSubDomains; preload X-Content-Type-Options: nosniff Server: cloudflare-nginx CF-RAY: 3c0a886a3b2270ce-SIN [root@linuxnix ~]#
Example 7: Use curl command in verbose mode
We can use the -v option with curl to perform a verbose mode while using it to display more detailed information.
[root@linuxnix ~]# curl -v https://www.linuxnix.com * About to connect() to www.linuxnix.com port 443 (#0) * Trying 104.28.4.110... connected * Connected to www.linuxnix.com (104.28.4.110) port 443 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * SSL connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 * Server certificate: * subject: CN=sni155736.cloudflaressl.com,OU=PositiveSSL Multi-Domain,OU=Domain Control Validated * start date: Nov 09 00:00:00 2017 GMT * expire date: May 18 23:59:59 2018 GMT * common name: sni155736.cloudflaressl.com * issuer: CN=COMODO ECC Domain Validation Secure Server CA 2,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB > GET / HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: www.linuxnix.com > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 20 Nov 2017 09:56:07 GMT < Content-Type: text/html; charset=UTF-8 < Transfer-Encoding: chunked < Connection: keep-alive -----------------------------------------------output truncated for brevity
The * symbols represent information printed by the curl tool and ‘<‘ and ‘>’ display the data sent and received respectively by Curl
Example 8: Save the file with the same name as that in the URL
To save a file with the same name as that mentioned in the URL, we use the -O option as shown below:
[root@linuxnix ~]# curl -O http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 22 436k 22 98906 0 0 10952 0 0:00:40 0:00:09 0:00:31 27704
Example 9: Resume interrupted downloads
I had canceled the ongoing download shown in the previous example at 22%. By using the -C option we can actually resume the download from where we left off.
[root@linuxnix ~]# curl -C - -O http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz ** Resuming transfer from byte position 151552 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 288k 100 288k 0 0 29356 0 0:00:10 0:00:10 --:--:-- 61218 [root@linuxnix ~]#
Note: You need to add an extra dash (-) in the command above. If you don’t curl seems to print the file on stdout which isn’t usable.
Example 10: Limit download speed
To limit the download speed while downloading a file we use the –limit-rate option.
[root@linuxnix ~]# curl --limit-rate 10K -O http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 18 436k 18 81146 0 0 9172 0 0:00:48 0:00:08 0:00:40 21949
In the above example, we’ve limited the download speed to 10Kb/s for the operation.
Example 11: Download files depending on the specified modified data
We can use the -z option with curl to download files only if they have been modified after or before a specified modified date.
[root@linuxnix ~]# curl -z 21-Nov-17 -O http://192.168.188.129/test.htm % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 9495 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 [root@linuxnix ~]# ls -l test.htm ls: cannot access test.htm: No such file or directory [root@linuxnix ~]# curl -z -21-Nov-17 -O http://192.168.188.129/test.htm % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 101 9495 101 9495 0 0 1773k 0 --:--:-- --:--:-- --:--:-- 9272k [root@linuxnix ~]# ls -l test.htm -rw-r--r--. 1 root root 9495 Nov 20 21:27 test.htm [root@linuxnix ~]#
If we specify a hyphen or dash before the date then curl checks if the file has been modified before the mentioned date and if there is no dash then curl downloads the file if it has been modified after the specified date.
Example 12: Download file from using FTP
To download a file via FTP or HTTP we use the -u option to provide the connecting username and password details separated by a colon to be used to authenticate with the URL.
Given below is an example:
[root@linuxnix ~]# curl -u sahil:$eCur1_p@$$w0rd -O ftp://192.168.188.129/test.txt % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 20 0 20 0 0 413 0 --:--:-- --:--:-- --:--:-- 20000 [root@linuxnix ~]#
In the above command, sahil is the username and $eCur1_p@$$w0rd is the password I’m using to authenticate with FTP server 192.168.188.129 and download the file test.txt
Example 13: Upload a file via FTP.
In order to upload a file to an FTP server, we use the -u option to specify the credentials and -T option to specify the file name to be uploaded.
[root@linuxnix ~]# curl -u sahil:$eCur1_p@$$w0rd -T test.htm ftp://192.168.188.129/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 101 9495 0 0 101 9495 0 188k --:--:-- --:--:-- --:--:-- 0 [root@linuxnix ~]#
Example 14: Upload multiple files
We can enclose multiple file names within parentheisis and seperated by commas to upload them in a single curl command.
[root@linuxnix ~]# curl -u sahil:$eCur1_p@$$w0rd -T "{test.htm,test.awk}" ftp://192.168.188.129/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
101 9495 0 0 101 9495 0 123k --:--:-- --:--:-- --:--:-- 0
109 109 0 0 109 109 0 2881 --:--:-- --:--:-- --:--:-- 2881Example 15: Upload file with content fed from STDIN
An interesting feature of curl is that while uploading files, you could create a file on the fly by supplying input directly from STDIN and the content would be available in the file at the destination.
Here is a demonstration:
[root@linuxnix ~]# curl -u sahil:$eCur1_p@$$w0rd -T - ftp://192.168.188.129/sahil.txt This is a test file % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 20 0 0 0 20 0 2 --:--:-- 0:00:06 --:--:-- 2To check upload from STDIN 0 47 0 0 0 47 0 2 --:--:-- 0:00:20 --:--:-- 2 [root@linuxnix ~]#
The above curl command will create a file named sahil.txt on the destination FTP server as soon as my credentials get authenticated.
After that, I can type content and press ctrl+d when done and this will add the content I wrote to the file on the destination.
Also, don’t mind the progress bar from curl interfering in the lines.
This would be there in the file updated at the destination location.
[root@linuxnix sahil]# cat sahil.txt This is a test file To check upload from STDIN [root@linuxnix sahil]#
Example 16: Upload/Download files by supplying password non-interactively.
Thus far we’ve successfully downloaded and uploaded files via FTP. But one problem with this approach is that we are typing passwords in plain text which isn’t a good thing.
So, in this example, we will modify our approach and will not pass the password in the curl command but enter it non-interactively instead.
[root@linuxnix ~]# curl -u sahil -T - ftp://192.168.188.129/sahil.txt Enter host password for user 'sahil': test file % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 9 0 0 0 9 0 1 --:--:-- 0:00:07 --:--:-- 1 0 10 0 0 0 10 0 0 --:--:-- 0:00:10 --:--:-- 0
As you can see from the above command I did not type in my password in the curl command.
I was prompted for it and when I typed in the correct password and authenticated successfully, things moved further.
Example 17: Download files via SFTP
To download or upload files via sftp we simply need to replace ftp with sftp in the server URL as shown below:
[root@linuxnix ~]# curl -u sahil -O sftp://192.168.188.129/home/sahil/sahil.txt Enter host password for user 'sahil': % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 47 0 47 0 0 113 0 --:--:-- --:--:-- --:--:-- 113
Example 18: Use key-based authentication to download a file.
In the given example we use public key based authentication to download a file using the as the desired SCP protocol.
[sahil@linuxnix ~]$ curl -k -u sahil: --pubkey ~/.ssh/id_rsa.pub -O scp://192.168.188.129/tmp/test.txt % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 33 0 33 0 0 87 0 --:--:-- --:--:-- --:--:-- 87 [sahil@linuxnix ~]$ ls -l test.txt -rw-rw-r--. 1 sahil sahil 33 Nov 20 22:17 test.txt
Conclusion:
This concludes our detailed exploration of the wide variety of features provided by the curl command.
Although we’ve covered a lot of features and options in this article we’ve not covered them all.
For example, you could use curl to access web content while being behind a proxy by using the -x option.
Also, you can use the curl command to send HTTP POST, PUT, DELETE requests in HTML.
We encourage you to explore these features and as always we welcome your suggestions and feedback.
Sahil Suri
Latest posts by Sahil Suri (see all)
- Google Cloud basics: Activate Cloud Shell - May 19, 2021
- Create persistent swap partition on Azure Linux VM - May 18, 2021
- DNF, YUM and RPM package manager comparison - May 17, 2021
- Introduction to the aptitude package manager for Ubuntu - March 26, 2021
- zypper package management tool examples for managing packages on SUSE Linux - March 26, 2021
