How to Use Curl Command with Examples [Download Files]

How to use curl command

Curl is a wonderful tool available by default in UNIX based systems. It is a command line utility and a library. It is very useful while troubleshooting URL accesses and for downloading files. Curl supports a wide variety of protocols including HTTP, HTTPS, FTP, FTPS, SFTP etc. If you haven't specified any protocols explicitly it will default to HTTP. Curl is powered by libcurl for all transfer-related features.

Curl offers a lot of useful tricks such as proxy support, user authentication, FTP upload, HTTP post, SSL connections, cookies, file transfer resume, Metalink, and more. In this tutorial, we will discuss how to use curl command and download files using curl options on Linux.

The curl package is pre-installed on most Linux distributions today. The -V or --version options will return the version, and also the supported protocols and features in your current version.

$ curl -V
curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smtp smtps telnet tftp 
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP 

Install Curl on Ubuntu/Debian

$ sudo apt install curl

Install Curl on CentOS/Fedora

$ sudo yum install curl

How to use curl command

The curl command allows you to download as well as upload data using CLI (the command line) in Linux. This is its syntax:

$ curl [options] [URL]
  • options - The Curl options starting with one or two dashes.
  • URL - URL of the remote server.

Simplest form that curl can be used is without any option and it will display the resource specified in the [url] to the standard output.

In the following example we will be retrieving the one of the linoxide.com blog pages:

The command above will print the source code of the linoxide.com blog page we specify in our terminal window.

If you don’t specify the protocol, curl will try to guess the protocol you want to use, and it will default to HTTP.

$ curl https://linoxide.com/linux-how-to/how-to-install-terraform-on-centos-ubuntu/

You can pass the URL as input to the curl command, and redirect the output to a file.

For example:

$ curl https://linoxide.com/linux-how-to/how-to-install-terraform-on-centos-ubuntu/ > install_terraform.html

The URL syntax is protocol dependent and multiple URLs can be written as sets like:

$ curl https://website.{one, two, three}.com

Download a file with same name

if you want, you can force curl to use the same name of the file being downloaded as the local file name. This can be done using the -O command line option.

$ curl -O http://releases.ubuntu.com/18.04/ubuntu-18.04.2-desktop-amd64.iso
$ ls
ubuntu-18.04.2-desktop-amd64.iso
$

Download multiple files

To download multiple files at once you can use multiple -O flags followed by the URL of the files you want to download.

$ curl -O [URL1] -O [URL2] -O [URL3] ...

URLs with numeric sequence series can be written as:

$ curl ftp://ftp.example.com/file[1-30].jpeg
 $ curl -O localhost/file.pdf -O localhost/file1.pdf
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20597  100 20597    0     0  1436k      0 --:--:-- --:--:-- --:--:-- 1436k
100 20597  100 20597    0     0   9.8M      0 --:--:-- --:--:-- --:--:--  9.8M

If you want to save multiple files using different name, you can do it with nested -o option as follows.

 $ curl -o file2.pdf localhost/file.pdf -o file3.pdf localhost/file1.pdf
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20597  100 20597    0     0   957k      0 --:--:-- --:--:-- --:--:--  957k
100 20597  100 20597    0     0  19.6M      0 --:--:-- --:--:-- --:--:-- 19.6M

Display progress bar to curl Output

Curl displays a progress meter during the file download to indicate the transfer rate, amount of data transferred, time left, etc...

$ curl -# -O http://releases.ubuntu.com/18.04/ubuntu-18.04.2-desktop-amd64.iso
###############                                                           20.9%

If you like a progress bar instead of meter, you can use the -# option as in the example above, or –silent if you want to disable it completely.

$ curl -O --silent http://releases.ubuntu.com/18.04/ubuntu-18.04.2-desktop-amd64.iso

If you want to see the time taken for the file download, you can add time command as follows.

$ time curl --output file1.pdf localhost/pdf-test.pdf -#
################################################################################################################################################## 100.0%
 
real    0m0.060s 
user    0m0.016s 
sys     0m0.047s

Curl to Get HTTP Headers of a URL

To fetch the HTTP headers of the specified website, use the -I flag:

$ curl -I www.linoxide.com
HTTP/1.1 301 Moved Permanently
Date: Sat, 13 Apr 2019 16:39:03 GMT
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Sat, 13 Apr 2019 17:39:03 GMT
Location: https://www.linoxide.com/
Server: cloudflare
CF-RAY: 4c6edfb8a8437c78-BEG

Sometimes you may come across an URL that gets errors like "Moved" or "Moved Permanently". This usually happens when the URL redirects to some other URL. For example, google.com redirects to an www URL, so you get an error like the following:

$ curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

To tell curl to follow the redirect, use the -L command line option instead.

$ curl -L google.com

If the URL from where you are trying to download the files has any 301 redirect, you may not get the file downloaded. For example if the URL has http to https redirection, you have to use the option -L to follow the redirect and download.

 $ curl -o test.txt -L http://www.test.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:09 --:--:--     0
100  364k    0  364k    0     0  18213      0 --:--:--  0:00:20 --:--:--  100k

How to limit the maximum transfer rate

Usage for limiting the data transfer rate is:

$ curl --limit-rate [value] [URL]

The value can be expressed in bytes, kilobytes with the k suffix, megabytes with the m suffix and gigabytes with the g suffix.

The following command will limit the download speed to 1mb:

$ curl --limit-rate 1m -O http://releases.ubuntu.com/18.04/ubuntu-18.04.2-desktop-amd64.iso

Download files from, or upload files to a FTP Server with or without authentication

To access a protected FTP server with the curl command you need to use the -u flag with a username and password.
The following command will list all of the files and directories in the user’s home directory.

$ curl -u USERNAME:PASSWORD ftp://ftp.linoxide.com/

To download the file use:

$ curl -u USERNAME:PASSWORD ftp://ftp.linoxide.com/file.tar.gz

To upload the file to FTP server use the -T flag:

$ curl -T newfile.tar.gz -u USERNAME:PASSWORD ftp://ftp.linoxide.com/

NOTE: If the FTP server allows anonymous logins, you don't need to use -u username:password

Resume an interrupted download

If a download was interrupted for some reason, you can resume it using the -C to resume the download beginning where it left off.

$ curl -C - -O http://releases.ubuntu.com/18.04/ubuntu-18.04.2-desktop-amd64.iso
$ curl -C - -O http://releases.ubuntu.com/18.04/ubuntu-18.04.2-desktop-amd64.iso
** Resuming transfer from byte position 2297856
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 1901M    0 9842k    0     0   734k      0  0:44:10  0:00:13  0:43:57  339k

Managing SSL certs in Curl

If the URL that you use in curl give SSL error, you can skip the SSL using -k option.

 $ curl -o -k https://www.test.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  364k    0  364k    0     0  34147      0 --:--:--  0:00:10 --:--:--  101k

If you want to verify your self signed cert in curl, you can use --cacert option.

curl --cacert myssl.crt -O https://test.com/file.tgz

Spoofing Useragent in Curl

If the URL that you use block curl due to lack of proper useragent, you can spoof the useragent using the option "-A" in curl to mimic the browser experience.

 $ curl -A "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5" -O http://localhost/file1.pdf
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20597  100 20597    0     0   874k      0 --:--:-- --:--:-- --:--:--  874k

HTTP POST URLs with Curl

You can send data with POST in curl using the -X option.

 $ curl -d "param1=value1&param2=value2" -H "Content-Type: application/x-www-form-urlencoded" -X POST http://localhost/data

Conclusion

In this tutorial, we learned how to use curl command and download files in linux. If you want to learn more about curl visit the Curl Documentation page. Please let me know your thoughts and suggestions on this tutorial on the below comment section.

Read Also:

Slavisa Milojkovic 6:49 pm

Comments

Your email address will not be published. Required fields are marked *

All comments are subject to moderation.