Curl is a wonderful tool available by default in UNIX based systems. It is a command line utility and a library. It is very useful while troubleshooting URL accesses and for downloading files. Curl supports a wide variety of protocols including HTTP, HTTPS, FTP, FTPS, SFTP etc. If you haven't specified any protocols explicitly it will default to HTTP. Curl is powered by libcurl for all transfer-related features.
Curl offers a lot of useful tricks such as proxy support, user authentication, FTP upload, HTTP post, SSL connections, cookies, file transfer resume, Metalink, and more. In this tutorial, we will discuss how to use curl command and download files using curl options on Linux.
The curl package is pre-installed on most Linux distributions today. The -V or --version options will return the version, and also the supported protocols and features in your current version.
$ curl -V curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3 Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smtp smtps telnet tftp Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP
Install Curl on Ubuntu/Debian
$ sudo apt install curl
Install Curl on CentOS/Fedora
$ sudo yum install curl
How to use curl command
The curl command allows you to download as well as upload data using CLI (the command line) in Linux. This is its syntax:
$ curl [options] [URL]
- options - The Curl options starting with one or two dashes.
- URL - URL of the remote server.
Simplest form that curl can be used is without any option and it will display the resource specified in the [url] to the standard output.
In the following example we will be retrieving the one of the linoxide.com blog pages:
The command above will print the source code of the linoxide.com blog page we specify in our terminal window.
If you don’t specify the protocol, curl will try to guess the protocol you want to use, and it will default to HTTP.
$ curl https://linoxide.com/how-to-install-terraform-on-centos-ubuntu/
You can pass the URL as input to the curl command, and redirect the output to a file.
For example:
$ curl https://linoxide.com/how-to-install-terraform-on-centos-ubuntu/ > install_terraform.html
The URL syntax is protocol dependent and multiple URLs can be written as sets like:
$ curl https://website.{one, two, three}.com
Download a file with same name
if you want, you can force curl to use the same name of the file being downloaded as the local file name. This can be done using the -O command line option.
$ curl -O http://releases.ubuntu.com/18.04/ubuntu-18.04.2-desktop-amd64.iso $ ls ubuntu-18.04.2-desktop-amd64.iso $
Download multiple files
To download multiple files at once you can use multiple -O flags followed by the URL of the files you want to download.
$ curl -O [URL1] -O [URL2] -O [URL3] ...
URLs with numeric sequence series can be written as:
$ curl ftp://ftp.example.com/file[1-30].jpeg
$ curl -O localhost/file.pdf -O localhost/file1.pdf
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 20597 100 20597 0 0 1436k 0 --:--:-- --:--:-- --:--:-- 1436k
100 20597 100 20597 0 0 9.8M 0 --:--:-- --:--:-- --:--:-- 9.8M
If you want to save multiple files using different name, you can do it with nested -o option as follows.
$ curl -o file2.pdf localhost/file.pdf -o file3.pdf localhost/file1.pdf
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 20597 100 20597 0 0 957k 0 --:--:-- --:--:-- --:--:-- 957k
100 20597 100 20597 0 0 19.6M 0 --:--:-- --:--:-- --:--:-- 19.6M
Display progress bar to curl Output
Curl displays a progress meter during the file download to indicate the transfer rate, amount of data transferred, time left, etc...
$ curl -# -O http://releases.ubuntu.com/18.04/ubuntu-18.04.2-desktop-amd64.iso ############### 20.9%
If you like a progress bar instead of meter, you can use the -# option as in the example above, or –silent if you want to disable it completely.
$ curl -O --silent http://releases.ubuntu.com/18.04/ubuntu-18.04.2-desktop-amd64.iso
If you want to see the time taken for the file download, you can add time command as follows.
$ time curl --output file1.pdf localhost/pdf-test.pdf -# ################################################################################################################################################## 100.0% real 0m0.060s user 0m0.016s sys 0m0.047s
Curl to Get HTTP Headers of a URL
To fetch the HTTP headers of the specified website, use the -I flag:
$ curl -I www.linoxide.com HTTP/1.1 301 Moved Permanently Date: Sat, 13 Apr 2019 16:39:03 GMT Connection: keep-alive Cache-Control: max-age=3600 Expires: Sat, 13 Apr 2019 17:39:03 GMT Location: https://www.linoxide.com/ Server: cloudflare CF-RAY: 4c6edfb8a8437c78-BEG
Sometimes you may come across an URL that gets errors like "Moved" or "Moved Permanently". This usually happens when the URL redirects to some other URL. For example, google.com redirects to an www URL, so you get an error like the following:
$ curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"> <TITLE>301 Moved</TITLE></HEAD><BODY> <H1>301 Moved</H1> The document has moved <A HREF="http://www.google.com/">here</A>. </BODY></HTML>
To tell curl to follow the redirect, use the -L command line option instead.
$ curl -L google.com
If the URL from where you are trying to download the files has any 301 redirect, you may not get the file downloaded. For example if the URL has http to https redirection, you have to use the option -L to follow the redirect and download.
$ curl -o test.txt -L http://www.test.com
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0
100 364k 0 364k 0 0 18213 0 --:--:-- 0:00:20 --:--:-- 100k
How to limit the maximum transfer rate
Usage for limiting the data transfer rate is:
$ curl --limit-rate [value] [URL]
The value can be expressed in bytes, kilobytes with the k suffix, megabytes with the m suffix and gigabytes with the g suffix.
The following command will limit the download speed to 1mb:
$ curl --limit-rate 1m -O http://releases.ubuntu.com/18.04/ubuntu-18.04.2-desktop-amd64.iso
Download files from, or upload files to a FTP Server with or without authentication
To access a protected FTP server with the curl command you need to use the -u flag with a username and password.
The following command will list all of the files and directories in the user’s home directory.
$ curl -u USERNAME:PASSWORD ftp://ftp.linoxide.com/
To download the file use:
$ curl -u USERNAME:PASSWORD ftp://ftp.linoxide.com/file.tar.gz
To upload the file to FTP server use the -T flag:
$ curl -T newfile.tar.gz -u USERNAME:PASSWORD ftp://ftp.linoxide.com/
NOTE: If the FTP server allows anonymous logins, you don't need to use -u username:password
Resume an interrupted download
If a download was interrupted for some reason, you can resume it using the -C to resume the download beginning where it left off.
$ curl -C - -O http://releases.ubuntu.com/18.04/ubuntu-18.04.2-desktop-amd64.iso
$ curl -C - -O http://releases.ubuntu.com/18.04/ubuntu-18.04.2-desktop-amd64.iso
** Resuming transfer from byte position 2297856
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 1901M 0 9842k 0 0 734k 0 0:44:10 0:00:13 0:43:57 339k
Managing SSL certs in Curl
If the URL that you use in curl give SSL error, you can skip the SSL using -k option.
$ curl -o -k https://www.test.com
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 364k 0 364k 0 0 34147 0 --:--:-- 0:00:10 --:--:-- 101k
If you want to verify your self signed cert in curl, you can use --cacert option.
curl --cacert myssl.crt -O https://test.com/file.tgz
Spoofing Useragent in Curl
If the URL that you use block curl due to lack of proper useragent, you can spoof the useragent using the option "-A" in curl to mimic the browser experience.
$ curl -A "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5" -O http://localhost/file1.pdf
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 20597 100 20597 0 0 874k 0 --:--:-- --:--:-- --:--:-- 874k
HTTP POST URLs with Curl
You can send data with POST in curl using the -X option.
$ curl -d "param1=value1¶m2=value2" -H "Content-Type: application/x-www-form-urlencoded" -X POST http://localhost/data
Conclusion
In this tutorial, we learned how to use curl command and download files in linux. If you want to learn more about curl visit the Curl Documentation page. Please let me know your thoughts and suggestions on this tutorial on the below comment section.
