How to Use Curl Command with Examples [Download Files]

How to use curl command

The curl command transfers data to or from a network server. It is by default available in all Linux-based systems. It is commonly used to troubleshoot URL accesses and for downloading files. Curl supports a wide variety of protocols including HTTP, HTTPS, FTP, FTPS, SFTP etc. If you haven't specified any protocols explicitly it will default to HTTP. Curl is powered by libcurl for all transfer-related features.

Curl offers a lot of useful tricks such as proxy support, user authentication, FTP upload, HTTP post, SSL connections, cookies, file transfer resume, Metalink, and more.

In this tutorial, we will discuss how to use curl command and its basic options with examples.

If you don't find curl installed in your Linux systems, use the following commands:

To install curl on Ubuntu/Debian

$ sudo apt install curl

To install Curl on CentOS/Fedora

$ sudo yum install curl

How to use curl command

Curl syntax:

$ curl [options] [URL]
  • options - The curl options starting with one or two dashes.
  • URL - URL of the remote server.

Curl command followed by website URL display the source code website to the standard output.

To print the source code of http://example.com:

$ curl http://example.com

You can pass the URL as input to the curl command, and redirect the output to a file.

For example:

$ curl http://exmaple.com > example.html

Download a file with same name

To download a file and save with the same name, use -O option.

For example to download Ubuntu 20.04 iso file, type:

$ curl -O http://releases.ubuntu.com/20.04/ubuntu-20.04.2.0-desktop-amd64.iso

Output:

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                  Dload  Upload   Total   Spent    Left  Speed
   5 2743M    5  156M    0     0  49.9M      0  0:00:54  0:00:03  0:00:51 49.9M

Verify the downloaded file:

$ ls
ubuntu-18.04.2-desktop-amd64.iso
$

To download a file and save it with a custom name, use -o option.

$ curl -o ubuntu-20-04.iso http://releases.ubuntu.com/20.04/ubuntu-20.04.2.0-desktop-amd64.iso

To verify, use ls command:

$ ls
ubuntu-20-04.iso

Download multiple files

To download multiple files at once, use multiple -O flags followed by the URL of the files.

Syntax:

$ curl -O [URL1] -O [URL2] -O [URL3] ...

For example:

 $ curl -O localhost/file.pdf -O localhost/file1.pdf

To save multiple files using different names, use nested -o option.

For example:

$ curl -o file2.pdf localhost/file.pdf -o file3.pdf localhost/file1.pdf

Download files which has numeric sequence, type:

$ curl ftp://ftp.example.com/file[1-30].jpeg

Download/Upload via FTP

To access a protected FTP server with the curl command you need to use the -u flag with a username and password.

The following command will list all of the files and directories in the user’s home directory.

$ curl -u USERNAME:PASSWORD ftp://ftp.example.com/

To download the file use:

$ curl -u USERNAME:PASSWORD ftp://ftp.example.com/file.tar.gz

To upload the file to FTP server use the -T flag:

$ curl -T newfile.tar.gz -u USERNAME:PASSWORD ftp://ftp.linoxide.com/

NOTE: If the FTP server allows anonymous logins, you don't need to use -u username:password

Upload file

Curl emulate a filled-in form where a user has pressed the submit button, using -F option.

For example to POST data using the Content-Type multipart/form-data:

$ curl -F @field_name=@path/to/local_file <upload_URL>

Resume an Interrupted Download

If a download was interrupted for some reason, you can resume it using the option -C - to resume the download beginning where it left off.

$ curl -C - -O http://releases.ubuntu.com/20.04/ubuntu-20.04.2.0-desktop-amd64.iso

Output:

** Resuming transfer from byte position 2297856
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 1901M    0 9842k    0     0   734k      0  0:44:10  0:00:13  0:43:57  339k

Curl to Get HTTP Headers

To fetch the HTTP header only use the -I option. When used on an FTP or FILE file, displays the file size and last modification time only.

For example:

$ curl -I redhat.com
 HTTP/1.0 301 Moved Permanently
 Location: https://www.redhat.com/en
 Server: BigIP
 Connection: Keep-Alive
 Content-Length: 0

Curl to Follow Redirect

Sometimes you may come across an URL that gets errors like "Moved" or "Moved Permanently". This usually happens when the URL redirects to some other URL.

For example, from the output google.com redirects to an www URL, so you get an error like the following:

$ curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

To tell curl to follow the redirect, use the -L option.

$ curl -L google.com

Use case 1: If the URL from where you are trying to download the files has any 301 redirect, you may not get the file downloaded. For example, if the URL has http to https redirection, you have to use the option -L to follow the redirect and download.

 $ curl -o test.txt -L http://www.test.com

Use case 2: To check a remote server support http2 use -L followed by --http2. For example

$ curl -I --http2 -s https://linoxide.com/ | grep HTTP

If the output shows HTTP/2 200, indicates that remote server supports HTTP/2.

Limit the Maximum Transfer Rate

Usage for limiting the data transfer rate is:

$ curl --limit-rate [value] [URL]

The value can be expressed in bytes, kilobytes with the k suffix, megabytes with the m suffix and gigabytes with the g suffix.

The following command will limit the download speed to 1mb:

$ curl --limit-rate 1m -O http://releases.ubuntu.com/20.04/ubuntu-20.04.2.0-desktop-amd64.iso

Skip SSL certs

By default, curl verify every SSL connection it makes. Sometimes you face issues downloading files due to SSL/TLS error or test API endpoint using a self-signed certificate or an invalid one. In such cases skip the SSL using -k option.

$ curl -o -k https://www.test.com

Verify Self Signed Certificate

To verify your self signed cert in curl, use --cacert option.

curl --cacert myssl.crt -O https://test.com/file.tgz

Specify User-agent string

To specify user-agent string to send to HTTP server use -A option.

Use case: Some http server block curl user-agent to download files, in such situations curl allows to specify user-agent. For example:

$ curl -A "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5" -O http://localhost/file1.pdf

Specify Proxy

To use the specified HTTP proxy to download use -x option. If the port number is not specified, it is assumed at port 1080.

For example to download file using specific proxy 192.168.35.8 on port 8080, type:

$ curl -x 192.168.35.8:8080 -O http://example.com/file01.tar.gz

Use -U or --proxy-user option to specific the authentication username and password.

For example:

$ curl -x 192.168.35.8:8080 -U user:password -O http://example.com/file01.tar.gz

To Store/Pass Cookies

To write cookies to a file that are downloaded when you access a resource, use --cookie-jar or -c option followed by text file name to store it.

For example to store cookies from http://www.nytimes.com/ to file named cookies.txt, use:

$ curl -c cookies.txt https://www.nytimes.com/index.html -O

You can tell curl reuse cookies received before by using -b or --cookie option followed by filename containing the cookies or a string. This will pass the data to the HTTP server as a cookie. By default, curl doesn't use cookies to request resources.

$ curl -b cookies.txt https://www.nytimes.com

Total time for a request

To get the total time curl took for a request to be successful use -w option.

For example to print total time after transfer complete to download ubuntu 20.04 iso file after transfer complete.

$ curl -w "%{time_total}\n" -o /dev/null -s http://releases.ubuntu.com/20.04/ubuntu-20.04.2.0-desktop-amd64.iso

Output:

276.044317

Local Name resolution

To test local version of a resource for example API endpoint use --resolve option. This developers to test an API for deploying.

$ curl --resolve www.example.com:80:localhost http://www.example.com/

Progress meter

Curl by default shows a progress meter when uploading or downloading data. It will indicate information such as total size, how much % and size received, time spent, time left, current speed, etc.

To show an alternative progress meter use -# or --progress-bar. For example:

$ curl -# -O http://releases.ubuntu.com/20.04/ubuntu-20.04.2.0-desktop-amd64.iso
###############                                                   20.9%

To disable the progress meter, use --silent or -s.

$ curl -O --silent http://releases.ubuntu.com/20.04/ubuntu-20.04.2.0-desktop-amd64.iso

Conclusion

In this tutorial, we learned how to use curl command to transfer data in Linux. If you want to learn more about curl visit the Curl Documentation page. Please let me know your thoughts and suggestions on this tutorial on the below comment section.

Read Also:

Leave a Comment