The curl command transfers data to or from a network server. It is by default available in all Linux-based systems. It is commonly used to troubleshoot URL accesses and for downloading files. Curl supports a wide variety of protocols including HTTP, HTTPS, FTP, FTPS, SFTP etc. If you haven't specified any protocols explicitly it will default to HTTP. Curl is powered by libcurl for all transfer-related features.
Curl offers a lot of useful tricks such as proxy support, user authentication, FTP upload, HTTP post, SSL connections, cookies, file transfer resume, Metalink, and more.
In this tutorial, we will discuss how to use curl command and its basic options with examples.
If you don't find curl installed in your Linux systems, use the following commands:
To install curl on Ubuntu/Debian
$ sudo apt install curlTo install Curl on CentOS/Fedora
$ sudo yum install curlHow to use curl command
Curl syntax:
$ curl [options] [URL]- options - The curl options starting with one or two dashes.
- URL - URL of the remote server.
Curl command followed by website URL display the source code website to the standard output.
To print the source code of http://example.com:
$ curl http://example.comYou can pass the URL as input to the curl command, and redirect the output to a file.
For example:
$ curl http://exmaple.com > example.htmlDownload a file with same name
To download a file and save with the same name, use -O option.
For example to download Ubuntu 20.04 iso file, type:
$ curl -O http://releases.ubuntu.com/20.04/ubuntu-20.04.2.0-desktop-amd64.isoOutput:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
5 2743M 5 156M 0 0 49.9M 0 0:00:54 0:00:03 0:00:51 49.9MVerify the downloaded file:
$ ls
ubuntu-18.04.2-desktop-amd64.iso
$To download a file and save it with a custom name, use -o option.
$ curl -o ubuntu-20-04.iso http://releases.ubuntu.com/20.04/ubuntu-20.04.2.0-desktop-amd64.isoTo verify, use ls command:
$ ls ubuntu-20-04.iso
Download multiple files
To download multiple files at once, use multiple -O flags followed by the URL of the files.
Syntax:
$ curl -O [URL1] -O [URL2] -O [URL3] ...For example:
$ curl -O localhost/file.pdf -O localhost/file1.pdfTo save multiple files using different names, use nested -o option.
For example:
$ curl -o file2.pdf localhost/file.pdf -o file3.pdf localhost/file1.pdfDownload files which has numeric sequence, type:
$ curl ftp://ftp.example.com/file[1-30].jpegDownload/Upload via FTP
To access a protected FTP server with the curl command you need to use the -u flag with a username and password.
The following command will list all of the files and directories in the user’s home directory.
$ curl -u USERNAME:PASSWORD ftp://ftp.example.com/To download the file use:
$ curl -u USERNAME:PASSWORD ftp://ftp.example.com/file.tar.gzTo upload the file to FTP server use the -T flag:
$ curl -T newfile.tar.gz -u USERNAME:PASSWORD ftp://ftp.linoxide.com/NOTE: If the FTP server allows anonymous logins, you don't need to use -u username:password
Upload file
Curl emulate a filled-in form where a user has pressed the submit button, using -F option.
For example to POST data using the Content-Type multipart/form-data:
$ curl -F @field_name=@path/to/local_file <upload_URL>Resume an Interrupted Download
If a download was interrupted for some reason, you can resume it using the option -C - to resume the download beginning where it left off.
$ curl -C - -O http://releases.ubuntu.com/20.04/ubuntu-20.04.2.0-desktop-amd64.isoOutput:
** Resuming transfer from byte position 2297856
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 1901M 0 9842k 0 0 734k 0 0:44:10 0:00:13 0:43:57 339kCurl to Get HTTP Headers
To fetch the HTTP header only use the -I option. When used on an FTP or FILE file, displays the file size and last modification time only.
For example:
$ curl -I redhat.com
HTTP/1.0 301 Moved Permanently
Location: https://www.redhat.com/en
Server: BigIP
Connection: Keep-Alive
Content-Length: 0Curl to Follow Redirect
Sometimes you may come across an URL that gets errors like "Moved" or "Moved Permanently". This usually happens when the URL redirects to some other URL.
For example, from the output google.com redirects to an www URL, so you get an error like the following:
$ curl google.com<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>To tell curl to follow the redirect, use the -L option.
$ curl -L google.comUse case 1: If the URL from where you are trying to download the files has any 301 redirect, you may not get the file downloaded. For example, if the URL has http to https redirection, you have to use the option -L to follow the redirect and download.
$ curl -o test.txt -L http://www.test.comUse case 2: To check a remote server support http2 use -L followed by --http2. For example
$ curl -I --http2 -s https://linoxide.com/ | grep HTTPIf the output shows HTTP/2 200, indicates that remote server supports HTTP/2.
Limit the Maximum Transfer Rate
Usage for limiting the data transfer rate is:
$ curl --limit-rate [value] [URL]The value can be expressed in bytes, kilobytes with the k suffix, megabytes with the m suffix and gigabytes with the g suffix.
The following command will limit the download speed to 1mb:
$ curl --limit-rate 1m -O http://releases.ubuntu.com/20.04/ubuntu-20.04.2.0-desktop-amd64.isoSkip SSL certs
By default, curl verify every SSL connection it makes. Sometimes you face issues downloading files due to SSL/TLS error or test API endpoint using a self-signed certificate or an invalid one. In such cases skip the SSL using -k option.
$ curl -o -k https://www.test.comVerify Self Signed Certificate
To verify your self signed cert in curl, use --cacert option.
curl --cacert myssl.crt -O https://test.com/file.tgzSpecify User-agent string
To specify user-agent string to send to HTTP server use -A option.
Use case: Some http server block curl user-agent to download files, in such situations curl allows to specify user-agent. For example:
$ curl -A "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5" -O http://localhost/file1.pdfSpecify Proxy
To use the specified HTTP proxy to download use -x option. If the port number is not specified, it is assumed at port 1080.
For example to download file using specific proxy 192.168.35.8 on port 8080, type:
$ curl -x 192.168.35.8:8080 -O http://example.com/file01.tar.gzUse -U or --proxy-user option to specific the authentication username and password.
For example:
$ curl -x 192.168.35.8:8080 -U user:password -O http://example.com/file01.tar.gzTo Store/Pass Cookies
To write cookies to a file that are downloaded when you access a resource, use --cookie-jar or -c option followed by text file name to store it.
For example to store cookies from http://www.nytimes.com/ to file named cookies.txt, use:
$ curl -c cookies.txt https://www.nytimes.com/index.html -OYou can tell curl reuse cookies received before by using -b or --cookie option followed by filename containing the cookies or a string. This will pass the data to the HTTP server as a cookie. By default, curl doesn't use cookies to request resources.
$ curl -b cookies.txt https://www.nytimes.com
Total time for a request
To get the total time curl took for a request to be successful use -w option.
For example to print total time after transfer complete to download ubuntu 20.04 iso file after transfer complete.
$ curl -w "%{time_total}\n" -o /dev/null -s http://releases.ubuntu.com/20.04/ubuntu-20.04.2.0-desktop-amd64.isoOutput:
276.044317Local Name resolution
To test local version of a resource for example API endpoint use --resolve option. This developers to test an API for deploying.
$ curl --resolve www.example.com:80:localhost http://www.example.com/Progress meter
Curl by default shows a progress meter when uploading or downloading data. It will indicate information such as total size, how much % and size received, time spent, time left, current speed, etc.
To show an alternative progress meter use -# or --progress-bar. For example:
$ curl -# -O http://releases.ubuntu.com/20.04/ubuntu-20.04.2.0-desktop-amd64.iso
############### 20.9%To disable the progress meter, use --silent or -s.
$ curl -O --silent http://releases.ubuntu.com/20.04/ubuntu-20.04.2.0-desktop-amd64.isoConclusion
In this tutorial, we learned how to use curl command to transfer data in Linux. If you want to learn more about curl visit the Curl Documentation page. Please let me know your thoughts and suggestions on this tutorial on the below comment section.
