Curl is a command-line tool that uploads or downloads data through the specified URL and displays the data. c in curl represents client, and URL is the URL. Here we introduce the use of curl.
1. Command line Basics
1.1 command line options
1. Short form
In curl, you can use short command-line options, such as telling curl to turn on verbose mode. You can use the - v option:
$ curl -v www.baidu.com
Here, - v is the short form option. We use a middle dash (-) followed by a letter to specify a short form option.
In this example, - v is like a switch that specifies whether a variable is false or true. We can use a dash followed by multiple single letter options:
$ curl -vL www.baidu.com
In curl, the command line parser always parses the entire command line, so options can be placed anywhere on the entire command line:
$ curl www.baidu.com -vL
This has the same effect as the above command. Of course, although it is in any position, it cannot be placed in front of curl:
$ -vL curl www.baidu.com // No command '-vL' found
2. Long form
Single letter options are easy to type and use, but the number of letters is limited and there are too many things to control. At this time, you can use the long form of options. Similarly, in order to make the command easy to read, most short forms have a corresponding long form.
Unlike the short form, the long form option is specified with two dashes (–), followed by the specific option. Also, when using long forms, - can only be followed by one option. For - v, the corresponding long form is as follows:
$ curl --verbose www.baidu.com
Similarly, long form options can appear anywhere in the command:
$ curl www.baidu.com --verbose
For - vL, the corresponding long form can be:
$ curl --verbose --location www.baidu.com
Or:
$ curl --location www.baidu.com --verbose
3. Parameters of options
In the above command, the options - v (or -- verbose) and - L (or -- location) are bool flag bits to tell curl to turn on or off some features. Curl also has a type of option, which requires passing some parameters. For example, if you want to pass a string to the server in an HTTP POST:
$ curl -d arbitrary http://example.com
Similarly, the corresponding long form can also be used:
$ curl --data arbitrary http://example.com
4. Is there a space in the parameter?
In the above example, our parameter array is a continuous string, but what if we need to pass a parameter with spaces? For example, Are you OK?, In this case, we need to use quotation marks to enclose the parameters:
$ curl -A "Are you OK?" http://example.com
Without quotation marks:
$ curl -A Are you OK? http://example.com
Then curl will only take Are as the user's parameter, and the remaining characters, you and OK? Will be handled by curl as an extra URL because it is not used here - specify that this is an option.
But what if the parameter itself has quotation marks? This is particularly common when passing parameters using JSON. We can use single quotation marks to enclose the parameters (but it doesn't work in Windows):
$ curl -d '{"name":"fool"}' http://example.com
When there is a lot of data, we can specify a file to pass to curl:
$ curl -d @params.json http://example.com
5. Say No
For flag options such as - t and - L, we can prefix the long form with no to specify to turn off the corresponding features, such as turning off verbose mode:
$ curl --no-verbose http://example.com
1.2 URL
curl supports processing multiple URLs in one command line, with spaces between them. curl will simply verify the incoming URL without verifying whether the URL is really valid. Therefore, the user needs to provide a valid URL here.
As mentioned earlier, curl first parses the entire command line and applies the resulting options to all URLs. If you want to use different options for each URL, you can use -- next to specify. For example:
$ curl --location http://example.com/1
--next --data sendthis http://example.com/2
--next head http://example.com/3
1. Configuration file
If there are too many options, making it difficult to enter commands, or exceeding the maximum length limit of system commands, we can use the configuration file to specify curl options.
Tell curl to read the options from the specified file by using the - K or -- config options, such as:
$ curl -K curl.options http://example.com
In the file curl Options, list all required options:
# ask curl to follow redirects --location # ask curl to do a HEAD request --head
As in the command line, you can also use the long form or short form in the configuration file, and even omit the two middle dashes (–) for the long form in the configuration file:
# ask curl to follow redirects location # ask curl to do a HEAD request head
For options that use parameters, you can also use the configuration file:
# ask curl to change the User-Agent in HTTP header user-agent "something-is-an-agent"
Since it is called a configuration file, the above options can also be written as follows:
# ask curl to change the User-Agent in HTTP header user-agent = "something-is-an-agent"
You can even omit quotation marks for parameters without spaces:
# ask curl to change the User-Agent in HTTP header user-agent = something-is-an-agent
Of course, you can't omit quotation marks if there are spaces in the parameter.
2. Start using curl
Earlier, we briefly introduced what curl is and some basic command-line knowledge. We hand over the URL to curl through the command line.
Here, we begin to use curl to understand what curl can do and how to do it.
2.1 Verbose mode
If the result obtained by curl is not the expected result, we can use - v or -- Verbose to enter Verbose mode for more information.
1. View the communication process
In Verbose mode, curl will get more conversational information to help us understand what happened. Curl will identify each message with * in front of it. In the following example, we save baidu's home page (use the - o option and specify the parameter baidu):
$ curl -v www.baidu.com -o baidu
We can get the following information:
* About to connect() to www.baidu.com port 80 (#0)
* Trying 14.215.177.39... connected
* Connected to www.baidu.com (14.215.177.39) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: www.baidu.com
> Accept: */*
>
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Cache-Control: private, no-cache, no-store, proxy-revalidate, no-transform
< Connection: Keep-Alive
< Content-Length: 2381
< Content-Type: text/html
< Date: Fri, 14 Sep 2018 09:55:18 GMT
< Etag: "588604dd-94d"
< Last-Modified: Mon, 23 Jan 2017 13:27:57 GMT
< Pragma: no-cache
< Server: bfe/1.0.8.18
< Set-Cookie: BDORZ=27315; max-age=86400; domain=.baidu.com; path=/
<
{ [data not shown]
103 2381 103 2381 0 0 113k 0 --:--:-- --:--:-- --:--:-- 232k* Connection #0 to host www.baidu.com left intact
* Closing connection #0
The following information is to establish a link:
* About to connect() to www.baidu.com port 80 (#0) * Trying 14.215.177.39... connected * Connected to www.baidu.com (14.215.177.39) port 80 (#0)
Then comes the HTTP request:
> GET / HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: www.baidu.com > Accept: */* >
Next is the data transmission process. Then there is the response:
< HTTP/1.1 200 OK < Accept-Ranges: bytes < Cache-Control: private, no-cache, no-store, proxy-revalidate, no-transform < Connection: Keep-Alive < Content-Length: 2381 < Content-Type: text/html < Date: Fri, 14 Sep 2018 09:55:18 GMT < Etag: "588604dd-94d" < Last-Modified: Mon, 23 Jan 2017 13:27:57 GMT < Pragma: no-cache < Server: bfe/1.0.8.18 < Set-Cookie: BDORZ=27315; max-age=86400; domain=.baidu.com; path=/ <
Even the arrows are different.
2. More detailed information
If you don't think the information of using - v is enough, you can also use the -- trace [filename] option to save the complete stream to filename. For example:
$ curl --trace dump www.baidu.com
After that, you can find a new file dump, which stores all the information of the previous session:
== Info: About to connect() to www.baidu.com port 80 (#0) == Info: Trying 14.215.177.39... == Info: connected == Info: Connected to www.baidu.com (14.215.177.39) port 80 (#0) => Send header, 166 bytes (0xa6) 0000: 47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d 0a GET / HTTP/1.1.. 0010: 55 73 65 72 2d 41 67 65 6e 74 3a 20 63 75 72 6c User-Agent: curl 0020: 2f 37 2e 31 39 2e 37 20 28 78 38 36 5f 36 34 2d /7.19.7 (x86_64- 0030: 72 65 64 68 61 74 2d 6c 69 6e 75 78 2d 67 6e 75 redhat-linux-gnu 0040: 29 20 6c 69 62 63 75 72 6c 2f 37 2e 31 39 2e 37 ) libcurl/7.19.7 0050: 20 4e 53 53 2f 33 2e 32 37 2e 31 20 7a 6c 69 62 NSS/3.27.1 zlib 0060: 2f 31 2e 32 2e 33 20 6c 69 62 69 64 6e 2f 31 2e /1.2.3 libidn/1. 0070: 31 38 20 6c 69 62 73 73 68 32 2f 31 2e 34 2e 32 18 libssh2/1.4.2 0080: 0d 0a 48 6f 73 74 3a 20 77 77 77 2e 62 61 69 64 ..Host: www.baid 0090: 75 2e 63 6f 6d 0d 0a 41 63 63 65 70 74 3a 20 2a u.com..Accept: * 00a0: 2f 2a 0d 0a 0d 0a /*.... <= Recv header, 17 bytes (0x11) 0000: 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK. 0010: 0a . <= Recv header, 22 bytes (0x16) 0000: 41 63 63 65 70 74 2d 52 61 6e 67 65 73 3a 20 62 Accept-Ranges: b 0010: 79 74 65 73 0d 0a ytes..
The first 21 lines of the file are shown above. Each sent and received data is saved in hexadecimal form for later analysis.
If hexadecimal is not helpful, you can use the -- trace ASCII [filename] option:
$ curl --trace-ascii dump www.baidu.com
The results are as follows:
== Info: About to connect() to www.baidu.com port 80 (#0) == Info: Trying 14.215.177.38... == Info: connected == Info: Connected to www.baidu.com (14.215.177.38) port 80 (#0) => Send header, 166 bytes (0xa6) 0000: GET / HTTP/1.1 0010: User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 0050: NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 0082: Host: www.baidu.com 0097: Accept: */* 00a4: <= Recv header, 17 bytes (0x11) 0000: HTTP/1.1 200 OK <= Recv header, 22 bytes (0x16) 0000: Accept-Ranges: bytes <= Recv header, 76 bytes (0x4c) 0000: Cache-Control: private, no-cache, no-store, proxy-revalidate, no 0040: -transform <= Recv header, 24 bytes (0x18) 0000: Connection: Keep-Alive <= Recv header, 22 bytes (0x16) 0000: Content-Length: 2381
The above is the output of the first 21 lines.
3. Silence
The opposite mode of verbose mode is silence. You can use the - s or -- silence option to tell curl not to output any program information or error information, but also the result of the response.
If you need to output an error message when there is an error, you can use - S or -- show error to specify.
2.2 browser to curl
After others use the browser to initiate a request, if they want to use curl to make the same request again, it is a common operation in daily work. Is there any quick and easy way to get curl command in curl?
Both Chrome browser and Firefox browser have realized the tool of copying into curl, which can quickly copy the browser's request into curl command, which is very convenient and fast.
1. Chrome
In Chrome, open more tools - > developer mode, select the Network tab, and then you can see all requests. Select the corresponding request, right-click the Copy as cURL option, and click.
2. Firefox
In Firefox, open the Web Developer - > network tool, and then right-click the link you want to copy. There is an option to Copy as cURL. Click it.
3. HTTP and curl
HTTP is the most used protocol with curl. Here we will introduce how to effectively use curl to send HTTP requests.
3.1 HTTP method
In each HTTP request, there is a corresponding method. The common methods are: GET, POST, HEAD and PUT.
If no specific method is specified in a curl command, the default is to use the GET method. For other methods, you can specify in curl command:
| method | option |
|---|---|
| POST | -d or - F |
| HEAD | -I |
| PUT | -T |
3.2 Header
In curl, use the - i option to display the Header information of the Response, together with the Body data:
$ curl -i www.baidu.com
result:
HTTP/1.1 200 OK Accept-Ranges: bytes Cache-Control: private, no-cache, no-store, proxy-revalidate, no-transform Connection: Keep-Alive Content-Length: 2381 Content-Type: text/html Date: Mon, 17 Sep 2018 10:26:42 GMT Etag: "588604dd-94d" Last-Modified: Mon, 23 Jan 2017 13:27:57 GMT Pragma: no-cache Server: bfe/1.0.8.18 Set-Cookie: BDORZ=27315; max-age=86400; domain=.baidu.com; path=/ <!DOCTYPE html> <!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=styleshee t type=text/css href=http://s1. bdstatic. com/r/www/cache/bdorz/baidu. Min.css > < title > Baidu, you can know</title></head> <body link=#0000cc> ... </body> </html>
Use the - I option to display only the Header information of the Response:
$ curl -I www.baidu.com
3.3 POST
POST is a method of submitting data to the server in HTTP. In the browser, but after filling in the data in the form, the browser will convert the filled data in the form of key=value string by default. In curl, we can use - d or -- data options to specify specific data:
$ curl -d key1=value1&key2=value2 http://example.com
We can also use multiple - d options to specify multiple groups of data, and curl will automatically connect these data. Therefore, the above example can also be like this:
$ curl -d key1=value1 -d key2=value2 http://example.com
Of course, if there is too much data, we can also put the data in one file:
$ curl -d @filename http://example.com
1. Content-Type
When POST method is used to submit data, there are four main forms of submitted data:
- application/x-www-form-urlencoded: the default form, that is, key1 = value1 & key2 = Value2;
- Multipart / form data: this form is used when uploading files using forms;
- application/json: submit data in JSON format;
- text/xml: submit data in XML format.
Content type is a Header. If it is not specified, the default is to transmit data in the form of application/x-www-form-urlencoded. If other forms of data transmission are required, this Header needs to be specified:
$ curl -d '{I Am A JSON FORM}' -H 'Content-Type: application/json' http://example.com
Where, - H is the option used to specify a specific Header, and the value is in the form of key=value. When you need to specify other headers, you can use the - H option.
2. POST a binary data
In curl, we can also submit a file. We can use the -- data binary option to specify a file:
$ curl --data-binary @filename http://example.com
3. Convert to a GET
Using the - G or - GET option, you can convert a POST request into a GET request. If there is a parameter specified by the - d option, curl will add the data after - d to the back of the URL? connect. For example:
$ curl -d "key1=value1" -G http://example.com
The obtained request URL is:
http://example.com/?key1=value1
4. URL code
If the data used is not encoded, you can specify curl to help you encode. In this case, you can use the -- data URLEncode option to specify. For example:
$ curl --data-urlencode "name=Alan Walker" http://example.com
5. multipart formposts
If an HTTP POST has the following form:
<form action="submit.cgi" method="post" enctype="multipart/form-data">
Name: <input type="text" name="person"><br>
File: <input type="file" name="secret"><br>
<input type="submit" value="Submit">
</form>
Users can fill in the Name in Name, select a File in File, and then click the Submit button to Submit data.
In order to simulate this request in curl, we can use the - F or -- form option to specify the data:
$ curl -F person=annonymous -F secret=@filename http://example.com/submit.cgi
In the above form, action specifies where the request is sent; method specifies that this is a POST request; enctype specifies that this is a multipart formpost.
After executing the curl command above, curl will generate the following request header:
POST /submit.cgi HTTP/1.1 Host: example.com User-Agent: curl/7.46.0 Accept: */* Content-Length: 313 Content-Type: multipart/form-data; boundary=------------------------d74496d66958873e
Content type is consistent with enctype.
When the - F option is used, the default content type is multipart / form data. However, we can also use - H to specify:
$ curl -F 'name=Dan' -H 'Content-Type: multipart/magic' https://example.com
6. -d vs -F
Earlier, we introduced the use of - d to construct a basic POST request and - F to construct a multipart formpost request. So what's the difference between the two options and when to use them?
Both options send the specified data to the server. The difference lies in the format of data transmission. Most of the time, the receiving end specifies the format that the client wants to send data. The client cannot specify the format at will.
- HTML form
When using an HTML form, a form is specified using the < form > tag, which allows the browser to use the POST method. If the tag contains enctype = multipart / form data, this means that multipart formpost is used, and the - F option is used in curl. A typical scenario is that the form contains < input type = file > tags.
- No HTML form
The POST method does not have to be in HTML. In many service s and APIs, POST requests can also be used.
If these service s expect to use JSON or other similar format data, then this is a normal POST request. You can use the - d option in curl. However, note whether the default content type of - d is the desired format. If not, you can use - H to change it.
3.4 HTTP redirect
Redirection is a fundamental part of the HTTP protocol. In redirection, what the server gives the client is not what the client wants, but a specific instruction to tell the client where to request if you want to obtain the desired data.
But not all redirects are the same. What method is used for the request after redirection? How long?
All redirects will return the Header of Location: to specify a new URL.
1. curl: redirect
In curl, there is no redirection by default. You can use the - L or -- location option to tell curl to redirect:
$ curl -L http://example.com
2. GET or POST
After the first request, the server will tell the client the method to use for the next request. The response code for redirection is as follows:
| Method | Permanent | Temporary |
|---|---|---|
| Switch to GET | 301 | 302 and 303 |
| Use the method of the first request | 308 | 307 |
We can specify what method curl uses when redirecting. If we don't use the GET method for the first request and don't want curl to use the GET method by default after resetting, we can use -- post301, - post302 and -- post303 options to specify.
3.5 modifying HTTP requests
Each request has a request line, some request headers and optional request bodies. Here we look at the parts that can be modified in curl, including request lines and request headers.
1. Request method
Include the method used for this request in the request line. We can use the following simple command to perform a GET method:
$ curl http://example.com/file
This generates the following request line:
GET /file HTTP/1.1
In * * * HTTP method * * *, we can specify what method to use through specific options. Here we can also use the - X option to specify:
$ curl -X POST http://example.com
2. Modify request header
In curl, we can use the - H or -- Header option to specify the Header. We used - h to specify the content type before. In fact, the Header is a key: value pair:
$ curl -H "HeaderName: HeaderValue" http://example.com
3. Referer
We can also specify where we jump from through the -- referer option in curl:
$ curl --referer http://fromexample.com http://toexample.com
4. User Agent
This field is used to represent the device information of the client. According to this field, the server will return web pages in different formats for different devices. In curl, you can use -- user agent to specify:
$ curl --user-agent "[User Agent]" http://example.com
3.6 Cookies
HTTP is a stateless protocol. Cookies can be used to save some states in a session. The server sets cookies through set Cookie:, and the client can carry these data in the next request.
1. Set cookies
We can use the -- Cookie option to set a Cookie:
$ curl --cookie "CookieName=CookieValue" http://example.com
2. Read Cookies from the file
curl will not remember the cookies set by the server by default, nor will it carry cookies in the next request. Unless the user sets it himself through options.
We can save the previous Cookies to a file, and then specify curl to read the Cookies in the file in the next request:
$ curl -b cookies.txt http://example.com
-The b option specifies curl to read Cookies from the given file.
However, it is important to note that here is only to read Cookies. If the server modifies Cookies in this request, curl will not be saved unless we specify it manually.
3. Write Cookies to the file
We can use the - c option to specify curl to save the Cookies set by the server in this request:
$ curl -c cookie.jar.txt http://example.com
Sometimes, we need to read Cookies from files and save Cookies set by the server. Then you can use both - b and - c options:
$ curl -b cookies.txt -c cookie.jar.txt http://example.com
Author: what do you have for lunch today
Link: https://www.jianshu.com/p/fc0eb6c60816
Source: Jianshu
The copyright belongs to the author. For commercial reprint, please contact the author for authorization, and for non-commercial reprint, please indicate the source.