1. Introduction to httpclient
HTTP protocol is one of the most widely used and important protocols on the Internet. More and more Java applications need to access network resources directly through HTTP protocol. Although the basic function of accessing HTTP protocol has been provided in the java net package of JDK, for most applications, the functions provided by JDK library itself are not rich and flexible enough. HttpClient is a sub project under Apache Jakarta Common, which is used to provide an efficient, up-to-date and feature rich client programming toolkit supporting HTTP protocol, and it supports the latest versions and suggestions of HTTP protocol. HttpClient has been used in many projects, such as the other two famous open source projects Cactus and HTMLUnit on Apache Jakarta. The Commons HttpClient project has been terminated and will no longer be developed. It has been replaced by the HttpClient and HttpCore modules in the Apache HttpComponents project, which provide better performance and greater flexibility-- Baidu Encyclopedia
HttpClient official website: https://hc.apache.org/
Usage scenario: crawler and interface interaction between multiple systems
2. Send http request through JDK native API
package com.lmc.httpclient;
import org.junit.jupiter.api.Test;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLConnection;
import java.nio.charset.StandardCharsets;
/**
* @description:
* @Author: lmc
* @date: 2022/1/9 11:31
*/
public class JDKAPITest {
@Test
public void testHttp() throws IOException {
String urlStr = "https://www.baidu.com";
URL url = new URL(urlStr);
URLConnection urlConnection = url.openConnection();
HttpURLConnection httpURLConnection = (HttpURLConnection) urlConnection;
//You can set the request type, etc
httpURLConnection.setRequestMethod("GET");//Request type
httpURLConnection.setRequestProperty("Accept-Charset","utf-8");//Request header settings
//Gets the input stream of HttpURLConnection
InputStream is = httpURLConnection.getInputStream();
InputStreamReader isr = new InputStreamReader(is, StandardCharsets.UTF_8);
BufferedReader br = new BufferedReader(isr);
String line;
while ((line = br.readLine()) != null) {
System.out.println(line);
}
}
}
The request result is: Baidu's web content
<!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=https://ss1. bdstatic. com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu. Min.css > < title > Baidu, you will know < / Title > < / head > < body link = #0000cc > < div id = wrapper > < div id = head > < div class = head_ wrapper> <div class=s_ form> <div class=s_ form_ wrapper> <div id=lg> <img hidefocus=true src=//www.baidu. com/img/bd_ logo1. png width=270 height=129> </div> <form id=form name=f action=//www.baidu. com/s class=fm> <input type=hidden name=bdorz_ come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_ bp value=1> <input type=hidden name=rsv_ idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ IPT value MaxLength = 255 autocomplete = off autofocus = autofocus > < / span > < span class = "BG s_btn_wr" > < input type = submit id = Su value = Baidu class = "BG s_btn" autofocus > < / span > < / form > < / div > < / div > < div id = U1 > < a href= http://news.baidu.com name=tj_ Trnews class = mnav > News < / a > < a href= https://www.hao123.com name=tj_ trhao123 class=mnav>hao123</a> <a href= http://map.baidu.com name=tj_ TRMAP class = mnav > map < / a > < a href= http://v.baidu.com name=tj_ Trvideo class = mnav > Video < / a > < a href= http://tieba.baidu.com name=tj_ Trtieba class = mnav > Post Bar < / a > < noscript > < a href= http://www.baidu.com/bdorz/login.gif?login& ; tpl=mn& u=http%3A%2F%2Fwww. baidu. com%2f%3fbdorz_ come%3d1 name=tj_ Login class = LB > log in to < / a > < / noscript > < script > document write('<a href=" http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u= '+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_ Login "class =" LB "> login < / a >);
</script> <a href=//www.baidu. com/more/ name=tj_ briicon class=bri style="display: block;"> More products < / a > < / div > < / div > < / div > < div id = ftcon > < div id = ftconw > < p id = LH > < a href= http://home.baidu.com >About Baidu < / a > < a href= http://ir.baidu.com >About Baidu</a> </p> <p id=cp>© 2017 Baidu < a href= http://www.baidu.com/duty/ >Must read < / a > & nbsp; before using Baidu< a href= http://jianyi.baidu.com/ Class = CP feedback > feedback < / a > & nbsp; Beijing ICP Certificate No. 030173 & nbsp< img src=//www.baidu. com/img/gs. gif> </p> </div> </div> </div> </body> </html>
3. get request of httpclient
3.1 get no participation request and forged browser
Code implementation. See the notes in the code for details.
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.lmc</groupId>
<artifactId>spring-httpclient</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.13</version>
</dependency>
<dependency>
<groupId>org.wso2.orbit.org.apache.httpcomponents</groupId>
<artifactId>httpmime</artifactId>
<version>4.5.13.wso2v1</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<version>RELEASE</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.60</version>
</dependency>
</dependencies>
</project>package com.lmc.httpclient;
import org.apache.http.HttpEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
import org.junit.jupiter.api.Test;
import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
/**
* @description:
* @Author: lmc
* @date: 2022/1/6 21:46
*/
/**
* Send get request using httpclient
* 1.No parameter request
* 2.Add request header: forged into browser and anti-theft chain
*/
public class HttpClientTest {
@Test
public void testGet() throws UnsupportedEncodingException {
//The httpClient client that can be closed is equivalent to you opening a browser
CloseableHttpClient closeableHttpClient = HttpClients.createDefault();
String urlStr = "https://www.baidu.com";
//Construct httpClient request object
HttpGet httpGet = new HttpGet(urlStr);
//User agent user agent is the browser you use to solve the problem that is not considered to be a real person
httpGet.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36");
//Anti theft chain, value: the url of the website where the anti-theft chain occurs
//httpGet.addHeader("Referer","https://www.baidu.com");
//Closable response flow
CloseableHttpResponse response = null;
try {
response = closeableHttpClient.execute(httpGet);
//Get response results
HttpEntity entity = response.getEntity();
String toString = EntityUtils.toString(entity, StandardCharsets.UTF_8);
System.out.println(toString);
//Make sure the flow is closed
EntityUtils.consume(entity);
} catch (IOException e) {
e.printStackTrace();
} finally {
if (closeableHttpClient != null) {
try {
closeableHttpClient.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if (response != null) {
try {
response.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}
3.2 get parameter request, parameter processing
package com.lmc.httpclient;
import org.apache.http.HttpEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
import org.junit.jupiter.api.Test;
import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
/**
* @description:
* @Author: lmc
* @date: 2022/1/6 21:46
*/
/**
* Send get request using httpclient
* 1.No parameter request
* 2.Add request header: forged into browser and anti-theft chain
* 3.urlencode is required if there is a parameter
*/
public class HttpClientTest {
@Test
public void testGet() throws UnsupportedEncodingException {
//The httpClient client that can be closed is equivalent to you opening a browser
CloseableHttpClient closeableHttpClient = HttpClients.createDefault();
//String urlStr = "https://www.baidu.com";
String passwordParam = "123+abc|456";
//Do urlencode: if it is a browser, the browser will automatically help us do it.
// However, when using httpClient, we need to use urlencoders for special characters to avoid that special characters cannot be accepted when parameters are passed// 123%2Babc%7C456
passwordParam = URLEncoder.encode(passwordParam, StandardCharsets.UTF_8.name());
String urlStr = "http://localhost:8081/test1?name=aaa&passWord=" + passwordParam;
//Construct httpClient request object
HttpGet httpGet = new HttpGet(urlStr);
//User agent user agent is the browser you use to solve the problem that is not considered to be a real person
httpGet.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36");
//Anti theft chain, value: the url of the website where the anti-theft chain occurs
//httpGet.addHeader("Referer","https://www.baidu.com");
//Closable response flow
CloseableHttpResponse response = null;
try {
response = closeableHttpClient.execute(httpGet);
//Get response results
HttpEntity entity = response.getEntity();
String toString = EntityUtils.toString(entity, StandardCharsets.UTF_8);
System.out.println(toString);
//Make sure the flow is closed
EntityUtils.consume(entity);
} catch (IOException e) {
e.printStackTrace();
} finally {
if (closeableHttpClient != null) {
try {
closeableHttpClient.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if (response != null) {
try {
response.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}
The interface requested by get is: (the interface called by the following post request is also here, where the User attributes are name and passWord)
package com.lmc.controllr;
import com.lmc.model.User;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;
import javax.servlet.http.HttpServletRequest;
import java.util.Enumeration;
import java.util.List;
/**
* @description:
* @Author: lmc
* @date: 2022/1/8 17:19
*/
@RestController
public class TestHttpClient {
/**
* Demo get request:
* 1.No content type
* 2.The browser automatically performs urlencode, and the web container (such as Tomcat) will automatically parse urlencode
* 3.If we request, we use 111 + 222 --- > to display in the address bar: 1111%2B222
* http://localhost:8081/test1?name=%E9%87%8C%E9%9D%A2%E7%A9%BF&passWord=1111%2B222
*
* @param name
* @param passWord
* @param request
*/
@RequestMapping(value = "/test1", method = RequestMethod.GET)
public String test1(String name, String passWord, HttpServletRequest request) {
System.out.println("The accepted parameters are:" + name + "::::" + passWord);
//Get all request headers
Enumeration<String> headerNames = request.getHeaderNames();
while (headerNames.hasMoreElements()) {
String headName = headerNames.nextElement();
System.out.println(headName + ":" + request.getHeader(headName));
}
System.out.println("=========================================");
System.out.println(request.getContentType());
return "get Request test";
}
/**
* post application/x-www-form-urlencoded type
*/
@PostMapping(value = "/test2")
public String test2(User user, HttpServletRequest request) {
//Get all request headers
System.out.println(user.toString());
Enumeration<String> headerNames = request.getHeaderNames();
while (headerNames.hasMoreElements()) {
String headName = headerNames.nextElement();
System.out.println(headName + ":" + request.getHeader(headName));
}
System.out.println("=========================================");
System.out.println(request.getContentType());
return "post form Request test";
}
/**
* post application/json type
*/
@PostMapping(value = "/testJson")
public String testJson(@RequestBody User user, HttpServletRequest request) {
//Get all request headers
System.out.println(user.toString());
Enumeration<String> headerNames = request.getHeaderNames();
while (headerNames.hasMoreElements()) {
String headName = headerNames.nextElement();
System.out.println(headName + ":" + request.getHeader(headName));
}
System.out.println("=========================================");
System.out.println(request.getContentType());
return "post json Request test";
}
/**
* post multipart/form-data type
*/
@PostMapping(value = "/test3")
public String test3(@RequestParam("fileName") MultipartFile[] multipartFiles,
User user, HttpServletRequest request) {
for (MultipartFile multipartFile : multipartFiles) {
System.out.println("The name of the uploaded file is:" + multipartFile.getOriginalFilename());
}
System.out.println("Other business input" + user.toString());
Enumeration<String> headerNames = request.getHeaderNames();
while (headerNames.hasMoreElements()) {
String headName = headerNames.nextElement();
System.out.println(headName + ":" + request.getHeader(headName));
}
System.out.println("=========================================");
System.out.println(request.getContentType());
return "post multipart/form-data test";
}
}
About the spooling of this interface:
3.3 get response first class
//Represents the status of success or failure of this request StatusLine statusLine = response.getStatusLine(); //Get response header Header[] allHeaders = response.getAllHeaders();
3.4 save network pictures to local
package com.lmc.httpclient;
import org.apache.http.Header;
import org.apache.http.HttpEntity;
import org.apache.http.HttpStatus;
import org.apache.http.StatusLine;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
import org.junit.jupiter.api.Test;
import java.io.FileOutputStream;
import java.io.IOException;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
/**
* @description:
* @Author: lmc
* @date: 2022/1/9 12:18
*/
public class HttpClientImg {
@Test
public void test() {
CloseableHttpClient closeableHttpClient = HttpClients.createDefault();
String urlStr = "https://img-blog.csdnimg.cn/1c1c279726e34a018c15f3c12ee45983.png";
HttpGet httpGet = new HttpGet(urlStr);
//Closable response flow
CloseableHttpResponse response = null;
try {
response = closeableHttpClient.execute(httpGet);
HttpEntity entity = response.getEntity();
//Image / JPG image / jpeg image / PNG image / picture suffix
String contentType = entity.getContentType().getValue();
System.out.println(entity.getContentType());//Content-Type: image/png
String suffix = ".jpg";
if (contentType.contains("jpg") || contentType.contains("jpeg")) {
suffix = ".jpg";
} else if (contentType.contains("bmp") || contentType.contains("bitmap")) {
suffix = ".bmp";
} else if (contentType.contains("png")) {
suffix = ".png";
} else if (contentType.contains("gif")) {
suffix = ".gif";
}
//Get file byte stream
byte[] bytes = EntityUtils.toByteArray(entity);
String localAbsPath = "E:\\iotest\\HttpClientTest\\123"+suffix;
FileOutputStream fileOutputStream = new FileOutputStream(localAbsPath);
fileOutputStream.write(bytes);
fileOutputStream.close();
EntityUtils.consume(entity);
} catch (IOException e) {
e.printStackTrace();
} finally {
if (closeableHttpClient != null) {
try {
closeableHttpClient.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if (response != null) {
try {
response.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}
4. post request of httpclient
Here are three methods of post request: (see the above for the requested interface)
① Send a post request of application/x-www-form-urlencoded type;
② Send application/json type post request, which is generally used for docking with other systems;
③ To send a post request for uploading files of multipart / form data type, a dependent httpmime is required.
package com.lmc.httpclient;
import com.alibaba.fastjson.JSONObject;
import org.apache.http.Consts;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHost;
import org.apache.http.NameValuePair;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.StringEntity;
import org.apache.http.entity.mime.HttpMultipartMode;
import org.apache.http.entity.mime.MultipartEntityBuilder;
import org.apache.http.entity.mime.content.FileBody;
import org.apache.http.entity.mime.content.StringBody;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.message.BasicHeader;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.util.EntityUtils;
import org.junit.jupiter.api.Test;
import java.io.File;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
/**
* @description:
* @Author: lmc
* @date: 2022/1/9 14:21
*/
public class HttpClientPostTest {
/**
* Send a post request of application/x-www-form-urlencoded type
*/
@Test
public void testPost1(){
CloseableHttpClient closeableHttpClient = HttpClients.createDefault();
String urlStr = "http://localhost:8081/test2";
HttpPost httpPost = new HttpPost(urlStr);
//Set parameters for the post object
List<NameValuePair> list = new ArrayList<>();
list.add(new BasicNameValuePair("name","lmc"));
list.add(new BasicNameValuePair("passWord","123456"));
//Set the parameter set to formEntity
UrlEncodedFormEntity formEntity = new UrlEncodedFormEntity(list, Consts.UTF_8);
httpPost.setEntity(formEntity);
//Closable response flow
CloseableHttpResponse response = null;
try {
response = closeableHttpClient.execute(httpPost);
//Get response results
//httpEntity can be used not only as a result, but also as a parameter entity of a request. There are many implementations.
HttpEntity entity = response.getEntity();
String toString = EntityUtils.toString(entity, StandardCharsets.UTF_8);
System.out.println(toString);
//Make sure the flow is closed
EntityUtils.consume(entity);
} catch (IOException e) {
e.printStackTrace();
} finally {
if (closeableHttpClient != null) {
try {
closeableHttpClient.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if (response != null) {
try {
response.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
/**
* Send application/json type post request, which is generally used for docking with other systems
*/
@Test
public void testPost2(){
CloseableHttpClient closeableHttpClient = HttpClients.createDefault();
String urlStr = "http://localhost:8081/testJson";
HttpPost httpPost = new HttpPost(urlStr);
JSONObject jsonObject = new JSONObject();
jsonObject.put("name","Ha ha ha");
jsonObject.put("passWord","123455667");
StringEntity jsonEntity = new StringEntity(jsonObject.toString(), Consts.UTF_8);
//Set the content type for entity
jsonEntity.setContentType(new BasicHeader("Content-Type","application/json; charset=UTF-8"));
//Set the encoding of entity
jsonEntity.setContentEncoding(Consts.UTF_8.name());
httpPost.setEntity(jsonEntity);
//Closable response flow
CloseableHttpResponse response = null;
try {
response = closeableHttpClient.execute(httpPost);
//Get response results
//httpEntity can be used not only as a result, but also as a parameter entity of a request. There are many implementations.
HttpEntity entity = response.getEntity();
String toString = EntityUtils.toString(entity, StandardCharsets.UTF_8);
System.out.println(toString);
//Make sure the flow is closed
EntityUtils.consume(entity);
} catch (IOException e) {
e.printStackTrace();
} finally {
if (closeableHttpClient != null) {
try {
closeableHttpClient.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if (response != null) {
try {
response.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
/**
* Send a post request for uploading files of multipart / form data type
*/
@Test
public void testPost3(){
CloseableHttpClient closeableHttpClient = HttpClients.createDefault();
String urlStr = "http://localhost:8081/test3";
HttpPost httpPost = new HttpPost(urlStr);
//Build an implementation object of ContentBody
FileBody fileBody = new FileBody(new File("E:\\iotest\\test01.txt"));
//Construct the entity used by the uploaded file
MultipartEntityBuilder builder = MultipartEntityBuilder.create();
builder.setCharset(Consts.UTF_8);//Set encoding
builder.setContentType(ContentType.create("multipart/form-data",Consts.UTF_8));
builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);//Set browser mode
//For different form fields, if they contain Chinese, you cannot use addTextBody, otherwise they are garbled
//text is the value entered
StringBody userNameTextBody = new StringBody("Xiao Ming", ContentType.create("text/plain", Consts.UTF_8));
HttpEntity httpEntity = builder.addPart("fileName", fileBody) //The first way to upload files
.addBinaryBody("fileName", new File("E:\\iotest\\javase.pdf")) //The second way to upload files
.addPart("name", userNameTextBody)
.addTextBody("passWord", "admin")
.build();
httpPost.setEntity(httpEntity);
//Closable response flow
CloseableHttpResponse response = null;
try {
response = closeableHttpClient.execute(httpPost);
//Get response results
//httpEntity can be used not only as a result, but also as a parameter entity of a request. There are many implementations.
HttpEntity entity = response.getEntity();
String toString = EntityUtils.toString(entity, StandardCharsets.UTF_8);
System.out.println(toString);
//Make sure the flow is closed
EntityUtils.consume(entity);
} catch (IOException e) {
e.printStackTrace();
} finally {
if (closeableHttpClient != null) {
try {
closeableHttpClient.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if (response != null) {
try {
response.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}
Reference: happy man java