使用Java原生库封装一个简单好用的Http请求工具类
需求
最近在写一个Spring boot的Java后端小项目,而在项目的需求中需要向网络发送http请求收集(爬取)网络上的信息。具体的需求如下:
- 能发送get、post请求;
- 通过方法添加查询参数而不是用拼接url的方式;
- 发送post请求能携带表单数据和json格式的数据;
- 能添加自定义头信息;
- 能添加和获取Cookie(其实添加头信息就能添加Cookie了,但是因为用的多所以直接加个方法);
- 能接收文本信息或二进制文件。
在网上搜了一下,比较出名有的Apache下的库HttpClient,非常强大,几乎包含了http所有的功能。虽说如此,但大部分功能我都是用不到的,所以我来说可能太重了,目测了一下HttpClient库中大概有三四百个类。在JDK 9版本中开始集成了HttpClient,但是项目环境是jdk8而且据说HttpClient库还在孵化的包里,应该还不是很成熟吧,所以想要自己封装一个http请求工具类。
如果想直接使用请点击链接—>使用Java原生库封装一个简单好用的Http请求工具类
构思
我个人是比较钟爱链式调用的,简单的请求一句话就搞定了,所以在所有的添加请求参数的方法中返回的应该是自身对象而不是像List
类那样返回void
。
原生的HttpURLConnection
类是请求信息和应答信息都是在同一个对象中的,因为叫"Connection"嘛,一个连接当然既要发送请求并且接收应答信息咯,而且一个HttpURLConnection
对象是可以多次使用的。不过这些并不是我需要的,而且请求信息和应答信息放在一个对象中有点乱,所以除了请求类(Request)外还要封装一个应答类(Rsponse)。
请求类
字段(feild)
现在可以开始动手封装了,首先是要保存在对象中的请求参数:
// 访问目标
private String url;
// 路径参数
private Map<String, Object> query = new HashMap();
// 访问method
private String method = Method.GET;
// Post data参数
private Map postData = new HashMap();
// Cookies 注:若headers中和属性cookies同时设置,属性cookies将会覆盖headers中的cookies
private String cookies = "";
// 头信息
private Map<String, String> headers = new HashMap<String, String>();
// 返回结果的编码
private String encode = Encode.UTF8;
// 返回连接对象
private HttpURLConnection conn;
// 请求后是否关闭连接
private boolean isCloseConnectionAfterRequest = true;
private boolean doInput = true;
private boolean doOutput = false;
private int readTimeout = -1;
private int connectTimeout = -1;
private boolean useCaches = false;
前面有注释字段的是在本类中要处理的数据,除了cookie外参数信息都用一个Map保存,其中postData是比较需要注意的,因为在HttpURLConnection
中post数据都是从相同的地方写入的,最初我只考虑了发送form表单的post请求,格式如下所示:
id=1&name=zhangsan&password=1342151345
而其实还有json格式的post请求,如下所示:
{
"touser": "",
"template_id": "",
"page": "",
"form_id": "",
"data": {
"keyword1": {
"value": ""
},
"keyword2": {
"value": ""
},
"keyword3": {
"value": ""
},
"keyword4": {
"value": ""
}
},
"emphasis_keyword": "keyword1.DATA"
}
所以简单的一维键值对是不能满足的,这里我的解决方案是使用Map对象和List对象嵌套,在解析数据前先判断头信息中是表单数据还是json格式数据,然后再将其分别解析成对应的字符串。
下面几个字段是用于HttpURLConnection
中的参数,虽说不需要,但也留一个接口吧,万一要用呢。
方法设计(method)
public class Urllib {
public Urllib();
public Urllib(String url);
public static Urllib builder();
public Urllib build();
private HttpURLConnection getConnection();
private void openConnection();
public UrlResp request();
/**
* 以GET方式访问
* @return
*/
public UrlResp get();
/**
* 以POST方式访问
* @return
*/
public UrlResp post();
/**
* 以PUT方式访问
* @return
*/
public UrlResp put();
/**
* 以DELETE方式访问
* @return
*/
public UrlResp delete();
/**
* 静态请求方法。
* params按顺序依次是query, postData, method, headers, cookies
* @param url
* @param params
* @return
*/
public static UrlResp request(String url, Object... params);
/**
* 静态get请求方法。
* @param url 目标url
* @param params 按顺序依次为query, headers, cookies
* @return
*/
public static UrlResp get(String url, Object... params);
/**
* 静态post请求方法。
* @param url 目标url
* @param params 按顺序依次为query, postData, headers, cookies
* @return
*/
public static UrlResp post(String url, Object... params);
/**
* 添加一条路径参数
* @param var 路径参数
* @return
*/
public Urllib addPathVariable(Object var);
/**
* 添加一条路径参数
* @param key 键
* @param value 值
* @return
*/
public Urllib addQuery(String key, Object value);
/**
* 添加一条body参数
* @param key
* @param value
* @return
*/
public Urllib addPostData(String key, Object value);
/**
* 添加一条cookie
* @param key
* @param value
* @return
*/
public Urllib addCookie(String key, String value);
/**
* 添加String类型的Cookies
* @param cookie
* @return
*/
public Urllib addCookies(String cookie);
/**
* 添加一条header
* @param key
* @param value
* @return
*/
public Urllib addHeader(String key, String value);
/**
* 转为Json字符串
* @param obj Map或List类型
* @return
*/
public String toJSONString(Object obj);
/**setter和getter省略**
}
前面几个静态方法是我最初设计方法的想法,可以像python那样一个方法可以选择地添加参数,不过还是感觉不够灵活,而链式调用就能随意添加参数(本是受到idea生成的链式setter方法的启发,没想到那几家大的http库都是这样使用的链式调用)。
方法的实现如下:
package com.example.net;
import java.io.IOException;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
public class Urllib {
// 几种method方法
public static final class Method {
private Method() {
}
public static final String GET = "GET";
public static final String POST = "POST";
public static final String HEAD = "HEAD";
public static final String OPTIONS = "OPTIONS";
public static final String PUT = "PUT";
public static final String DELETE = "DELETE";
public static final String TRACE = "TRACE";
}
// 常见编码
public static final class Encode {
private Encode() {
}
public static final String UTF8 = "UTF-8";
public static final String UTF16 = "UTF-16";
public static final String GB18030 = "GB18030";
public static final String GBK = "GBK";
public static final String GB2312 = "GB2312";
public static final String ISO_8859_1 = "ISO-8859-1";
public static final String ASCII = "ASCII";
}
// 访问目标
private String url;
// 路径参数
private Map<String, Object> query = new HashMap();
// 访问method
private String method = Method.GET;
// Post data参数
private Map postData = new HashMap();
// Cookies 注:若headers中和属性cookies同时设置,属性cookies将会覆盖headers中的cookies
private String cookies = "";
// 头信息
private Map<String, String> headers = new HashMap<String, String>();
// 返回结果的编码
private String encode = Encode.UTF8;
// 返回连接对象
private HttpURLConnection conn;
// 请求后是否关闭连接
private boolean isCloseConnectionAfterRequest = true;
private boolean doInput = true;
private boolean doOutput = false;
private int readTimeout = -1;
private int connectTimeout = -1;
private boolean useCaches = false;
public Urllib() {
}
public Urllib(String url) {
this.url = url;
}
public static Urllib builder() {
return new Urllib();
}
public Urllib build() {
return this;
}
private HttpURLConnection getConnection() {
this.openConnection();
return this.conn;
}
private void openConnection() {
if (url == null || "".equals(url))
throw new RuntimeException("url不能为空");
try {
// 构造url
URL url = new URL(this.url + toQueryString());
// 获取连接对象
this.conn = (HttpURLConnection) url.openConnection();
} catch (IOException e) {
e.printStackTrace();
}
}
public UrlResp request() {
UrlResp urlResp = null;
openConnection();
// 设置请求参数
try {
setRequestProperties(conn);
// 请求
urlResp = new UrlResp(conn, encode);
} catch (IOException e) {
e.printStackTrace();
} finally {
if (isCloseConnectionAfterRequest && conn != null) {
conn.disconnect();
}
}
return urlResp;
}
/**
* 以GET方式访问
* !会覆盖之前设置的method
*
* @return
*/
public UrlResp get() {
this.method = Method.GET;
return request();
}
/**
* 以POST方式访问
* !会覆盖之前设置的method
*
* @return
*/
public UrlResp post() {
this.method = Method.POST;
return request();
}
/**
* 以PUT方式访问
* !会覆盖之前设置的method
*
* @return
*/
public UrlResp put() {
this.method = Method.PUT;
return request();
}
/**
* 以DELETE方式访问
* !会覆盖之前设置的method
*
* @return
*/
public UrlResp delete() {
this.method = Method.DELETE;
return request();
}
/**
* 静态请求方法。
* params按顺序依次是query, postData, method, headers, cookies
*
* @param url
* @param params
* @return
*/
public static UrlResp request(String url, Object... params) {
Urllib urllib = new Urllib(url);
parseParams(urllib, params);
return urllib.request();
}
/**
* 静态get请求方法。
*
* @param url 目标url
* @param params 按顺序依次为query, headers, cookies
* @return
*/
public static UrlResp get(String url, Object... params) {
Object query = null;
Object headers = null;
Object cookies = null;
//参数长度
int paramLength = params.length;
if (paramLength > 0)
query = params[0];
if (paramLength > 1)
headers = params[1];
if (paramLength > 2)
cookies = params[3];
return request(url, query, null, Method.GET, headers, cookies);
}
/**
* 静态post请求方法。
*
* @param url 目标url
* @param params 按顺序依次为query, postData, headers, cookies
* @return
*/
public static UrlResp post(String url, Object... params) {
Object query = null;
Object postData = null;
Object headers = null;
Object cookies = null;
//参数长度
int paramLength = params.length;
if (paramLength > 0)
query = params[0];
if (paramLength > 1)
postData = params[1];
if (paramLength > 2)
headers = params[2];
if (paramLength > 3)
cookies = params[3];
return request(url, query, postData, Method.POST, headers, cookies);
}
private static void parseParams(Urllib urllib, Object[] params) {
//参数长度
int paramLength = params.length;
if (paramLength > 0) {
Object param = params[0];
if (param instanceof Map)
urllib.setQuery((Map) param);
}
if (paramLength > 1) {
Object param = params[1];
if (param instanceof Map)
urllib.setPostData((Map) param);
}
if (paramLength > 2) {
Object param = params[2];
if (param instanceof String)
urllib.setMethod((String) param);
}
if (paramLength > 3) {
Object param = params[3];
if (param instanceof Map)
urllib.setHeaders((Map) param);
}
if (paramLength > 4) {
Object param = params[4];
if (param instanceof String)
urllib.setCookies((String) param);
}
}
/**
* 设置请求属性
*
* @param conn
* @throws IOException
*/
private void setRequestProperties(HttpURLConnection conn) throws IOException {
conn.setDoInput(doInput);
conn.setDoOutput(doOutput);
conn.setUseCaches(useCaches);
if (readTimeout > 0)
conn.setReadTimeout(readTimeout);
if (connectTimeout > 0)
conn.setConnectTimeout(connectTimeout);
conn.setRequestMethod(method);
if (!headers.isEmpty()) {
for (Map.Entry<String, String> header : headers.entrySet())
conn.setRequestProperty(header.getKey(), header.getValue());
}
if (cookies != null && !"".equals(cookies.trim())) {
conn.setRequestProperty("Cookie", cookies);
}
if (Method.POST.equals(method.toUpperCase())) {
conn.setUseCaches(false);
// indicates that the application intends to write postData to the URL connection.
conn.setDoOutput(true);
OutputStream os = conn.getOutputStream();
os.write(toPostDataString().getBytes());
os.flush();
}
}
private String toPostDataString() {
if (postData.size() < 1)
return "";
String postDataString = null;
String contentType = headers.get("Content-Type");
if (contentType != null && contentType.contains("form")) {
postDataString = toQueryString(this.postData);
postDataString = postDataString.substring(1, postDataString.length());
}
// else if (ContentType.contains("json"))
else {
postDataString = toJSONString(this.postData);
}
return postDataString;
}
/**
* 查询参数转为字符串
*
* @return
*/
private String toQueryString(Map<String, Object> query) {
StringBuilder sb = new StringBuilder();
if (!query.isEmpty()) {
if (url.contains("?"))
sb.append("&");
else
sb.append("?");
for (Map.Entry entry : query.entrySet())
sb.append(entry.getKey().toString() + "=" + entry.getValue().toString() + "&");
sb.delete(sb.length() - 1, sb.length());
}
return sb.toString();
}
/**
* 查询参数转为字符串
*
* @return
*/
private String toQueryString() {
return this.toQueryString(this.query);
}
/**
* 添加一条路径参数
*
* @param var 路径参数
* @return
*/
public Urllib addPathVariable(Object var) {
this.url += "/" + var;
return this;
}
/**
* 添加一条路径参数
*
* @param key 键
* @param value 值
* @return
*/
public Urllib addQuery(String key, Object value) {
if (key == null || value == null)
throw new IllegalArgumentException("key或value不可为空");
query.put(key, value);
return this;
}
/**
* 添加一条body参数
*
* @param key
* @param value
* @return
*/
public Urllib addPostData(String key, Object value) {
if (key == null || value == null)
throw new IllegalArgumentException("key或value不可为空");
postData.put(key, value);
return this;
}
/**
* 添加一条cookie
*
* @param key
* @param value
* @return
*/
public Urllib addCookie(String key, String value) {
if (key == null || value == null)
throw new IllegalArgumentException("key或value不可为空");
cookies = cookies.trim() + ("".equals(cookies.trim()) ? "" : ";") + key + "=" + value;
return this;
}
public Urllib addCookies(String cookie) {
if (cookie == null)
throw new IllegalArgumentException("cookie不可为空");
this.cookies = this.cookies.trim() + ("".equals(this.cookies.trim()) ? "" : ";") + cookie;
return this;
}
/**
* 添加一条header
*
* @param key
* @param value
* @return
*/
public Urllib addHeader(String key, String value) {
if (key == null || "".equals(key) || value == null)
throw new IllegalArgumentException("key或value不可为空");
headers.put(key, value);
return this;
}
//<editor-fold desc="工具方法">
/**
* 转为Json字符串
*
* @param obj Map或List类型
* @return
*/
public String toJSONString(Object obj) {
if (obj instanceof Map) {
Map dataMap = (Map) obj;
if (dataMap == null) {
return "{}";
}
StringBuffer sb = new StringBuffer();
Iterator entries = dataMap.entrySet().iterator();
while (entries.hasNext()) {
Map.Entry entry = (Map.Entry) entries.next();
sb.append("\"" + entry.getKey() + "\":");
Object value = entry.getValue();
parseValue(sb, value);
if (entries.hasNext())
sb.append(",");
}
sb.insert(0, "{").append("}");
return sb.toString();
} else if (obj instanceof List) {
List dataList = (List) obj;
if (dataList == null)
return "[]";
StringBuffer sb = new StringBuffer();
Iterator entries = dataList.iterator();
while (entries.hasNext()) {
Object value = entries.next();
parseValue(sb, value);
if (entries.hasNext())
sb.append(",");
}
sb.insert(0, "[").append("]");
return sb.toString();
} else
throw new RuntimeException("类型不支持");
}
private void parseValue(StringBuffer sb, Object value) {
if (value == null)
sb.append("null");
else if (value instanceof Integer || value instanceof Double || value instanceof Float)
sb.append(value);
else if (value instanceof Map || value instanceof List)
sb.append(toJSONString(value));
else
sb.append("\"" + value + "\"");
}
//</editor-fold>
//<editor-fold desc="Getter and Setter方法">
public String getUrl() {
return url;
}
public Urllib setUrl(String url) {
if (url == null || "".equals(url))
throw new IllegalArgumentException("url不能为空或空串");
this.url = url;
return this;
}
public Map<String, Object> getQuery() {
return query;
}
public Urllib setQuery(Map<String, Object> query) {
if (query == null)
throw new IllegalArgumentException("param不能为空");
this.query = query;
return this;
}
public String getMethod() {
return method;
}
public Urllib setMethod(String method) {
if (method == null || "".equals(method))
throw new IllegalArgumentException("method不能为空");
this.method = method;
return this;
}
public Map getPostData() {
return postData;
}
public Urllib setPostData(Map postData) {
if (postData == null)
throw new IllegalArgumentException("data不能为空");
this.postData = postData;
return this;
}
public String getCookies() {
return cookies;
}
public Urllib setCookies(String cookies) {
if (cookies == null || "".equals(cookies))
throw new IllegalArgumentException("cookie不能为空");
this.cookies = cookies;
return this;
}
public Map<String, String> getHeaders() {
return headers;
}
public Urllib setHeaders(Map<String, String> headers) {
if (headers == null)
throw new IllegalArgumentException("headers不能为空");
this.headers = headers;
return this;
}
public String getEncode() {
return encode;
}
public Urllib setEncode(String encode) {
if (encode == null || "".equals(encode))
throw new IllegalArgumentException("encode不能为空");
this.encode = encode;
return this;
}
public boolean isDoInput() {
return doInput;
}
public Urllib setDoInput(boolean doInput) {
this.doInput = doInput;
return this;
}
public boolean isDoOutput() {
return doOutput;
}
public Urllib setDoOutput(boolean doOutput) {
this.doOutput = doOutput;
return this;
}
public int getReadTimeout() {
return readTimeout;
}
public Urllib setReadTimeout(int readTimeout) {
this.readTimeout = readTimeout;
return this;
}
public int getConnectTimeout() {
return connectTimeout;
}
public Urllib setConnectTimeout(int connectTimeout) {
this.connectTimeout = connectTimeout;
return this;
}
public boolean isUseCaches() {
return useCaches;
}
public Urllib setUseCaches(boolean useCaches) {
this.useCaches = useCaches;
return this;
}
//</editor-fold>
}
其中最主要的是request方法,其他get、post等都只是设置method后在再内部调用request方法。
在request方法中,主要步骤是
- 通过url参数打开连接
openConnection
; - 设置请求参数;
- 构造并获取Response。
设置参数中如果是post或put请求,需要设置conn对象为setDoOutput(true),这样都会输出数据,在toPostDataString()方法中需要分别判断form和json格式的数据(还有xml、html等格式的数据,如果有需求可以补充一个)。其中对Json格式数据的解析是我自己写的一个方法,其实就是判断是Map还是List,如果是Map,则在外层包一个"{}",List则包一个"[]",再进行递归调用。自己写json解析就不用引入外部的库了。
应答类
应答类就简单了,常用的信息就是状态码,头信息(Cookie)和数据,因为数据可能是二进制文件,所以保存的是二进制(注意不要获取太大的文件),主要的方法如下:
package com.example.net;
import java.io.BufferedInputStream;
import java.io.BufferedReader;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;
import java.net.HttpURLConnection;
import java.security.Permission;
import java.util.List;
import java.util.Map;
public class UrlResp {
private byte[] byteContent;
private String errorMsg;
private int responseCode;
private HttpURLConnection conn;
public UrlResp(HttpURLConnection conn, String encode) {
this.conn = conn;
BufferedInputStream bis = null;
ByteArrayOutputStream baos = null;
try {
bis = new BufferedInputStream(conn.getInputStream());
baos = new ByteArrayOutputStream();
byte[] bytes = new byte[2048];
int len = 0;
while (-1 != (len = bis.read(bytes))) {
baos.write(bytes, 0, len);
}
byteContent = baos.toByteArray();
responseCode = conn.getResponseCode();
} catch (IOException e) {
e.printStackTrace();
BufferedReader in = null;
// 错误信息
try {
StringBuffer msg = new StringBuffer(30);
in = new BufferedReader(new InputStreamReader(conn.getErrorStream(), encode));
String line = null;
while ((line = in.readLine()) != null) {
msg.append(line);
}
this.errorMsg = msg.toString();
responseCode = -1;
} catch (UnsupportedEncodingException e1) {
e1.printStackTrace();
} catch (IOException e1) {
e1.printStackTrace();
} finally {
if (in != null) {
try {
in.close();
} catch (IOException e1) {
e1.printStackTrace();
}
}
}
} finally {
if (baos != null) {
try {
baos.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if (bis != null){
try {
bis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
public byte[] getByteContent() {
return byteContent;
}
public String getText(){
return byteContent == null ? "" : new String(byteContent);
}
public String getErrorMsg() {
return errorMsg;
}
@Override
public String toString() {
if (errorMsg != null)
return errorMsg;
if (byteContent != null) {
return new String(byteContent);
}
return "";
}
public String getHeaderFieldKey(int n) {
return conn.getHeaderFieldKey(n);
}
public String getHeaderField(int n) {
return conn.getHeaderField(n);
}
public int getResponseCode() {
return responseCode;
}
public String getResponseMessage(){
try {
return conn.getResponseMessage();
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
public long getHeaderFieldDate(String name, long Default) {
return conn.getHeaderFieldDate(name, Default);
}
public Permission getPermission(){
try {
return conn.getPermission();
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
public int getContentLength() {
return conn.getContentLength();
}
public long getContentLengthLong() {
return conn.getContentLengthLong();
}
public String getContentType() {
return conn.getContentType();
}
public String getContentEncoding() {
return conn.getContentEncoding();
}
public long getExpiration() {
return conn.getExpiration();
}
public long getDate() {
return conn.getDate();
}
public String getHeaderField(String name) {
return conn.getHeaderField(name);
}
public Map<String, List<String>> getHeaderFields() {
return conn.getHeaderFields();
}
public int getHeaderFieldInt(String name, int Default) {
return conn.getHeaderFieldInt(name, Default);
}
public long getHeaderFieldLong(String name, long Default) {
return conn.getHeaderFieldLong(name, Default);
}
public HttpURLConnection getConn() {
return conn;
}
}
示例
- 发送get请求
UrlResp res = Urllib.builder()
.setUrl("http://httpbin.org")
.addPathVariable("get")
.addQuery("keyword", "csdn")
.addHeader("User-Agent", "Chrome")
.addHeader("Content-Type", "text/html")
.addCookies("JSESSIONID=2454;aie=adf")
.addCookie("username", "bin")
.get();
if (res.getResponseCode() == UrlResp.HTTP_OK){
System.out.println(res.getText());
}
打印结果
- 发送post请求并携带表单数据
public void test(){
UrlResp res = Urllib.builder()
.setUrl("http://httpbin.org")
.addPathVariable("post")
.addQuery("keyword", "csdn")
.addHeader("User-Agent", "Chrome")
.addHeader("Content-Type", "application/x-www-form-urlencoded")
.addCookies("JSESSIONID=2454;aie=adf")
.addCookie("yes", "no")
.addPostData("username", "zhangsan")
.addPostData("password", "154134513")
.addPostData("gender", "male")
.addPostData("age", 18)
.post();
if (res.getResponseCode() == UrlResp.HTTP_OK){
System.out.println(res.getText());
}
}
打印结果
- 发送post请求并携带json格式数据
构造json数据稍微复杂一点,毕竟不能像python和php语言。
public void test(){
Map data = new HashMap();
Map keyword1 = new HashMap();
keyword1.put("value", "98gadf");
data.put("keyword1", keyword1);
Map keyword2 = new HashMap();
keyword1.put("value", "9fghfsgdf");
data.put("keyword2", keyword1);
List list = new ArrayList();
list.add("a");
list.add("b");
list.add("c");
UrlResp res = Urllib.builder()
.setUrl("http://httpbin.org")
.addPathVariable("post")
.addHeader("User-Agent", "Chrome")
.addHeader("Content-Type", "application/json")
.addPostData("touser", "orafd98bu")
.addPostData("template_id", "34589u")
.addPostData("page", 1)
.addPostData("form_id", 345)
.addPostData("data", data)
.addPostData("list", list)
.post();
if (res.getResponseCode() == UrlResp.HTTP_OK){
System.out.println(res.getText());
}
}
打印结果
- 保存二进制文件
@Test
public void test() throws IOException {
UrlResp res = Urllib.builder()
.setUrl("https://img-blog.csdnimg.cn/20190324160739729.png")
.addHeader("User-Agent", "Chrome")
.get();
if (res.getResponseCode() == UrlResp.HTTP_OK){
OutputStream os = new BufferedOutputStream(
new FileOutputStream(
new File("photo.png")
)
);
os.write(res.getByteContent());
}
}
输出结果
- 感谢你赐予我前进的力量