注册 登录  
 加关注
   显示下一条  |  关闭
温馨提示!由于新浪微博认证机制调整,您的新浪微博帐号绑定已过期,请重新绑定!立即重新绑定新浪微博》  |  关闭

和申的个人主页

专注于java开发,1985wanggang

 
 
 

日志

 
 

HttpClient多线程并发问题  

2013-12-20 19:47:07|  分类: 爬虫相关 |  标签: |举报 |字号 订阅

  下载LOFTER 我的照片书  |
HttpClient多线程并发问题 - 和申 - 和申的个人主页
使用httpclient的经验:
       1.  创建的MultiThreadedHttpConnectionManager 实例最好是全局的,否则会有多个连接池
       2.  HttpClient可以多个实例。
       3. 处理请求的最后,要调用method.releaseConnection();回收连接,释放资源。
 

static final  MultiThreadedHttpConnectionManager connectionManager =

                 new MultiThreadedHttpConnectionManager();


        public static String getHtml(String url){

String ret = "";

HttpClient client = new HttpClient(connectionManager);

...

// and then from inside some thread executing a method

GetMethod get = new GetMethod(url);

try {

client.executeMethod(get);

// get response

ret = get.getResponseBodyAsString();

} finally {

// be sure the connection is released back to the connection manager

get.releaseConnection();

}

return ret;

        }

 


import org.apache.commons.httpclient.DefaultHttpMethodRetryHandler;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.MultiThreadedHttpConnectionManager;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.commons.httpclient.params.HttpConnectionManagerParams;
import org.apache.commons.httpclient.params.HttpMethodParams;


/**
* 先这样,后续搞到配置中<br>
* 类HttpHelper.java的实现描述:
*/
public class HttpHelper {

protected static final Logger log = Logger .getLogger(HttpHelper.class);
static MultiThreadedHttpConnectionManager mhconnectionpool;

static {
mhconnectionpool = new MultiThreadedHttpConnectionManager();
HttpConnectionManagerParams httpConnManagerParams = mhconnectionpool.getParams();
// max connections per Host
httpConnManagerParams.setDefaultMaxConnectionsPerHost(20);
// max total connections
httpConnManagerParams.setMaxTotalConnections(20);
// connections timeout
httpConnManagerParams.setConnectionTimeout(2000);
// socket timeout
httpConnManagerParams.setSoTimeout(4000);
// retry handler
httpConnManagerParams.setParameter(HttpMethodParams.RETRY_HANDLER, new DefaultHttpMethodRetryHandler(3, false));

}

public static String httpConnect(String url) {
try {
GetMethod getMethod = new GetMethod(url);
HttpClient httpClient = new HttpClient(mhconnectionpool);
int statusCode = httpClient.executeMethod(getMethod);
if (statusCode == HttpStatus.SC_OK) {
String stringBody = getMethod.getResponseBodyAsString();
return stringBody;
}
getMethod.releaseConnection();
} catch (Exception e) {
log.error("execute http error!", e);
}
return null;
}

}




参考: http://fishyych.iteye.com/blog/1056774统计信息唧唧歪歪唧唧网ggyygg.net
  评论这张
 
阅读(1322)| 评论(3)
推荐 转载

历史上的今天

在LOFTER的更多文章

评论

<#--最新日志,群博日志--> <#--推荐日志--> <#--引用记录--> <#--博主推荐--> <#--随机阅读--> <#--首页推荐--> <#--历史上的今天--> <#--被推荐日志--> <#--上一篇,下一篇--> <#-- 热度 --> <#-- 网易新闻广告 --> <#--右边模块结构--> <#--评论模块结构--> <#--引用模块结构--> <#--博主发起的投票-->
 
 
 
 
 
 
 
 
 
 
 
 
 
 

页脚

网易公司版权所有 ©1997-2016