Caught in a Web

31.08.2006

Curl HTTP Client

Frequently, in my daytime job I have to fetch data from various url’s, either by sending get or post request, binding to different IP address etc. Long time I used my own socket based HTTP class, although I wasn’t quite happy with perfomances and various other things with it. I already used curl cli tool (mostly for debugging purposes), but didn’t really liked it’s php api, so I’ve decided to take some spare time and make some kind of oop wrapper for it, which should be easier to use for easy stuff like sending get/post request etc.

I use this class several months since then, and it evolved over time whenever I needed some new feature. Since various of my colleagues found it very usefull (some of them even sent me new methods for it), I’ve decided to put it out for public. Recently I submitted a code to phpclasses.org, and today got confirmation that class is officially approved.

Update 03/01/2007
New version 1.1 released with new features: fetch into file, upload, proxy etc.

Update 15/02/2008
New version 1.2 released with few bug fixes. New features are ability to send post string as string argument in send_post_data method, ability to accept gzipped content, close curl session etc.
You can download class and example files from here.

Here are few usage examples.

<?php
/**
 * @version $Id$
 * @package dinke.net
 * @copyright © 2005 Dinke.net
 * @author Dragan Dinic
 */

require_once("curl_http_client.php");

$curl = &new Curl_HTTP_Client();

//pretend to be IE6 on windows
$useragent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)";
$curl->set_user_agent($useragent);

//uncomment next two lines if you want to manage cookies
//$cookies_file = "/tmp/cookies.txt";
//$curl->store_cookies($cookies_file);

//Uncomment next line if you want to set credentials
//$curl->set_credentials($username, $password);

//Uncomment next line if you want to set specific referrer
//$curl->set_referrer("http://my.referrer.url");

//if you want to send some post data
//form post data array like this one
$post_data = array('login' => 'pera', 'password' => 'joe', 'other_foo_field' => 'foo_value');
//and send request to http://www.foo.com/login.php. Result page is stored in $html_data string
$html_data = $curl->send_post_data("http://www.foo.com/login.php", $post_data);

//You can also fetch data from somewhere using get method!
//Fetch html from url
$html_data = $curl->fetch_url("http://www.foo.com/foobar.php?login=pera&password=joe&other_foo_field=foo_value");

//if you have more than one IP on your server,
//you can also bind to specific IP address like ...
//$bind_ip = "192.168.0.1";
//$curl->fetch_url("http://www.foo.com/login.php", $bind_ip);
//$html_data = $curl->send_post_data("http://www.foo.com/login.php", $post_data, $bind_ip);
?>

August 31st 2006 Posted to Programming, Software

Comments

Indu

September 6th, 2006 | 9:34

Hello,

I have a requirement where i need to fetch an XMl file from an http request along with some cookies.I used this class and i could save that XML. But after that i require some requests along with the saved cookies and it should be redirected to some html page. How can this be possible ?

How can i pass the cookies again in the coming requests. Also how can i redirect the data i get to a browser.

Thanks
dinke

September 6th, 2006 | 9:59

I guess use something like this:

require_once(”curl_http_client.php”);

$curl = &new Curl_HTTP_Client();

//pretend to be IE6 on windows
$useragent = “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)”;
$curl->set_user_agent($useragent);

//cookies will be saved in $cookies_file
//change to other location
$cookies_file = “/tmp/cookies.txt”;
$curl->store_cookies($cookies_file);

//Uncomment next line if you want to set credentials
//$curl->set_credentials($username, $password);

//Uncomment next line if you want to set specific referrer
//$curl->set_referrer(”http://my.referrer.url”);

$xml_data = $curl->fetch_url(”http://www.foo.com/foo.xml”);

//parse xml or whatever
//don’t worry, cookies are saved in $cookies_file location
//and will be sent on each requests automaticly

// if page contains http redirection, you will be redirected automaticly
// if page contains js redirection - you will have
//to parse js and redirect manually
$new_html_data = $curl->fetch_url(”http://www.foo.com/foo.html”);
stefan

December 7th, 2006 | 2:29

i want to send post-data with serial from my domain to external server to connect a database server, with
“select serial, domain from server where user = test” for example. what can i do to, to send external server a return call back example : “serial ok”
Cor

January 4th, 2007 | 17:03

Thanks.
I have tried the whole day with different approaches and couldn’t get it right. This was my last try - and it worked. I just trief to automatically log into my WP Blog and change the settings programmatically.
Thanks for your effort - and providing something that works.
Cora
Howto Monster

August 11th, 2007 | 0:11

This is cool.
I’m going to try it out.
haiza

September 22nd, 2007 | 4:48

Your code works great in my localhost, but it doesnt work on my server…

it’s like taking so long to connect and it didnt return any response…

what could be the problem?
is is the server settings?
dinke

September 22nd, 2007 | 9:37

It could be that your server doesn’t have curl support? Did you get any error message (turn display_error On to make sure they are displayed). Also, perhaps it is timing out, so make sure timeout param in curl requests is sufficient.
Steve Calderwood

December 30th, 2007 | 2:44

In the next release, you should include a function that closes the curl session, which allows for, among other things, the cookie file to be unlink-able. This is what I added to your curl_http_client class:

function closeCurl() {
curl_close($this->ch);
}
Doug

January 25th, 2008 | 13:09

I found an error with cookie handling, when you call obj->store_cookies you only set the COOKIEJAR and not COOKIEFILE so it only stores cookies and never sends them. Took me a while to figure out as I thought I was doing something wrong. This is a great class and I was hoping to help people out in the future this should be

function store_cookies($cookie_file)
{
// use cookies on each request (cookies stored in $cookie_file)
curl_setopt ($this->ch, CURLOPT_COOKIEJAR, $cookie_file);
curl_setopt ($this->ch, CURLOPT_COOKIEFILE, $cookie_file);

}
Doug

January 25th, 2008 | 13:11

I found an error with cookie handling, when you call obj->store_cookies you only set the COOKIEJAR and not COOKIEFILE so it only stores cookies and never sends them. Took me a while to figure out as I thought I was doing something wrong. This is a great class and I was hoping to help people out in the future this should be

function store_cookies($cookie_file)
{
// use cookies on each request (cookies stored in $cookie_file)
curl_setopt ($this->ch, CURLOPT_COOKIEJAR, $cookie_file);
curl_setopt ($this->ch, CURLOPT_COOKIEFILE, $cookie_file);

}

Before I added this line it would only send cookies on subsequent requests. But after requesting a page then submitting a post to the server it wrote out a new cookie file with a new session cookie.
dinke

February 15th, 2008 | 20:19

ok, uploaded new version with updates, thanks to your suggestions and submitted bugs.
satya prakash

February 29th, 2008 | 14:41

I just downloaded your file and i will check.

Thanks!
farzaneh

March 4th, 2008 | 10:48

I have problem
I want to copy 2 image file from one server to another server,but I dont know
PLZ help me for solution this problem
Chris

April 9th, 2008 | 20:38

Found an issue when sending options in the send_post_data function. Wasn’t creating the necessary string with all the options.

This fixed it for me.

foreach($postdata as $key=>$value)
{
if(is_array($value)){
// THIS IS IMPORTANT IF YOU ARE SENDING options
foreach($value as $k=>$v){
$post_array[] = urlencode($key) . “=” . urlencode($v);
}

}else{
$post_array[] = urlencode($key) . “=” . urlencode($value);
}
}
JCAVARD

April 24th, 2008 | 18:10

Hello!
Great work!
I have a question, why do I get an error
“Occured in Curl Error number: 22 Error message: The requested URL returned error: 401″ when I try to fetch an URL. But when I copy paste it in my browser, it works…
dinke

April 24th, 2008 | 20:48

@JCAVARD
You ‘ve probably forgot to put ‘http://’ part. In browser it works without it, but not in curl :)
Alberto

July 28th, 2008 | 10:26

Hi!! I’ve been using this class for some time without problems and suddenly it’s not working any more. I’ve found out my ISP has switched to Plesk control panel and PHP version plus some security settings have been changed.

The error I get is this one:

Warning: curl_setopt() [function.curl-setopt]: CURLOPT_FOLLOWLOCATION cannot be activated when in safe_mode or an open_basedir is set in /var/www/vhosts/xxxx.xxx/httpdocs/xxx/curl_http_client.php on line 74

How can this be fixed??

Thanks for your great work!!

Regards,
Alberto
dinke

July 28th, 2008 | 17:06

@Alberto
Well, the best way to resolve problem is to ask your ISP to remove safe_mode/open_basedir restriction. You can see more info about those restrictions at http://www.php.net/safe_mode

As an workaround you can simple comment line in curl_http_client.php which set curl to follow location header:

curl_setopt($this->ch, CURLOPT_FOLLOWLOCATION, true);

but in that case any location header sent by server when calling fetch_url/send_post_data methods would be ignored, so class my not work correctly in case such headers are sent in http response.
Kashif

August 12th, 2008 | 23:03

Hi,

i found a Script http://www.QuickMiniSite.com, its URL Fopen Based and Free Hosting Not Allow URL Fopen Enable, but some free hsoting allow cURL, Can you convert this little PHP Script into cURL Based.

http://www.quickminisite.com/download.php

Please help me , I really need this Script in cURL Based.

Waiting for your Reply.

Thanks and Warm Regards
Kashif

Languages

About Dinke

Archives

Categories

Meta

31.08.2006

Curl HTTP Client

Comments

Leave a reply