Curl HTTP Client

Frequently, in my daytime job I have to fetch data from various url’s, either by sending get or post request, binding to different IP address etc. Long time I used my own socket based HTTP class, although I wasn’t quite happy with perfomances and various other things with it. I already used curl cli tool (mostly for debugging purposes), but didn’t really liked it’s php api, so I’ve decided to take some spare time and make some kind of oop wrapper for it, which should be easier to use for easy stuff like sending get/post request etc.

I use this class several months since then, and it evolved over time whenever I needed some new feature. Since various of my colleagues found it very usefull (some of them even sent me new methods for it), I’ve decided to put it out for public. Recently I submitted a code to phpclasses.org, and today got confirmation that class is officially approved.

Update 03/01/2007
New version 1.1 released with new features: fetch into file, upload, proxy etc.

Update 15/02/2008
New version 1.2 released with few bug fixes. New features are ability to send post string as string argument in send_post_data method, ability to accept gzipped content, close curl session etc.

Update 25/03/2013
Version 2.0 released. Pretty much the same features as old versions, but with updated code in order to match latest PHP changes. Code will be regularly updated on github.

You can download class and example files directly from gitgub https://github.com/dinke/curl_http_client

Here are few usage examples.

<?php
/**
 * @version 2.0
 * @package dinke.net
 * @copyright © 2013 Lampix.net
 * @author Dragan Dinic 
 */

require_once("curl_http_client.php");

$curl = new Curl_HTTP_Client();

//pretend to be Firefox 19.0 on Mac
$useragent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:19.0) Gecko/20100101 Firefox/19.0";
$curl->set_user_agent($useragent);

//uncomment next two lines if you want to automatically manage cookies
//which will store them to file and send them on each http request
//$cookies_file = "/tmp/cookies.txt";
//$curl->store_cookies($cookies_file);

//Uncomment next line if you want to set credentials for basic http_auth
//$curl->set_credentials($username, $password);

//Uncomment next line if you want to set specific referrer
//$curl->set_referer('http://www.foo.com/referer_url/');

//if you want to send some post data
//form post data array like this one
$post_data = array('login' => 'pera', 'password' => 'joe', 'other_foo_field' => 'foo_value');
//or like a string: $post_data = 'login=pera&password=joe&other_foo_field=foo_value';

//and send request to http://www.foo.com/login.php. Result page is stored in $html_data string
$html_data = $curl->send_post_data("http://www.foo.com/login.php", $post_data);

//You can also fetch data from somewhere using get method!
//Fetch html from url
$html_data = $curl->fetch_url("http://www.foo.com/foobar.php?login=pera&password=joe&other_foo_field=foo_value");

//if you have more than one IP on your server,
//you can also bind to specific IP address like ...
//$bind_ip = "192.168.0.1";
//$curl->fetch_url("http://www.foo.com/login.php", $bind_ip);
//$html_data = $curl->send_post_data("http://www.foo.com/login.php", $post_data, $bind_ip);

//and there are many other things you can do like

//use proxy
//$curl->set_proxy('http://www.proxyurl.com');

//get http response code for last request
//$http_code = $curl->get_http_response_code();

//get last http request duration in sec
//$duration = $curl->get_request_duration();
?>

Comments

  1. Indu
    September 6th, 2006 | 9:34 am

    Hello,

    I have a requirement where i need to fetch an XMl file from an http request along with some cookies.I used this class and i could save that XML. But after that i require some requests along with the saved cookies and it should be redirected to some html page. How can this be possible ?

    How can i pass the cookies again in the coming requests. Also how can i redirect the data i get to a browser.

    Thanks

  2. September 6th, 2006 | 9:59 am

    I guess use something like this:

    require_once(“curl_http_client.php”);

    $curl = &new Curl_HTTP_Client();

    //pretend to be IE6 on windows
    $useragent = “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)”;
    $curl->set_user_agent($useragent);

    //cookies will be saved in $cookies_file
    //change to other location
    $cookies_file = “/tmp/cookies.txt”;
    $curl->store_cookies($cookies_file);

    //Uncomment next line if you want to set credentials
    //$curl->set_credentials($username, $password);

    //Uncomment next line if you want to set specific referrer
    //$curl->set_referrer(“http://my.referrer.url”);

    $xml_data = $curl->fetch_url(“http://www.foo.com/foo.xml”);

    //parse xml or whatever
    //don’t worry, cookies are saved in $cookies_file location
    //and will be sent on each requests automaticly

    // if page contains http redirection, you will be redirected automaticly
    // if page contains js redirection – you will have
    //to parse js and redirect manually
    $new_html_data = $curl->fetch_url(“http://www.foo.com/foo.html”);

  3. stefan
    December 7th, 2006 | 2:29 am

    i want to send post-data with serial from my domain to external server to connect a database server, with
    “select serial, domain from server where user = test” for example. what can i do to, to send external server a return call back example : “serial ok”

  4. Cor
    January 4th, 2007 | 5:03 pm

    Thanks.
    I have tried the whole day with different approaches and couldn’t get it right. This was my last try – and it worked. I just trief to automatically log into my WP Blog and change the settings programmatically.
    Thanks for your effort – and providing something that works.
    Cora

  5. August 11th, 2007 | 12:11 am

    This is cool.
    I’m going to try it out.

  6. haiza
    September 22nd, 2007 | 4:48 am

    Your code works great in my localhost, but it doesnt work on my server…

    it’s like taking so long to connect and it didnt return any response…

    what could be the problem?
    is is the server settings?

  7. September 22nd, 2007 | 9:37 am

    It could be that your server doesn’t have curl support? Did you get any error message (turn display_error On to make sure they are displayed). Also, perhaps it is timing out, so make sure timeout param in curl requests is sufficient.

  8. Steve Calderwood
    December 30th, 2007 | 2:44 am

    In the next release, you should include a function that closes the curl session, which allows for, among other things, the cookie file to be unlink-able. This is what I added to your curl_http_client class:

    function closeCurl() {
    curl_close($this->ch);
    }

  9. January 25th, 2008 | 1:09 pm

    I found an error with cookie handling, when you call obj->store_cookies you only set the COOKIEJAR and not COOKIEFILE so it only stores cookies and never sends them. Took me a while to figure out as I thought I was doing something wrong. This is a great class and I was hoping to help people out in the future this should be

    function store_cookies($cookie_file)
    {
    // use cookies on each request (cookies stored in $cookie_file)
    curl_setopt ($this->ch, CURLOPT_COOKIEJAR, $cookie_file);
    curl_setopt ($this->ch, CURLOPT_COOKIEFILE, $cookie_file);

    }

  10. January 25th, 2008 | 1:11 pm

    I found an error with cookie handling, when you call obj->store_cookies you only set the COOKIEJAR and not COOKIEFILE so it only stores cookies and never sends them. Took me a while to figure out as I thought I was doing something wrong. This is a great class and I was hoping to help people out in the future this should be

    function store_cookies($cookie_file)
    {
    // use cookies on each request (cookies stored in $cookie_file)
    curl_setopt ($this->ch, CURLOPT_COOKIEJAR, $cookie_file);
    curl_setopt ($this->ch, CURLOPT_COOKIEFILE, $cookie_file);

    }

    Before I added this line it would only send cookies on subsequent requests. But after requesting a page then submitting a post to the server it wrote out a new cookie file with a new session cookie.

  11. February 15th, 2008 | 8:19 pm

    ok, uploaded new version with updates, thanks to your suggestions and submitted bugs.

  12. February 29th, 2008 | 2:41 pm

    I just downloaded your file and i will check.

    Thanks!

  13. farzaneh
    March 4th, 2008 | 10:48 am

    I have problem
    I want to copy 2 image file from one server to another server,but I dont know
    PLZ help me for solution this problem

  14. Chris
    April 9th, 2008 | 8:38 pm

    Found an issue when sending options in the send_post_data function. Wasn’t creating the necessary string with all the options.

    This fixed it for me.

    foreach($postdata as $key=>$value)
    {
    if(is_array($value)){
    // THIS IS IMPORTANT IF YOU ARE SENDING options
    foreach($value as $k=>$v){
    $post_array[] = urlencode($key) . “=” . urlencode($v);
    }

    }else{
    $post_array[] = urlencode($key) . “=” . urlencode($value);
    }
    }

  15. April 24th, 2008 | 6:10 pm

    Hello!
    Great work!
    I have a question, why do I get an error
    “Occured in Curl Error number: 22 Error message: The requested URL returned error: 401″ when I try to fetch an URL. But when I copy paste it in my browser, it works…

  16. April 24th, 2008 | 8:48 pm

    @JCAVARD
    You ‘ve probably forgot to put ‘http://’ part. In browser it works without it, but not in curl :)

  17. Alberto
    July 28th, 2008 | 10:26 am

    Hi!! I’ve been using this class for some time without problems and suddenly it’s not working any more. I’ve found out my ISP has switched to Plesk control panel and PHP version plus some security settings have been changed.

    The error I get is this one:

    Warning: curl_setopt() [function.curl-setopt]: CURLOPT_FOLLOWLOCATION cannot be activated when in safe_mode or an open_basedir is set in /var/www/vhosts/xxxx.xxx/httpdocs/xxx/curl_http_client.php on line 74

    How can this be fixed??

    Thanks for your great work!!

    Regards,
    Alberto

  18. July 28th, 2008 | 5:06 pm

    @Alberto
    Well, the best way to resolve problem is to ask your ISP to remove safe_mode/open_basedir restriction. You can see more info about those restrictions at http://www.php.net/safe_mode

    As an workaround you can simple comment line in curl_http_client.php which set curl to follow location header:

    curl_setopt($this->ch, CURLOPT_FOLLOWLOCATION, true);

    but in that case any location header sent by server when calling fetch_url/send_post_data methods would be ignored, so class my not work correctly in case such headers are sent in http response.

  19. Kashif
    August 12th, 2008 | 11:03 pm

    Hi,

    i found a Script http://www.QuickMiniSite.com, its URL Fopen Based and Free Hosting Not Allow URL Fopen Enable, but some free hsoting allow cURL, Can you convert this little PHP Script into cURL Based.

    http://www.quickminisite.com/download.php

    Please help me , I really need this Script in cURL Based.

    Waiting for your Reply.

    Thanks and Warm Regards
    Kashif

  20. subzero
    August 28th, 2008 | 4:10 am

    Hi

    I thnik it is a great class!, thanks, but I’m having a little trouble…after requesting a page (and getting a cookie) then submitting a post to the server sometimes I get the correct information…. other times I get the error web page, any idea what is happening??

    thanks in advance

  21. August 28th, 2008 | 4:25 pm

    Sorry but my crystall ball doesn’t work these days. Perhaps you should give me more infos, ie. your code etc. :)

  22. hexbase
    October 7th, 2008 | 9:06 pm

    Hey,

    How do i put extra headers when posting some data, like

    X-Forwarded, etc.

    Thanks

  23. October 14th, 2008 | 6:45 pm

    Right now you can’t do that by using class, but implementing additional method shouldn’t be a problem.

    Will add it for to-do list and implement it when I get some time.

  24. October 17th, 2008 | 6:19 pm

    you are really great..its a superb classs for curl….I worked with another class and I lost 2 days….using your class I got output in 1 hour…thanks..thanks a lot…keep going..:)

  25. July 3rd, 2009 | 1:59 pm

    [...] Curl HTTP Client — простая, но эффективная ООП-оболочка над библиотекой Curl. Содержит все методы, необходимые для получения данных со страниц, подмены referrer, авторизации, отправки POST-запросов, управления cookies и т.д. [...]

  26. December 21st, 2009 | 11:23 pm

    [...] new version from here. For detailed list of features with some usefull examples please visit this blog entry. February 15th 2008 Posted to [...]

  27. March 25th, 2013 | 12:43 am

    [...] been a while since I last updated my Curl HTTP Client class. That’s the class that we’ve been using for years now, for all kinds of site [...]

  28. Carlos Viana
    April 23rd, 2013 | 4:49 pm

    Hi,
    please, how can I retrieve AJAX generated data from a site? The site in particular is this one (in Portuguese):

    http://www.citius.mj.pt/Portal/execucoes/ListaPublicaExecucoes.aspx

    If I fill “Nº de Processo:” with “16331/11.1YYLSB” and select RadioBox “Todos”, after pressing “Pesquisar”, I get below the form, a report, but I have no idea how to get the data.
    Thank you

  29. April 23rd, 2013 | 5:13 pm

    @Carlos Viana
    The best way to track this is by installing some browser extension, personally I use tamper data for firefox. When you click to “Pesquisar” it sends post request with plenty of data that you’d have to manually collect from page before you do actual submission (like __VIEWSTATE etc.).

    I’d write some tutorial about site scrapping once, it’s beyond subject of one blog post ;)

Leave a reply