Automated HTTP GET (with cookies) in Curl

  • Problem: I have a list of url stored  in a file /tmp/myurls and needs to do a HTTP GET (with cookies) for each url wiithin the file.
  • Solution: Here is how I use curl in a perl script to do a HTTP GET per ul.  At first I create a cookie from the first url, then hit each one with the cookies. Of course there will be more error checking for creating the cookies such as the first url is not working.
  • Impression: Curl serves pretty well with -c to store cookies in a file, -b to use cookies from a file in this case. Python can be used to do the similar thing and I can keep the cookies in memory instead of reading it every time. However Python seems unknown/unpopular to the sys admins here, so it is not even an option. Writing C with libcurl is another option as well.

./hits.pl /tmp/myurls myhost.mydomain.com /tmp/cookies

hits.pl

=================

#!/usr/bin/perl -w

$host=$ARGV[1];
$cookiesFile=$ARGV[2];
open( INPUT, “<$ARGV[0]” );
$init=1;
while( $line = <INPUT> ){
$url=’http://&#8217;.$host.$line;
print $url,”n”;
# only need to get the cookies for the first time
if ($init) {
system “curl -c  $cookiesFile -s -o /dev/null “$url””;
$init=0;
}
system “curl -b  $cookiesFile -s -o /dev/null “$url””;
}
close INPUT;

Advertisements
This entry was posted in curl and tagged . Bookmark the permalink.

5 Responses to Automated HTTP GET (with cookies) in Curl

  1. Asian Chick says:

    Hey. How’s it going?

  2. Victor says:

    Hi cedia,I am reading ur blog and knowing u ‘ve been pretty active.I will fly to Seattle this week. Currently I am spending some spare time writing some code on linked list, tree, string etc. You got the idea what I am up to:-). Maybe in a few months I will live closer to ya.

  3. Asian Chick says:

    Wow. That’s great! Is this permanent? Temporary?

  4. Victor says:

    I am tired after jumped around a few airports. It is going to be perm. I am looking for sth. on west coast, vancouver comes as the first, rest of WA except Seattle is second. I will stick around here if nothing interesting comes up.

  5. Victor says:

    cedia,Seattle is a great city. It has a big farmer’s market which I love it. However parking can be hard to find, the kool part of the city is bus is free at downtown at daytime. There are even buses from Tacoma to downtown with $1. amazing. We have to pay almost $2 US one way for public transit here in Toronto. However the company I interviewed doesn’t have a match for me. I was also surprised with their fame in industry that not everyone working here ‘s kool, esp. not some manager.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s