Comments on: Scraping Websites for Fun and Profit Part 2

By: Dan

Dan — Wed, 29 Jul 2009 19:30:48 +0000

Took a long time to find your site, but this did the trick. The site I was trying to log into required additional data items (form ‘names’) for the data array. I was able to discover the additional fields using the Web Developer toolbar and comparing the results to facebook.

By: malini

malini — Sat, 06 Sep 2008 16:41:38 +0000

hi…I have tried your code and i get this error:
Parse error: parse error, unexpected ‘[‘ in E:\Program Files\EasyPHP 2.0b1\www\http_get.php on line 6

here is my code:

By: nickycakes

nickycakes — Wed, 09 Jul 2008 21:30:40 +0000

yeah gotta have curl enabled =)

By: Chip

Chip — Wed, 09 Jul 2008 21:14:20 +0000

Sorry Dude, got it fixed. It is the php.ini under php5-cli is not setup properly. I have two php.ini, one under apache2 and one under php5-cli. The apache2 one already have the line “extension=curl.so” and mislead me thinking that I have it setting up correctly. I appended the line into the php5-cli php.ini file and is now working.

By: Chip

Chip — Wed, 09 Jul 2008 20:07:34 +0000

Nicky,

Just wonder how you make it work. I brought the book and try to run the first couple of examples. But I got the

Fatal error: Call to undefined function curl_init() in /home/admin/php/LIB_http.
php on line 210

I check the LIB_http.php file, there is a line calling curl_init() but nowhere in the code define it.

Do I missing anything here?

Rgds,
Chip

By: nickycakes

nickycakes — Mon, 28 Jan 2008 17:12:21 +0000

Yeah, I looked closer a little later and found out that it does use regex. However, you can use the str functions in php, which are much quicker.

By: nohatter

nohatter — Mon, 28 Jan 2008 14:02:11 +0000

Nice article, but i don’t get your comment above, since regex is exactly what this library is using. I don’t see any other way to parse. What am i missing here?

“Regular Expression functions in php, in addition to being hard to learn for a newbie, are stupidly inefficient and will slow your programs down, so you don’t want to use them for parsing websites anyway.”

By: nickycakes

nickycakes — Wed, 26 Dec 2007 06:51:09 +0000

yeah it really rocks, i love it

By: Robert Norton

Robert Norton — Mon, 24 Dec 2007 06:13:40 +0000

Dude, I’ve never seen that firefox add-on (the web dev toolbar) before, but it kicks ass. Thanks for the link.

By: Adam

Adam — Sun, 23 Dec 2007 10:16:09 +0000

Very useful stuff Nicky, thanks a lot for sharing!