Comments on: Wikipedia Scraper http://www.nickycakes.com/wikipedia-scraper/ Confessions of a Reformed Blackhat Fri, 10 Feb 2012 17:34:28 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Steve Lownds http://www.nickycakes.com/wikipedia-scraper/comment-page-1/#comment-121946 Steve Lownds Mon, 02 May 2011 17:17:04 +0000 http://www.nickycakes.com/wikipedia-scraper/#comment-121946 Oh yeah! Oh well, still a handy little script - ones I have used in the past stopped working a while ago. Oh yeah! Oh well, still a handy little script – ones I have used in the past stopped working a while ago.

]]> By: nickycakes http://www.nickycakes.com/wikipedia-scraper/comment-page-1/#comment-120618 nickycakes Wed, 27 Apr 2011 22:48:13 +0000 http://www.nickycakes.com/wikipedia-scraper/#comment-120618 to be fair, i posted that several years ago to be fair, i posted that several years ago

]]> By: Steve Lownds http://www.nickycakes.com/wikipedia-scraper/comment-page-1/#comment-120530 Steve Lownds Wed, 27 Apr 2011 17:13:57 +0000 http://www.nickycakes.com/wikipedia-scraper/#comment-120530 Thanks for the script - very useful. The script can be a little picky with regard to capital letters - by constructing the exact wikipedia url from the topic you provide, the capitalisation has to be exact or the search will fail. I solved this by pointing my script at the search function instead. This also gets around the problem of topics with more than one word - the wikipedia urls use underscores ( _ ) rather than spaces, so wouldn't return a result. It's easy and just requires one line to be changed: change: $target = "http://en.wikipedia.org/wiki/".urlencode($topic); to: $target="http://en.wikipedia.org/w/index.php?title=Special:Search&search=".urlencode($topic)."&ns0=1&redirs=0"; I also put a character limit in to prevent very small paragraphs being returned: change: if ($paragraph){ to: if ($paragraph AND strlen($paragraph)>80){ Thanks for the script – very useful. The script can be a little picky with regard to capital letters – by constructing the exact wikipedia url from the topic you provide, the capitalisation has to be exact or the search will fail. I solved this by pointing my script at the search function instead. This also gets around the problem of topics with more than one word – the wikipedia urls use underscores ( _ ) rather than spaces, so wouldn’t return a result.

It’s easy and just requires one line to be changed:

change:
$target = “http://en.wikipedia.org/wiki/”.urlencode($topic);

to:
$target=”http://en.wikipedia.org/w/index.php?title=Special:Search&search=”.urlencode($topic).”&ns0=1&redirs=0″;

I also put a character limit in to prevent very small paragraphs being returned:

change:
if ($paragraph){

to:
if ($paragraph AND strlen($paragraph)>80){

]]> By: Daniel http://www.nickycakes.com/wikipedia-scraper/comment-page-1/#comment-25972 Daniel Fri, 23 Apr 2010 22:21:28 +0000 http://www.nickycakes.com/wikipedia-scraper/#comment-25972 Hello! I am using your wonderful script, but only offline, when using it on my live webserver nothing is displayed! Is there any settings or modules I need to install on the webserver in order for it to get working? Please help, I must get this Wikipedia scraping working. Kind regards and thanks for a wonderful article! /Daniel Hello!

I am using your wonderful script, but only offline, when using it on my live webserver nothing is displayed!

Is there any settings or modules I need to install on the webserver in order for it to get working?

Please help, I must get this Wikipedia scraping working.

Kind regards and thanks for a wonderful article!

/Daniel

]]> By: Joe http://www.nickycakes.com/wikipedia-scraper/comment-page-1/#comment-4894 Joe Thu, 15 Jan 2009 19:18:03 +0000 http://www.nickycakes.com/wikipedia-scraper/#comment-4894 Thx for the script - it works great! But how can you limit the amount of data returned? Ie - stop the data returned after about 3 paragraphs, or limit the number of characters returned? Thx for the script – it works great! But how can you limit the amount of data returned? Ie – stop the data returned after about 3 paragraphs, or limit the number of characters returned?

]]> By: digga http://www.nickycakes.com/wikipedia-scraper/comment-page-1/#comment-4166 digga Thu, 04 Sep 2008 02:04:22 +0000 http://www.nickycakes.com/wikipedia-scraper/#comment-4166 u the man yet again ..thnx nickycakes. u the man yet again ..thnx nickycakes.

]]> By: nickycakes http://www.nickycakes.com/wikipedia-scraper/comment-page-1/#comment-3970 nickycakes Mon, 04 Aug 2008 20:47:12 +0000 http://www.nickycakes.com/wikipedia-scraper/#comment-3970 =)) =))

]]> By: steve http://www.nickycakes.com/wikipedia-scraper/comment-page-1/#comment-3969 steve Mon, 04 Aug 2008 19:52:30 +0000 http://www.nickycakes.com/wikipedia-scraper/#comment-3969 sweet. worked. thanks, script works charmingly. sweet. worked. thanks, script works charmingly.

]]> By: nickycakes http://www.nickycakes.com/wikipedia-scraper/comment-page-1/#comment-3964 nickycakes Mon, 04 Aug 2008 15:58:25 +0000 http://www.nickycakes.com/wikipedia-scraper/#comment-3964 that's a matter of encoding. if you're in firefox and go to view->character encoding and select utf8 or whatever, it should work fine. that’s a matter of encoding. if you’re in firefox and go to view->character encoding and select utf8 or whatever, it should work fine.

]]> By: steve http://www.nickycakes.com/wikipedia-scraper/comment-page-1/#comment-3963 steve Mon, 04 Aug 2008 15:34:56 +0000 http://www.nickycakes.com/wikipedia-scraper/#comment-3963 seems like pretty much all the results have a bunch of special chars that don't want to render. any ideas? seems like pretty much all the results have a bunch of special chars that don’t want to render. any ideas?

]]>