Welcome dumbfound Students!

This is a site I have dedicated to help students like yourself to get around with digesting what you have and shall learn in class about the world of computers and programming and to silently fulfill my fetish for bashing new people in my territory.

Free iPhone 3G S - Well, I got mine!

Build Thy Search: Planning a crawler

Thursday, December 23, 2010



For some time now, I have been planning on building a search engine dedicated to delivering torrent links, megavideo links and direct http and ftp downloads for movie titles. I am a movie addict and am doing this solely for my viewing pleasure, plus the fact that I might get a few dollars by selling this to someone.
So, I have begun to plan on how the crawler shall work. FYI, the crawler takes a list of urls and downloads them to my disk, finds more links in the pages and keeps any downloads found.
I've taken C++ as the choice of language here; I myself don't know why.
The bare functionality:
Fetch Webpage.
Scan for Links.
Keep download links in DB.
Add other links to the queue of URLs to be crawled.

I was thinking of using libcurl but apparently, the libcpp c++ wrapper for libcurl isn't around anymore. Frankly, I would hate to code with the wininet. I still in search for a perfect HTTP library.

0 comments:

The Transformation Challenge

You need to stop taking the computer for granted. It is not a point and click shit that it seems to be. For a month from now you need to stop using GUI based systems in any form or manner. You are only allowed to use CLI, and I persuade you to use FreeBSD.
You'll be a totally different person after the month is over.

  © Blogger templates Newspaper by Ourblogtemplates.com 2008

Back to TOP