What is Web Crawler and How Does It Work?

Do you ever wonder what makes the search engines go around?
It’s fascinating, isn’t it?
The way some mechanism can systematically browse the World Wide Web for web indexing or web spidering…

Yes, you are right.
A #webcrawler is precisely what we are talking about!!!
It’s an #internetbot,
also known as #webspider,
#automaticindexer,
#webrobot or simply #crawler…

Guess what else, web crawlers can do?
Don’t be surprised,
By the functions it performs:
1. update their web content or indices of others sites' web content
2. copy all the visited pages for subsequent processing by a search engine which will index the downloaded pages to provide lightning fast searches
3. automate maintenance tasks on a website, such as checking links or validating HTML code…

Do you know the popular open source web crawlers?
Here’s the list:
1. Scrapy
2. Apache Nutch
3. Heritrix
4. HTTack

How the web crawler works :
Enough of the theory, let’s jump right into
How a web crawler works:

1. Select a starting seed URL or URLs
2. Add it to the frontier
3. Now pick the URL from the frontier
4. Fetch the web-page corresponding to that URL
5. Parse that web-page to find new URL links
6. Add all the newly found URLs into the frontier
7. Go to step 3 and reiterate till the frontier is empty


Did you notice what’s happening here?
Wonderful, isn’t it?

It’s amazing how search engines work,
websites update their content or
do their maintenance tasks
But…
What’s more amazing is the way web crawlers make it happen…
Don’t you think so???

for more info about web crawler : http://www.prowebscraper.com/blog/50-...


Follow us on Twitter!
  / prowebscraper