A Web Crawler is a small, single user web application, which works like a spider. Web crawler finds information from the web. In 1994 Brian Pinkerton, a student from University of Washington, built a web interface to his web crawler programme. Released on April 20, 1994 it contained databases from approximately 6000 web servers.

Web crawler was a unique discovery of the era, as it was the first web robot with the ability to index every word on the internet. As per the technical definition “Web Crawler is an automated programme or script that scans methodically or crawls through html contents to look for an index of data. This concept of scanning the web contents is called Web Crawling”. In other terms, it is also called as ants, automatic indexers, bots, web robots, or web spiders.

Web Crawler uses a concept of data mining which is nothing but an extraction of related or meaningful data from large data sets. Data mining can be of many types, but mostly, text mining and web mining is done for knowledge discovery.

Real time usage of crawler in an enterprise is seen in many platforms. In social media, considering twitter which is much vastly used for services starting like brand monitoring, consumer pattern research, etc., this crawler helps us to collect data from twitter in real time. A crawler works on keyword basis, i.e. suppose if user A provides crawler with list of keywords, crawler looks for the tweets that have the keywords provided by user A. The tweets related to the keywords are later modified into a structured format with related information. Using this crawling technique, further filtering of data from large data sets becomes easy. Logical combinations can be applied with your keyword, search and analysis of data becomes easier. Hence it saves time and cost for the enterprise.

Recently in the news I heard about the mobile app called ‘Ticket Jugad’. It solves many issues which people face daily during booking of their tickets or during the time of finding alternative routes to their destination. This app aims to provide availability of confirmed ticket in day to day travelling. Here, web crawler crawls through the database of the railways search for the unreserved number of seats and produces results on the application’s dashboard. It reduces the time required to process users’ queries for various train routes.