If your business requires getting information or some specific data from the Web fast, without manual efforts and precisely what you wish to get, the best solution is to use web mining software, customized for your individual needs. There are so called web spiders or crawlers.
Web spider is a software agent that scans the Web in a methodical, automated manner, searches and extracts from web pages, static or dynamic, the data you need, found by specific criteria. Web spider is a key component of any search engine. It crawls web pages by links they include, passing from one to another and caching the copies of web pages into database. Then, when people search the pages in search engine, on key words, they receive the copies of pages, cached in database. The same way spider can find and extract other types of data, not only web pages copies. It depends on criteria by which spider searches and elicits the information or data. So, this “intelligent” agent can be “trained” to find and deliver accurately what you ask. Spider software can be programmed the way you need, by changing the configurable parameters in code. Then data collected are delivered into database, in format needed. Web spider can extract pages URLs, titles and meta data, email addresses, images, ads, articles, news, data from catalogs, and other content from web pages. So, web spider program lets automate specific web data search and collection process.
Previous Topic
