Extracting Product Information from Merchant Sites

Inktomi’s shopping assistant product required assembling a large amount of consumer product data by parsing thousands of Merchant web sites.

Telesoft wrote the web SPIDERS which crawled the web at night, extracting and cataloging the product data.

Telesoft team wrote these SPIDERS at a breathtaking pace, helping Inktomi market this feature ahead of the competition. This product helped popularize the Internet as a shopping medium, when the consumer awareness of E-Commerce was just taking off.

We wrote the drivers in Java 1.1. The drivers took anywhere from a few seconds to half a minute to index a merchant site depending on the connections speed, Web site response time, and the complexity of the web site.