A quick look at some of the web technologies
web crawling : start with a list of URLs, visit them according to policy, find more and add them to the list.