Spider/Crawler/Bot

An automated programme that visits or “crawls” web sites to collect information about them in order to understand their function and relevance.