The Burp Suite Spider tool works by parsing through all known HTML content and extracting links to other content on the Web. The linked content is then analyzed for additional linked content that is discovered within it. This process will continue indefinitely and is only limited by the amount of available linked content, the layers of depth specified, and the number of concurrent threads processing additional requests.