.Google.com's Gary Illyes uncovers Googlebot collects web links as opposed to "observing" all of them, negating main paperwork.Googlebot gathers hyperlinks rather than observing all of them in real-time.Gary Illyes discussed this info in Google.com's 'Look Off The Document' podcast.Google's paperwork opposes the true crawler habits.