.Google's Gary Illyes exposes Googlebot picks up hyperlinks instead of "complying with" all of them, negating main documentation.Googlebot collects links instead of following them in real-time.Gary Illyes discussed this relevant information in Google's 'Browse Off The Record' podcast.Google.com's documents contradicts the genuine spider habits.