.Google's John Mueller answered an inquiry about why Google.com marks webpages that are actually disallowed coming from creeping by robots.txt as well as why the it's safe to overlook the associated Explore Console reports regarding those crawls.Crawler Web Traffic To Concern Specification URLs.The person inquiring the inquiry documented that crawlers were actually generating hyperlinks to non-existent concern specification URLs (? q= xyz) to web pages along with noindex meta tags that are actually also blocked out in robots.txt. What motivated the inquiry is actually that Google is crawling the hyperlinks to those web pages, acquiring blocked by robots.txt (without watching a noindex robots meta tag) then getting turned up in Google.com Look Console as "Indexed, though obstructed through robots.txt.".The individual asked the observing inquiry:." However right here is actually the huge inquiry: why would Google index web pages when they can't also find the material? What's the perk during that?".Google's John Mueller confirmed that if they can't crawl the page they can't view the noindex meta tag. He likewise makes an intriguing mention of the web site: hunt operator, advising to dismiss the outcomes due to the fact that the "ordinary" consumers won't view those end results.He wrote:." Yes, you are actually correct: if our experts can't crawl the webpage, our company can't find the noindex. That stated, if our team can't crawl the pages, after that there is actually certainly not a whole lot for our company to mark. Therefore while you might find some of those pages with a targeted site:- question, the normal customer won't observe them, so I definitely would not bother it. Noindex is actually likewise great (without robots.txt disallow), it just means the URLs are going to find yourself being crawled (as well as wind up in the Browse Console document for crawled/not indexed-- neither of these standings create issues to the remainder of the internet site). The important part is actually that you don't make them crawlable + indexable.".Takeaways:.1. Mueller's response affirms the limitations in operation the Site: hunt evolved search operator for diagnostic reasons. Some of those factors is actually because it's not linked to the routine search mark, it's a different thing entirely.Google's John Mueller commented on the website search operator in 2021:." The brief answer is that an internet site: query is not indicated to become full, neither made use of for diagnostics reasons.A website concern is actually a details kind of search that limits the end results to a certain site. It is actually essentially only the word web site, a colon, and after that the website's domain.This concern confines the outcomes to a details site. It's not indicated to become an extensive compilation of all the web pages from that web site.".2. Noindex tag without using a robots.txt is actually fine for these type of situations where a robot is actually connecting to non-existent pages that are obtaining discovered through Googlebot.3. Links along with the noindex tag will certainly create a "crawled/not indexed" entry in Search Console and also those will not possess a damaging effect on the rest of the internet site.Check out the question and address on LinkedIn:.Why will Google index web pages when they can't even observe the information?Included Image through Shutterstock/Krakenimages. com.