.Google's John Mueller addressed a question about why Google.com marks pages that are actually refused from crawling through robots.txt as well as why the it is actually risk-free to overlook the similar Browse Console documents regarding those creeps.Robot Traffic To Query Guideline URLs.The person asking the inquiry recorded that crawlers were creating links to non-existent concern parameter URLs (? q= xyz) to pages with noindex meta tags that are also shut out in robots.txt. What cued the inquiry is that Google is crawling the links to those web pages, receiving blocked through robots.txt (without envisioning a noindex robots meta tag) then obtaining turned up in Google.com Explore Console as "Indexed, though blocked through robots.txt.".The individual asked the complying with question:." Yet below's the huge inquiry: why would Google mark web pages when they can not also observe the material? What is actually the benefit during that?".Google.com's John Mueller confirmed that if they can't creep the page they can't view the noindex meta tag. He likewise creates an appealing reference of the site: hunt operator, encouraging to disregard the results because the "typical" individuals will not view those outcomes.He composed:." Yes, you're right: if we can't crawl the web page, our company can not see the noindex. That pointed out, if our experts can not creep the web pages, at that point there's certainly not a whole lot for our team to mark. Therefore while you might observe a number of those webpages with a targeted site:- concern, the average consumer won't see them, so I would not bother it. Noindex is additionally alright (without robots.txt disallow), it only indicates the Links will certainly wind up being actually crept (as well as end up in the Browse Console report for crawled/not catalogued-- neither of these statuses induce problems to the rest of the internet site). The essential part is actually that you do not make all of them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the constraints in operation the Internet site: search evolved hunt operator for diagnostic factors. Some of those explanations is actually because it is actually not hooked up to the frequent search mark, it's a different thing entirely.Google's John Mueller commented on the web site hunt driver in 2021:." The brief response is that a site: inquiry is actually not implied to become comprehensive, nor used for diagnostics reasons.An internet site query is a details sort of hunt that restricts the results to a specific web site. It is actually essentially simply words site, a colon, and after that the internet site's domain name.This concern limits the outcomes to a particular site. It's certainly not meant to be a thorough selection of all the webpages from that site.".2. Noindex tag without using a robots.txt is actually alright for these sort of conditions where a bot is actually connecting to non-existent pages that are actually acquiring found out by Googlebot.3. Links with the noindex tag are going to create a "crawled/not listed" item in Browse Console and also those won't possess an unfavorable impact on the remainder of the web site.Check out the inquiry as well as answer on LinkedIn:.Why will Google mark webpages when they can not even see the content?Featured Photo through Shutterstock/Krakenimages. com.