Did you know that sometimes those bot protection services will serve Google and other search engines noindex directives? Google’s John Mueller said this on X, saying, “Sometimes there’s bot-protection (or a login, interstitial, etc) triggering that has a noindex on it.”

He also added that it would be better for them to serve 503 server status codes. “Better would be to use 503 for server-side blocks like bot-protection,” he wrote.

In the end, the person he replied to was having a different issue causing the noindex, it was not bot protection. He said it was “some shenanigans with NextJS doing client side rendering instead of server rendering.” “The HTML had a noindex tag, which was removed after rendering on the browser. The HTML shown on GSC is the one AFTER JavaScript rendering, which is why I couldn’t find it and thought it was a GSC bug,” he added.

This was the error he was seeing:

Here are those posts:

wtf is going on here? why is Google not indexing this one?

They say there’s a “noindex” tag, but there isn’t one. robots.txt looks good too.

Anyone? #seo help pic.twitter.com/DYzJsLQ4Yk

— Guilherme (@goenning) December 26, 2023

Thanks John, eventually found it thanks to @HamilcDev

Some shenanigans with NextJS doing client side rendering instead of server rendering

The HTML had a noindex tag, which was removed after rendering on the browser. The HTML shown on GSC is the one AFTER JavaScript rendering,… https://t.co/NffU7suVLM

— Guilherme (@goenning) December 26, 2023

Forum discussion at X.