In a recent Search Off the Record podcast episode about crawl budget, Dave Smart asked how a new site would get crawled if no one knew about the content. Gary Illyes replied that natural, outbound links are the answer. 

To gain more context of the discussion, Illyes stated that the more sought-after the content is, the more the site gets crawled by Googlebot. Specifically, he said,

“To get more URLs, that’s basically just how sought after your content is in search, because if we see that people are looking for your content and they are linking to your content, then that will naturally increase how much search wants to crawl from your site.”

To which Smart countered with:

“If you’ve got a brand new site you’ve just launched to the world and your marvelous content is brilliant, but nobody knows about it. So how would you know to schedule to crawl that if there’s not those signals out there that people can find you and then create links or click on your search results?”

After a quip about buying links (which is against Google’s guidelines), Illyes states that the best way for a site to be discovered is by acquiring links through social media or natural link building tactics. 

“I would probably do some like social media promotion or whatever. Basically just let people know that I have this site and they should link to me because links are actually really important for discovery. So the more links we see from a natural source of course, like not bot links and crypto links and whatever…”

He concludes his point with:

“Anyway, but yeah, like, I would just probably somehow try to get links in some natural way.”

There has been a long-standing debate among SEO experts about the importance of link building. Some consider this practice to be too spammy and often resort to buying cheap links to increase their online visibility. However, it’s worth noting that paid links (without a “sponsored” tag) are against Google’s guidelines. 

Moreover, Google has recently announced that they plan to reduce spammy results by 40% with their March 2024 spam update. This update is aimed at minimizing spam content and spammy links. 

That being said, Google has emphasized the importance of natural outbound links in helping their Googlebot crawler discover new sites.

Illyes explains:

“Natural links will tell us that this link exists. This important person, Lizzi, talked about this topic, plus, gives us some context about what we might find on the other side of the gate.”

One aspect that cannot be overstated is the natural characteristics of these links. While Illyes does not define what “natural links” mean, we can glean from context and Google documentation that natural can mean relevant to a search query. 

Over the years, Google has emphasized the importance of relevance in content and links, stating that relevant links help improve user experience. 

According to Illyes, natural outbound links help give context to web content and display topical authority. 

To clarify further, Illyes gives an example of how outbound links can help encourage crawling from Google:

“When we push a new piece of documentation, then Barry Schwart’s from SEOroundtable.com will write about us very soon. A couple other sites will write about us very soon after we publish the documentation, and they will usually link because hopefully they trust the resource that they are linking to. And that’s a very good sign for search engines that they should crawl the particular page that they are linking to.

In conclusion, natural outbound links are crucial for the Googlebot crawler to discover sites and display topical authority. As an SEO, you can encourage the discovery of your site by acquiring links through social media or natural link building tactics. Remember to focus on relevance when building links and prioritize quality over quantity. By doing so, you can help improve your site’s crawl rate and ultimately enhance your overall search visibility.