There are potentially 1,000 different answers to this question.
If your website is not listed anywhere on the search engines, something needs fixing pretty damn quick.
If you are certain that your site does not rank anywhere in Google (or other search engines), then the most obvious issue to check is whether the search engines can index your site.
What is indexing?
I make a point of never assuming people know stuff (as assumption is the mother of all f**k ups).
Indexing is when search engine bots visit your website and ‘crawl’ it.
Crawling is when they follow the links on your pages to other site pages.
Each page they crawl is then potentially added to the search engines ‘index, which means that it will show up in searches.
Where your website shows up is down to many factors that I discuss in other posts.
So, if your site does not appear in the search results, it is likely set to no-index.
What is no-index?
There’s a setting in WordPress on the Settings > Reading page that allows you to block search engines from indexing the site. It’s very easy to turn this on and forget about it.
To fix the no-index problem (or at least the first place to check) is this setting – if the box is checked, uncheck it and click the update button.
If you check your site and this checkbox is checked, there’s your problem.
Why does this happen?
When building a site, developers often check this box to stop the in-progress site from getting indexed and then forget to uncheck it when the site goes live.
This is a rookie error, and unfortunately, a very common one, but you have the information to fix it, so get it checked and sorted.
I know what you’re thinking.
How did my website developers make such a rookie mistake? Sh*t does happen, and websites are complex affairs, but the truth is that you we’rent working with an expert.
Now you’ve fixed the issue, engage a professional SEO consultant moving forward.
Book a meeting with DaveWhat else can prevent your website from showing in the search results?
The no-index issue above is the most common cause of sites not getting indexed and showing in the search results, but you need to check other things if that’s not the cause.
Robots.txt
This small file sits on your hosting and can also control bot access to your site.
Sometimes, this file can be set to reject access to search engine bots, so this is the next thing to check.
The easiest way to do this is to install the free YOAST SEO plugin and go into the Tools > Files section, where you can then see the contents of your robots.txt file.
It should look like this:
If it has anything in the disallow bit, it could be causing no-indexing issues, so update it to look like this example.
Now this is a little technical, but it takes less than 10 minutes, so if you have this problem, let me know and I will fix it for you for free.
Note the sitemap reference in this file – this tells the search engines where my sitemap can be found – the search engines then use this to help them index.
No-indexed pages and posts.
The Yoast plugin also allows you to no-index pages and posts on an individual basis.
This can mean that your homepage is set to no-index even if your settings and robots.txt file are not.
These settings are found on the pages and posts themselves.
Two settings to look out for here:
- Allow search engines to show this content in search results
- Should search engines follow links on this content
Both of these should be set to YES. If they are not on your site, then again, the bots will not index the content, and they will not follow the links from this page to other content on your website.
Again, this is easy to miss when a site goes live, especially as the settings are hidden on the page.
Widowed content.
This means pages or posts that have no internal links to them.
If your robots.txt file does not have a site map URL as per the example above, Google cannot find any pages on your site that are not linked to from other content.
To be honest, this issue does not happen often, but it can. So if you’ve worked through the above fixes and there are still issues, particularly with single pages missing from the search results, this can be the culprit.
Sh*t content.
Getting Google to crawl your site is one thing, but when it does and then discovers poorly written or AI-generated content, it might simply decide not to index it, as it does not want lazy content in its search results.
Your content is the most important thing on your website, so if you’ve just used GPT to generate loads of crappy content, don’t be surprised if it does not get indexed.
A recent survey in America discovered that over 80% of people can spot AI-generated copy, and of that 80%, half the people would not trust a website with AI-generated content.
If that’s the case, why would the search engines bother to index it?
Don’t forget that indexing your website costs money!
It costs Google money to index and store site information on their database, so they don’t want crap in there: it’s that simple.
They have to crawl the site, store the data, recrawl and so on, so whilst it might be a seemingly insignificant cost, it quickly adds up across the entire internet.
Bear this in mind when you are planning and writing your site – if you don’t make your site as good as it can possibly be, that may be another reason it’s not getting indexed.
This leads to another potential reason that some of your content might not be indexed.
Crawl Budget.
Google’s bots only have a certain amount of time to crawl your site – they may not crawl the entire thing, so that they may return to index more of it over time.
If your site is on crap hosting and slow to load, these factors all consume crawl budget, so the faster your site, the more Google can crawl in one go.
This is one (of the many) reasons that professional SEO focuses on site speed and technical SEO—a slow site will turn away human visitors and search engine bots.
Dave
Dave is a WordPress SEO expert with over 20 years experience in designing, building and optimising WordPress (and non-WordPress) websites for people who are serious about SEO and want to generate more new business from their websites. He's run a creative agency for 25+ years, employs 17 staff and enjoys a good rant about SEO.
