Indexing is how Google compiles, categorizes and lists websites to provide information to searchers.
According to Google, the index is “like an ever-growing library with billions of books and no central filing system … It contains hundreds of billions of webpages and acts like the index in a back of a book – with an entry for every word seen on every web page [they] index.”
Google uses a program called Googlebot (also known as a robot, bot, or spider) that fetches information and uses an algorithm to determine which sites to crawl, how often and how many pages to fetch from each site to compile the index.
It’s important that your website is properly set up to be indexed so it has a chance to appear in search results. While the Googlebot analyzes factors such as key content tags and attributes such as Title tags and ALT attributes to index your website, other algorithms are used to determine how high to rank your content.
To know if a search engine has indexed your website, insert “site:” in front of your page’s URL to see the results. You can also submit your sitemap directly to a search engine.
Here are a few solutions to common indexing issues:
- Setup Google Search Console. Indexing errors will appear on your dashboard, so you know exactly what to fix.
- Run a fresh sitemap and re-submit it.
- Check for accidental page exclusions in your robots.txt file or for poorly configured htaccess files.
- Ensure that your site contains relevant internal and external links (the number of pages crawled by Google is proportional to your PageRank, so securing backlinks will help).
- Reference Google’s webmaster tools to learn best practices for getting your site crawled