How search engines work

Crawling, indexing and ranking in plain English, and what each stage means for a small business website.

3 min read · updated 24 September 2026

Before a page can rank, a search engine has to find it, read it and decide it deserves a place in its index. Most ranking problems on small business sites are really problems at one of those three stages, so it pays to know what each one does.

Crawling: finding pages

Search engines run programs called crawlers (Google's is Googlebot, Microsoft's is Bingbot) that fetch pages and follow the links on them to find more. A crawler discovers your pages in three main ways:

• Links from other pages on your site. Your navigation and internal links are the crawler's map.
• Links from other websites. A link from a directory, supplier or local news site leads the crawler to you.
• Your XML sitemap. A file, usually at /sitemap.xml, that lists the pages you want found.

A page nothing links to and no sitemap lists is called an orphan page. It may never be crawled at all.

You control what crawlers may fetch with robots.txt, a small text file at the root of the site. It is a request, not a lock: well-behaved crawlers obey it, others do not. A single wrong line in it can stop a whole site being crawled, which is why the AI crawler checker also tells you whether Google and Bing are blocked.

Indexing: understanding pages

After fetching a page, the search engine works out what it is about and whether to store it. It reads the title, headings, text, images' alt text and structured data, and it decides which version of a page is the main one when several addresses show the same content (the canonical).

A page can be crawled and still left out of the index. Common reasons:

• It carries a noindex tag, often left over from when the site was being built.
• It is a near-copy of another page, so only one version is kept.
• It has very little useful content.
• It returns an error, or redirects somewhere else.

Ranking: ordering results

When someone searches, the engine picks the pages from its index that best answer that query, for that person, in that place, at that moment. Nobody outside Google knows the exact formula, but the broad ingredients are well established:

• Relevance. Does the page clearly cover what was asked? This is where keyword research and on-page SEO come in.
• Quality and trust. Is it written by someone who knows the subject, and do other reputable sites link to it? See links and authority.
• Experience. Does the page load quickly, work on a phone and avoid intrusive pop-ups?
• Location. For searches like "plumber" or "dentist near me", distance and a consistent local presence matter a great deal. See local SEO.

Where AI answers fit

Search results increasingly include AI-written answers, and many people now ask ChatGPT, Claude or Perplexity directly. Those assistants still depend on crawling and on the open web: they fetch pages to answer questions and cite the sources they trust. The same foundations apply, with a few extra points covered in AI search.

What to do next

1. Check that your site can be crawled: run a site audit or the AI crawler checker.
2. Make sure every page you care about is linked from somewhere and listed in your sitemap.
3. Look for accidental noindex tags on important pages; the SEO page checker flags them.