How does a search engine work?
According to Google, search works in three stages:
- Crawling: Googlebot finds pages by following links and downloads their text, images and videos.
- Indexing: Google analyses the text, images and videos on the page and stores this information in its index.
- Serving results: When someone searches, Google shows the pages in its index that are most relevant to the query.
If a page gets stuck at any of these stages, it doesn’t appear in search. Google doesn’t guarantee that it will crawl, index or serve a page, and it doesn’t accept payment to crawl a site more often or rank it higher.
A page that isn’t indexed doesn’t appear in search
A noindex tag added to a page by mistake doesn’t stop it being crawled, but it does stop it being indexed: however good the content, it won’t appear in results.
Why does SEO matter?
If a search engine can’t find, read or understand your site, your content won’t reach the person searching for it. SEO removes these obstacles and makes the page appear in results accurately and in a way that’s worth clicking:
- Findability: pages can be crawled and indexed (robots.txt, noindex, internal links, sitemap).
- Clarity: the page’s topic is stated clearly through headings, text and structured data.
- How it looks in results: Google says the title link is often the primary piece of information people use to decide which result to click.
- Page experience: Google says Core Web Vitals are used by its ranking systems; but a good experience alone doesn’t guarantee a top ranking, because Google always seeks to show the most relevant content.
The title and description drive the click
The same page with two different titles: a generic title like “Home” and a description taken from a cookie notice don’t tell the user what they’ll find. A descriptive title and summary show that the page answers the query.
If the page shifts, users tap the wrong thing
If an ad that loads late pushes the content down, the user taps something they didn’t mean to. The CLS metric in Core Web Vitals measures this shift.
The layers of SEO
Technical SEO
Making pages crawlable and indexable: robots.txt, canonical, redirects, sitemap, HTTP status codes, speed.
Content
Reliable, people-first content that answers the user’s question; a clear heading structure and the right terms.
Page experience
Pages that load fast, don’t shift, respond to taps right away and are easy to use on mobile (Core Web Vitals).
Links and structured data
Google finds pages by following links; structured data tells machines explicitly what the page is about and makes it eligible for rich results.
Is your content in the initial HTML?
Google can run JavaScript across its crawling, rendering and indexing phases; but as Google itself notes, not all bots can run JavaScript, and server-side rendering is still a great idea. Having important content in the initial HTML response is the safest approach for both search engines and AI bots.
What does a bot that doesn’t run JavaScript see?
If content is only built by JavaScript in the browser, the initial HTML response is an empty shell. On a server-rendered page, the text is ready in the first response.
What SEO isn’t
- It isn’t a guarantee: in Google’s words, no one can guarantee a #1 ranking on Google; beware of anyone promising guaranteed rankings or a “special relationship” with Google.
- It isn’t a trick: practices like keyword stuffing, cloaking and buying links violate Google’s spam policies; sites that violate these policies may rank lower or not appear in results at all.
- It isn’t a one-off job: as a site changes, new technical problems appear; regular audits are needed.
Where to start
- Crawl your site with the Technical SEO Audit: find pages that can’t be crawled or indexed, and broken pages.
- Check the title, description and content structure of your most important pages with the On-Page SEO Checker.
- Measure the real user experience with the Core Web Vitals Test.
- See what Googlebot and AI bots can access with the Robots.txt Tester.
Frequently asked questions
How long does SEO take to work?
There’s no fixed timeframe: how long it takes for changes to be crawled, reflected in the index and affect rankings varies from site to site. Google suggests asking up front what results to expect from SEO work and in what timeframe; no one can guarantee a #1 ranking.
What’s the difference between SEO and ads?
Ads are shown for a fee and labelled “sponsored”. You don’t pay to appear in organic results: Google doesn’t accept payment to rank a site higher or crawl it more often.
Is GEO replacing SEO?
No. To be shown as a source in Google’s AI features (AI Overviews, AI Mode), a page must be indexed and eligible to be shown in Google Search with a snippet; Google says there are no additional requirements. For details, see the What is GEO? page.
Sources
- Google · How Search works
- Google · Do you need an SEO?
- Google · Title links
- Google · Page experience
- Google · JavaScript SEO basics
- Google · Spam policies
- Google · AI features and your website
Last updated: