Technical SEO: making sure search engines can actually read your site

You can have excellent content, a sensible architecture and genuine authority, and still rank for nothing, because a single line in a configuration file is telling search engines not to look.

Technical SEO is the layer underneath everything else. It rarely wins anything by itself. It quietly prevents everything else from working.

This article covers the technical mechanics. For how pages should relate to one another, see how to build a solid site structure for SEO.

The three things that must happen

Before a page can rank, three things have to work in order.

Crawling. A search engine has to be able to reach the URL.

Rendering. It has to be able to build the page and see the content, including anything JavaScript adds.

Indexing. It has to decide the page is worth storing.

Break any one and the page is invisible. Most “our SEO isn’t working” problems we’re called in for turn out to be a failure at one of these three, not a content problem at all.

Robots.txt

A plain text file at the root of your domain telling crawlers where they may go.

It’s short, it’s powerful, and it’s responsible for some of the most expensive accidents in the industry. Because it’s easy to write:

User-agent: *
Disallow: /

That says “no search engine may crawl anything”. It’s the correct setting for a staging site, and it gets pushed to production more often than anyone likes to admit. If a site’s traffic vanished overnight after a launch, check this first, always.

Two things worth understanding:

It controls crawling, not indexing. A blocked URL can still appear in results if other sites link to it, showing without a description because nothing could be read. To keep a page out of the index, allow the crawl and use a noindex meta tag. Blocking it in robots.txt actually prevents the crawler from seeing the noindex you added.

Don’t block your CSS and JavaScript. Search engines need them to render the page as a user sees it. Blocking them makes your site look broken to the crawler.

XML sitemaps

A list of the URLs you want indexed, with a hint about when each last changed.

A sitemap doesn’t make pages rank. It helps them get discovered, which matters most for large sites, new sites, and pages with few internal links.

Worth getting right:

• Include only canonical, indexable URLs that return a 200. A sitemap listing redirects, 404s or noindex pages is a quality signal, and not a good one.
• Use honest lastmod dates. Marking every page as modified today, every day, trains crawlers to ignore the field.
• Submit it in Search Console and check the coverage report afterwards. That report is where you learn what Google actually did with it, which is frequently not what you assumed.
• Reference it from robots.txt.

Canonical tags

The canonical tag tells search engines which URL is the real one when the same content is reachable at several addresses.

This matters more than most people expect, because sites generate duplicates without anyone deciding to. http and https. With and without www. Trailing slash and without. ?utm_source= tracking parameters on every shared link. Filter and sort parameters on listing pages. Printer-friendly versions.

Each variant can be treated as a separate page, splitting the signals that should be concentrated on one.

Set a self-referencing canonical on every page as a baseline. Pick one preferred format for www and trailing slashes, redirect the alternatives, and be consistent about it everywhere, including in your internal links.

Indexation: check what’s actually indexed

Search Console’s Pages report is the single most useful diagnostic available, and it’s under-read.

It tells you which pages are indexed and, more usefully, why the others aren’t. Common reasons and what they mean:

• Crawled, currently not indexed. Google saw it and decided it wasn’t worth storing. Usually a quality or duplication problem.
• Discovered, currently not indexed. Google knows the URL exists but hasn’t prioritised crawling it. Often a sign of thin internal linking or a very large site.
• Duplicate, Google chose a different canonical. Your canonical was overruled. Worth investigating, because it means Google disagrees with your judgement about which page is primary.
• Excluded by noindex tag. Intentional, hopefully.

Read this report monthly. It’ll tell you about problems long before the traffic graph does.

Page speed and Core Web Vitals

Speed is a ranking factor, and a modest one. It’s a much larger conversion factor, which is the better reason to care.

Three measurements matter:

Largest Contentful Paint is how long until the main content appears. Aim under 2.5 seconds. Usually improved by optimising whichever image or heading is the largest element, and by removing render-blocking requests.

Interaction to Next Paint is how quickly the page responds when someone taps something. Aim under 200 milliseconds. Usually a JavaScript problem.

Cumulative Layout Shift is how much the page jumps around while loading. Aim under 0.1. Almost always caused by images without dimensions, ads, or fonts swapping late.

The reliably effective fixes, in order of return: serve images in modern formats at the size actually displayed and always with width and height set; remove third-party scripts you aren’t using, which after a few years is usually several; self-host fonts rather than pulling them from a third-party origin; and eliminate render-blocking CSS and JavaScript from the initial load.

Measure with real-user data in Search Console rather than a single lab test. Lab tools are useful for diagnosis, but they test one page from one location on one connection.

HTTPS, redirects and status codes

HTTPS everywhere. It’s a ranking signal, browsers flag its absence, and certificates are free.

Redirect chains. A redirecting to B redirecting to C wastes crawl budget and leaks a little signal at each hop. After a migration these accumulate quietly. Audit them.

Correct status codes. A missing page should return 404, not a 200 with “page not found” written on it. The second is a soft 404, and it means search engines keep crawling a page that doesn’t exist. Permanent moves are 301; temporary ones are 302, and using 302 for a permanent move is a common and avoidable error.

Structured data

Schema markup describes what a page is about in a format search engines parse directly. It doesn’t improve rankings by itself, but it enables rich results, which improve click-through rates, which is worth having.

The types most business sites benefit from: LocalBusiness or Organization, Service, Article, FAQPage, and BreadcrumbList.

Mark up what’s genuinely on the page. Marking up things that aren’t there’s a manual action waiting to happen.

How to find what’s broken

A workable order:

1. Search Console first. Coverage, Core Web Vitals, mobile usability, manual actions. It’s Google telling you directly.
2. A crawler such as Screaming Frog. Crawl the whole site and look for broken links, redirect chains, missing titles, duplicate metadata and orphaned pages.
3. site:yourdomain.com in Google, as a rough sanity check on how many pages are known.
4. PageSpeed Insights on your most important templates, not just the homepage.

If that sounds like a day you’d rather not spend, a technical audit is where we start with every client, and it’s usually where the fastest gains are hiding.