Every page you load comes with a set of headers most people never look at. Learn what each one controls, which ones affect SEO and security, and how to spot the header that's quietly breaking something.

Every response your server sends arrives with a block of headers before the page content. They decide how long the browser caches the page, whether it is allowed to be embedded in an iframe, whether search engines may index it, and how the content should be interpreted.

Most of the time nobody looks at them, because when they are right, nothing draws attention to them. When one is wrong, the symptom rarely points back to the header. A page that will not update, a font that will not load, a page missing from Google with no obvious reason — all of these are commonly a header, and all of them get investigated somewhere else first.

What You Are Looking At

Request a page and the server responds with a status line, then headers, then the body. A typical set looks like this:

HTTP/2 200
content-type: text/html; charset=UTF-8
cache-control: max-age=3600, public
content-encoding: gzip
strict-transport-security: max-age=31536000
x-frame-options: SAMEORIGIN

The status line comes first and is the single most important thing there — it tells you whether anything else is worth reading. A status code of 200 means what follows describes a real page. A 301 means the headers describe a redirect, and the location header is the only one that matters.

You can see all of this with an HTTP headers parser, or from a terminal with curl -I https://example.com.

The Headers That Decide How Content Is Handled

• Content-Type tells the browser what the response is. text/html; charset=UTF-8 for a page, application/json for an API response. Get the charset wrong, and accented or non-Latin characters render as garbage—a common cause of broken Urdu or Arabic text that looks like a font problem but isn't.
• Content-Encoding shows whether the response was compressed. If gzip or br is absent, compression is not working for that file type.
• Content-Length is the size in bytes of what was sent. Useful for confirming compression actually reduced anything.
• Location appears on 3xx responses and gives the destination. If it is missing on a redirect, the redirect is broken.

The Caching Headers

This is where most mysterious behaviour originates.

cache-control is the main one. max-age=3600 means a browser may reuse the response for an hour without asking. no-cache means it must revalidate before reuse. no-store means do not keep it at all.

The trap is aggressive caching on HTML. Static assets — CSS, JavaScript, images — should be cached for a long time, because their filenames change when they change. HTML should not, because the URL stays the same while the content underneath it updates.

The version of this I see most often is someone updating their homepage, checking it, and seeing the old version. They clear their browser cache, it looks fine, and they move on believing it is fixed. It is not fixed for anyone else. A max-age of a week on HTML means visitors who came last Tuesday will keep getting Tuesday's page until it expires, and there is nothing you can do from your end to pull it back.

ETag and Last-Modified support conditional requests. The browser sends the value back, and the server replies 304 Not Modified if nothing changed, saving the transfer. These are helpful, but they only work if you have not already told the browser it need not ask.

The Headers That Affect SEO

x-robots-tag does the same job as a robots meta tag, but from the header. This matters because it is easy to miss — you can inspect a page's HTML, see no noindex anywhere, and still find the page absent from search because the instruction was in the header the whole time.

If a page is not being indexed and nothing in the HTML explains it, check this header before anything else. It also applies to files that cannot carry meta tags, like PDFs, which is a legitimate reason to use it deliberately.

Link rel="canonical" can also be sent as a header rather than in the HTML, and is again invisible to anyone reading the source. Same principle: if canonicalisation is behaving strangely, look at the headers.

Vary tells caches which request headers change the response. Vary: Accept-Encoding is standard where compression is enabled. If you serve different HTML to mobile and desktop from the same URL, Vary: User-Agent becomes important, and its absence causes caches to serve the wrong version to the wrong device.

The Security Headers

• strict-transport-security forces HTTPS for a period, so the browser will not even attempt HTTP. Effective, but the duration is enforced by the browser and cannot be withdrawn early — set it once you are certain HTTPS works everywhere.
• x-content-type-options: nosniff stops the browser guessing at content types instead of trusting yours.
• x-frame-options or the frame-ancestors directive controls whether your pages can be embedded in someone else's iframe.
• Content-Security-Policy restricts where scripts, styles, and images may load from. The most useful of these and by far the most likely to break your own site if applied carelessly.

Content Security Policy is worth adding, but not on a Friday. Enforcing it immediately tends to block something legitimate — an analytics tag, a font, an embedded widget nobody remembered. Deploy it in report-only mode first, collect what it would have blocked for a couple of weeks, then enforce.

Reading Headers in Practice

1. Check the status line first

Everything else is conditional on it. Headers on a 404 describe an error page, not your content.

2. Follow redirects rather than stopping at the first response

curl -I shows only the first response. curl -IL follows the chain, which is what you want when checking whether a redirect eventually reaches a 200. For anything more than one hop, a redirect checker is easier to read.

3. Test the actual URL users reach

Headers can differ between HTTP and HTTPS, between www and non-www, and between a trailing slash and none. Test the canonical version, not whichever you typed.

4. Check different file types separately

Caching and compression headers are configured per type. Your HTML, your CSS, and your JavaScript can each have different behaviour, and usually do.

5. Test from outside your network

A logged-in session, an admin cookie, or an office IP can all produce different headers than an anonymous visitor sees. Testing from an external tool removes that variable.

Frequently Asked Questions

What is the difference between request and response headers?

Request headers are sent by the browser and describe what it wants and can accept. Response headers are sent by the server and describe what it is returning. Accept-Encoding is a request header; Content-Encoding is the server's answer to it.

Why does my page not update even after I change it?

Almost always a Cache-Control header with a long max-age on HTML. Your own browser can be cleared, but visitors who already have the response cached will keep it until it expires.

My page has no noindex tag, but Google will not index it. Why?

Check for X-Robots-Tag: noindex in the response headers. It does the same thing as the meta tag and is invisible when reading the page source.

Are security headers necessary for a small site?

x-content-type-options and strict-transport-security are quick, safe, and worth setting on any site. Content Security Policy is more work and needs testing, so it is reasonable to leave it until the basics are in place.

How do I see headers without a terminal?

Open developer tools, go to the Network tab, reload, and click the first request — the headers appear under Response Headers. An external checker is often easier, since it also removes your cookies and cache from the picture.

The Short Version

Read the status line first, then the caching headers, then check for X-Robots-Tag if a page is missing from search. The headers that cause the most confusion are the ones that are invisible in the page source, which is exactly why checking them is worth the two minutes.

Written by Baleeghuddin Shaikh, SEO consultant with 8 years of experience in technical SEO and site diagnostics.