Security headers without breaking AI visibility

Adding security headers to our own site broke the language switcher and nearly blocked every AI crawler we spend our working life trying to attract. Both problems were self-inflicted, both were silent, and both are easy to repeat. Here is the full account.

Why this comes up at all

Technical audits check for a handful of HTTP response headers: a content security policy, X-Content-Type-Options, Strict-Transport-Security, Referrer-Policy. Their direct effect on AI visibility is modest, but they appear on every technical quality checklist, and some static hosts cannot send custom headers at all.

The standard fix is to put a CDN proxy in front of the host and add the headers there. That works. It also introduces two traps.

Trap one: the proxy offers to block AI crawlers, and the default is yes

During setup, our CDN offered a panel of AI bot controls: search crawlers, agent crawlers, training crawlers. The recommended default for training crawlers was block, with an additional toggle — enabled by default — to write those blocks into robots.txt automatically.

Think about what that means. Our robots.txt explicitly welcomes every AI crawler. Our blog argues that businesses should let them in. And the infrastructure layer was one click away from overriding all of it, silently, by rewriting the file.

For a publisher protecting original work, blocking training crawlers is a defensible position. For any business that wants to be found and recommended by AI assistants, it is self-sabotage — and it happens at a layer most people never re-check after setup.

What to do: after any CDN or security change, re-fetch your own robots.txt over the public internet and read it. Not the file in your repository — the one your domain actually serves. They can differ, and the difference is invisible from your editor.

Trap two: a strict CSP kills inline event handlers

We set a deliberately strict content security policy: script-src 'self' plus one analytics domain, with no 'unsafe-inline'. We checked the site for inline <script> blocks, found none, and shipped it.

Two weeks later the language switcher stopped working.

The switcher was a <select onchange="location.href=this.value">. An inline event handler attribute is inline script, and the same directive blocks it. It does not error loudly. The handler simply never registers — the console stays clean, the element looks fine in the inspector, and nothing happens on interaction.

The failure was also delayed. The code had not changed in weeks; it broke the moment the headers went live. Nothing in the deployment log pointed at the cause.

The fix, and the wrong fix. The tempting repair is adding 'unsafe-inline' to script-src. One line, everything works, and you have just cancelled the main benefit of the policy you installed.

The correct repair is to move the handler into your existing script file:

document.querySelectorAll("select.lang-select").forEach(function (sel) {
  sel.addEventListener("change", function () {
    if (this.value) location.href = this.value;
  });
});

Policy stays strict, feature works. Inline style attributes are unaffected — style-src is a separate directive and can keep 'unsafe-inline' without weakening script protection.

Writing a CSP that does not break your own site

A policy copied from a checklist will block something you depend on. Ours had to allow three external origins: the analytics script, the endpoint it reports to, and the form-handling worker. A generic default-src 'self' would have silently killed both analytics and lead capture.

Practical sequence:

1. Inventory what your pages actually load. Scripts, fonts, images, and every endpoint your JavaScript calls.
2. Grep for on*= attributes across your templates. Every one is about to stop working.
3. Deploy, then click through the live site with the console open. Every violation prints a directive name — that name tells you exactly what to add.
4. Test the things that submit, not just the things that render. Forms fail differently from layouts.

The general lesson

Infrastructure changes are invisible in your codebase. Nothing in the repository records that headers were added, so when something breaks two weeks later, the cause is not where anyone will look.

After any change at that layer, verify three things from the public internet: your robots.txt reads as intended, your response headers are what you set, and your interactive elements still work. Ten minutes, and it catches exactly the class of failure that otherwise goes unnoticed for months.