Technical Readiness & Security

Why This Matters

HTTP-only sites are completely excluded from all major AI systems. GPTBot, ClaudeBot, Perplexity, and enterprise RAG pipelines refuse to connect to non-HTTPS origins due to security policies. Your entire site is invisible to AI-generated answers, product recommendations, and agentic workflows.

How to Fix

Enable HTTPS by obtaining a TLS certificate (free via Let's Encrypt) and configuring your web server to serve all traffic over HTTPS. Set up a 301 redirect from HTTP to HTTPS to ensure all traffic is encrypted.

Example

# Certbot (Let's Encrypt) quick setup:
sudo certbot --nginx -d yoursite.com -d www.yoursite.com

# Nginx HTTPS config:
server {
  listen 443 ssl;
  ssl_certificate /etc/letsencrypt/live/yoursite.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/yoursite.com/privkey.pem;
}

Why This Matters

Without HSTS, AI agents that first connect over HTTP must follow a redirect to HTTPS, wasting a round-trip on every new connection. Worse, strict enterprise AI security policies may reject non-HSTS sites outright, blocking your content from enterprise RAG pipelines and AI-powered procurement tools.

How to Fix

Add the Strict-Transport-Security header to your HTTPS responses. Use a max-age of at least one year (31536000 seconds), include subdomains, and add the preload directive to be included in browser preload lists.

Example

Strict-Transport-Security: max-age=31536000; includeSubDomains; preload

Why This Matters

AI trust-scoring systems treat a missing Content-Security-Policy as a sign of weak security posture. Sites without CSP are vulnerable to XSS attacks that can inject malicious content into the pages AI agents crawl, poisoning AI knowledge bases with attacker-controlled text. This lowers your trust score and may cause AI systems to deprioritize or exclude your content from recommendations.

How to Fix

Add a Content-Security-Policy header to your server responses. Start with a restrictive policy that only allows resources from your own origin, then gradually expand as needed.

Example

Content-Security-Policy: default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self'

Why This Matters

Without the nosniff directive, browsers and AI agents may MIME-sniff your responses and misinterpret their format. JSON-LD can be treated as HTML, API responses parsed as plain text, and structured data silently ignored — breaking schema extraction and product data ingestion.

How to Fix

Add the X-Content-Type-Options: nosniff header to all responses. Most web servers and CDNs support this as a single configuration line.

Example

X-Content-Type-Options: nosniff

Why This Matters

Without a Referrer-Policy header, your site leaks full URL paths (including query parameters) in HTTP Referer headers when users navigate to external links. AI security audits flag this as a privacy vulnerability, and trust-scoring systems lower your site's rating. Sensitive URL parameters like session tokens or search queries may be exposed to third parties.

How to Fix

Add a Referrer-Policy header to your server responses. The recommended value is "strict-origin-when-cross-origin", which sends the full URL for same-origin requests but only the origin for cross-origin requests.

Example

Referrer-Policy: strict-origin-when-cross-origin

Why This Matters

Without a Permissions-Policy header, AI browser agents visiting your site may trigger unexpected permission prompts for camera, microphone, or geolocation. These prompts block automated agent workflows entirely and are flagged by AI trust-scoring systems as a security concern, reducing your site's trust score.

How to Fix

Add a Permissions-Policy header that disables sensitive browser features your site does not use. Deny camera, microphone, and geolocation unless your site explicitly requires them.

Example

Permissions-Policy: camera=(), microphone=(), geolocation=(), payment=()

Why This Matters

AI trust-scoring systems check for /.well-known/security.txt as a signal that your organization follows responsible disclosure practices. Its presence contributes to a higher overall trust score in enterprise AI frameworks that evaluate site maturity, making your content more likely to be recommended in AI-generated answers.

How to Fix

Create a security.txt file at /.well-known/security.txt with at minimum a Contact field (email or URL), an Expires date, and preferred languages. Optionally include your security policy URL and PGP key.

Example

Contact: mailto:[email protected]
Expires: 2027-12-31T23:59:59.000Z
Preferred-Languages: en
Policy: https://yoursite.com/security-policy

Why This Matters

Browser-based AI tools, ChatGPT plugins, and MCP clients all run in browser contexts governed by the same-origin policy. Without CORS headers on your llms.txt and AI catalog, these agents receive a network error instead of your content — making your AI-facing files completely invisible to the fastest-growing category of AI consumers.

How to Fix

Add Access-Control-Allow-Origin and Access-Control-Allow-Methods headers to your /llms.txt and /.well-known/ai-catalog.json responses. Use a wildcard origin (*) unless you need to restrict access to specific domains.

Example

Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, OPTIONS

Why This Matters

AI agents running in browser contexts (ChatGPT plugins, MCP clients, browser extensions) must pass CORS preflight checks before making API calls. Without CORS headers, every cross-origin API request is silently blocked, preventing all agentic workflows that need to query your product catalog, check inventory, or place orders on behalf of users.

How to Fix

Configure your API routes to respond to OPTIONS preflight requests with Access-Control-Allow-Origin, Access-Control-Allow-Methods, and Access-Control-Allow-Headers. Apply this to all /api/ routes that AI agents may call.

Example

Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Headers: Content-Type, Authorization

Why This Matters

Incorrect Content-Type headers cause AI agents to misparse your files. JSON served as text/html breaks structured data extraction, XML sitemaps served as text/plain prevent crawl discovery, and llms.txt served as application/octet-stream triggers downloads instead of being read. This silently breaks the AI data pipeline.

How to Fix

Configure your web server or CDN to serve each file with the correct MIME type: application/json for JSON files, application/xml for XML sitemaps, and text/plain for llms.txt.

Example

llms.txt:        Content-Type: text/plain
openapi.json:    Content-Type: application/json
ai-catalog.json: Content-Type: application/json
sitemap.xml:     Content-Type: application/xml

Why This Matters

Without Cache-Control headers on AI-facing files, every AI agent request re-fetches the full file from your server. This wastes bandwidth, increases server load, and slows down agent crawling — especially during traffic spikes when multiple agents poll simultaneously.

How to Fix

Add a Cache-Control response header to your /llms.txt and /openapi.json routes. A value of "public, max-age=3600" lets agents cache the file for one hour, which is a good balance between freshness and efficiency.

Example

Cache-Control: public, max-age=3600

Why This Matters

AI crawlers like GPTBot and ClaudeBot enforce strict per-page timeout budgets (typically 2-3 seconds). A slow TTFB means fewer of your pages get crawled per session, and pages that exceed the timeout are abandoned entirely. Over time, this results in incomplete coverage in AI knowledge bases — your products and content simply do not appear in AI-generated answers.

How to Fix

Reduce Time to First Byte (TTFB) to under 800ms. Use server-side caching (Redis, in-memory), deploy behind a CDN with edge caching, minimize database queries on critical paths, and consider pre-rendering or static generation for content pages.

Example

# Nginx microcaching example:
proxy_cache_path /tmp/cache levels=1:2 keys_zone=ai:10m max_size=1g;
proxy_cache_valid 200 1m;
add_header X-Cache-Status $upstream_cache_status;

Why This Matters

AI crawlers (GPTBot, ClaudeBot, PerplexityBot) do not execute JavaScript. If your content is only rendered client-side, these crawlers see an empty or near-empty page. Your products, articles, and brand information are completely absent from AI knowledge bases, meaning AI-generated answers never reference your site.

How to Fix

Switch from client-side rendering to server-side rendering (SSR) or static site generation (SSG). Frameworks like Next.js, Nuxt, SvelteKit, and Astro all support SSR/SSG. Ensure your homepage and key landing pages return meaningful HTML content in the initial response.

Example

// Next.js App Router (server component by default):
export default async function Page() {
  const data = await fetchProducts();
  return <ProductList items={data} />;
}

// Or with getServerSideProps (Pages Router):
export async function getServerSideProps() {
  const data = await fetchProducts();
  return { props: { data } };
}

Why This Matters

Synchronous scripts in <head> block HTML parsing and delay content delivery. Since AI crawlers do not execute JavaScript, these scripts add pure latency with zero benefit — the crawler waits for scripts to download but never runs them. This wastes crawl budget and slows TTFB, reducing the number of pages AI agents can index.

How to Fix

Add the defer or async attribute to all <script> tags in <head>, or convert them to type="module". Move non-critical scripts to the end of <body> if they cannot be deferred.

Example

<!-- Before (blocking): -->
<script src="app.js"></script>

<!-- After (non-blocking): -->
<script src="app.js" defer></script>

Why This Matters

Missing width and height attributes on images cause layout shifts (CLS) as images load. AI agents that use visual screenshots (such as Claude computer use) see elements jump positions between frames, breaking coordinate-based click targeting. High CLS also hurts Core Web Vitals scores, which AI trust-scoring systems factor into content quality rankings.

How to Fix

Add explicit width and height attributes to all <img> tags. Use CSS aspect-ratio or the HTML attributes to reserve space before the image loads. For responsive images, set width/height to the intrinsic dimensions and use CSS to scale.

Example

<img src="product.jpg" width="800" height="600" alt="Product photo"
     style="max-width: 100%; height: auto;">

Why This Matters

Lazy-loading the Largest Contentful Paint (LCP) element delays the most important visual content on the page. AI agents using visual screenshots see a blank placeholder instead of your hero image, and Core Web Vitals scores suffer — both of which reduce your content's ranking in AI trust-scoring systems.

How to Fix

Remove loading="lazy" from the first/largest image on the page (the LCP candidate). Instead, add fetchpriority="high" to tell the browser to prioritize loading this image immediately.

Example

<!-- Before (bad): -->
<img src="hero.jpg" loading="lazy" alt="...">

<!-- After (good): -->
<img src="hero.jpg" fetchpriority="high" alt="...">

Why This Matters

Without preconnect hints, each third-party resource requires a full DNS lookup, TCP handshake, and TLS negotiation before loading can begin. This adds hundreds of milliseconds per origin, slowing overall page load and reducing the number of pages AI crawlers can process within their time budget.

How to Fix

Add <link rel="preconnect"> tags in <head> for your most critical third-party origins (CDNs, font providers, analytics). Limit to 2-4 origins to avoid diminishing returns.

Example

<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://cdn.yoursite.com" crossorigin>

Why This Matters

Broken URLs in your AI manifest files (ai-catalog.json, llms.txt, navigation.json) cause agents to lose trust in your entire manifest. After encountering broken links, AI systems may stop following any of your listed endpoints, effectively making all your AI-facing resources undiscoverable.

How to Fix

Audit all URLs referenced in your ai-catalog.json, llms.txt, and navigation.json files. Fix or remove any that return 404, 500, or connection errors. Set up monitoring to catch broken endpoints before AI agents do.

Example

# Verify your AI endpoint URLs:
curl -sI https://yoursite.com/llms.txt | head -1
curl -sI https://yoursite.com/.well-known/ai-catalog.json | head -1
curl -sI https://yoursite.com/openapi.json | head -1

Why This Matters

Enterprise AI frameworks and compliance-aware recommendation systems check for a privacy policy before surfacing your site in regulated industries (health, finance, legal, e-commerce). A missing privacy policy lowers your trust score and may cause AI systems to exclude your site from recommendations where data handling transparency is required.

How to Fix

Create a dedicated privacy policy page at /privacy-policy/ or /privacy/. Cover data collection practices, cookie usage, third-party data sharing, and user rights under GDPR/CCPA.

Example

<!-- Create /privacy-policy/ with these key sections:
  1. What data you collect
  2. How you use cookies and tracking
  3. Third-party data sharing
  4. User rights (access, deletion, opt-out)
  5. Contact information for privacy inquiries -->

Why This Matters

AI agents check for terms of service to determine whether automated access and content usage are permitted. A missing ToS creates legal ambiguity that causes enterprise AI systems to deprioritize your site to avoid compliance risk. Clear terms also let you explicitly define AI usage policies for your content.

How to Fix

Create a terms of service page at /terms/ covering acceptable use, AI/automated access policy, content licensing, and liability limitations. Consider adding an explicit section on AI agent access and content reuse permissions.

Example

<!-- Create /terms/ with these key sections:
  1. Acceptable use policy
  2. AI and automated access policy
  3. Content licensing and reuse terms
  4. Liability limitations
  5. Termination conditions -->

Why This Matters

This is an informational audit. Knowing your frontend framework helps identify potential rendering issues — for example, pure client-side React apps are invisible to AI crawlers that do not execute JavaScript. Framework detection also helps prioritize other audit recommendations.

How to Fix

No action required. This audit detects your framework automatically. If your framework relies on client-side rendering (e.g., Create React App, Vue SPA), consider switching to a server-rendering mode (Next.js, Nuxt, or SvelteKit) so AI crawlers can read your content.