One Page Load, 42.8 MB of Data — and Twelve Security Headers That Were Never Sent

TL;DR: The server was fast (TTFB below 0.5 s, CDN hit). The browser still loaded 42.8 MB per detail page, because the app loaded all 83 data files to resolve a single slug — uncached, because the local storage cache cannot hold 42 MB. In parallel: the deployment config's headers rules were never sent, because a low-level routes array overrode them. After the fixes: homepage transfer −73 %, detail pages from 42.8 MB down to a few megabytes, security headers verified live, including an enforced CSP.

Starting point

I measured instead of guessing. One uncached page load, recorded resource timings, five page types:

First look at the server — and it was unremarkable: time_starttransfer between 0.11 and 0.50 seconds, x-vercel-cache: HIT, bundles served with Brotli and immutable. The problem sat entirely on the client.

Finding 1 — Every detail page loaded the whole database

A detail page needs exactly one record: the business whose slug is in the URL. The code loaded every category file instead — 83 CSV files, 122.6 MB raw, 37.2 MB compressed. Two amplifiers came on top:

1. The cache could not work. The CSVs were also kept in localStorage (one-hour TTL). Quota: 5–10 MB. A 42 MB dataset does not fit; every write hits QuotaExceeded, and the next page view downloads everything again.
2. No HTTP cache rule. /data/* shipped with max-age=0, must-revalidate, so the browser revalidated on every navigation.

This dual finding explains why the site feels extremely slow while every individual request is answered quickly: the volume was the problem, not the latency.

By the way: a look at file sizes shows where the bytes live — 87 to 93 % of every large CSV are review texts, of which the detail page displays exactly one.

Finding 2 — The security headers existed only in the file

While checking headers, HSTS answered with the platform default instead of the configured value. Cross-check against three rules from the config:

$ curl -sI https://example.org/booking | grep -i x-robots
# no output — the rule exists in the config but is never sent

$ dig +short TXT _dmarc.example.org
"v=DMARC1; p=quarantine; rua=mailto:...; pct=100"

The deployment config contained two routing layers: a low-level routes array (with handle: filesystem and a rewrite to a serverless function) and the higher-level redirects/headers properties. Runtime result: routes apply, redirects and headers are ignored.

Three pieces of evidence, all reproducible:

• Three redirects from the config answer with 404 instead of 301.
• The X-Robots-Tag rule for the booking page is missing.
• HSTS carries the platform default, not the configured preload value.

The lesson is uncomfortably simple: configuration is not delivery. Anything not proven by curl -I does not exist.

Finding 3 — A redirect loop, live

Two redirect targets were misconfigurations:

• One stub pointed to a path that 301s back to the stub — an infinite loop for real visitors.
• A second stub pointed to a 404 target.

Both lived in the old, inert redirects section — while the redirect that actually fired came from the routes array. The contradiction went unnoticed for a long time, because nobody had ever called the target URL.

The approach: simulate first, build second

The obvious fix — "just load the files for the matching category" — would have been data loss. Slug resolution depends on a global ordering across all files (collisions are disambiguated with city suffixes). A simulation of the naive approach against all 73,676 built routes produced:

• 284 routes would have pointed at the wrong business,
• 38 routes would have come up empty,
• plus 186 routes that are not in the index at all (removed by a dedupe stage).

Only then did the build start — with a regression test that now guards every change to resolution: 73,685 built routes against the index, plus a name sample checked against each page's JSON-LD. Current state: zero deviations.

The fixes

1. Cache first. /data/* gets Cache-Control: public, max-age=3600, stale-while-revalidate=86400 — as a routes entry, because only that is demonstrably applied.
2. Load sections lazily. The heavy comparison section on the homepage mounts through an IntersectionObserver only when it approaches the viewport; its default filter loads one category instead of all of them.
3. Move computation to build time. One overview page loaded 7.9 MB of CSVs to compute three numbers — the processed dataset was never used in rendering. Those numbers are now produced at build time (266 bytes of JSON), with identical logic in a shared module and a runtime CSV fallback.
4. Headers as routes. HSTS with preload, X-Frame-Options: SAMEORIGIN, nosniff, Referrer-Policy, Permissions-Policy — and CSP as Report-Only first.
5. Observe CSP instead of guessing. Violations also land in the browser console in report-only mode. A headless pass over 22 page types surfaced exactly one violation: a demo frame of our own accessibility toolbar was blocked by frame-ancestors 'none'. Fix: frame-ancestors 'self' plus X-Frame-Options: SAMEORIGIN — clickjacking stays blocked, the internal demo works. Then enforcement, then re-measurement: 0 violations across 18 pages.
6. Remove dead configuration. The inert headers section was deleted, its sensible cache rules moved into routes.

Results

Important for the assessment: the server was never slow. Latency was always good — the volume was the problem. A performance debate that starts at the server would have found nothing here.

Check it yourself

# 1. What does the page really load? (count transfer, not requests)
$ curl -sI https://example.org/data/example.txt | grep -i cache-control
$ curl -sI https://example.org/booking | grep -i x-robots-tag

# 2. Do redirect targets resolve — or lead nowhere?
$ curl -s -o /dev/null -w '%{http_code} -> %{redirect_url}\n' https://example.org/old-path

# 3. Header proof after every deploy (only what shows up here exists)
$ curl -sI https://example.org/ | grep -iE 'strict-transport|x-frame|x-content-type|content-security'

For the client side, resource timings beat the network panel:

// In the console: who loads the most?
performance.getEntriesByType('resource')
  .map(r => [r.name, r.transferSize])
  .sort((a, b) => b[1] - a[1])
  .slice(0, 10)

Takeaways

1. Measure before you optimise. The question was never "is the server fast?", it was "what does the browser load?". Only resource timings made the 42 MB visible.
2. Configuration is not delivery. Every header and routing rule needs a curl -I proof after deploy.
3. Observe CSP before enforcing it. Report-only plus a headless console pass finds the one rule that would have broken real users.
4. Simulate the obvious fix before building it. The category fix would have damaged 322 pages — the simulation against all routes prevented it.
5. Computation belongs at build time, not in the browser. 7.9 MB of download for three numbers is not a caching problem, it is an architecture problem.