How Much of the Web Disappears Without JavaScript?

I read about celld last week: an open-source daemon that re-implements Cloudflare Workers and Durable Objects, self-hosted, no Cloudflare account required. I wanted to try it on my hobby project — a link-rot checker for SPAs that can tell a dead link apart from an unhydrated shell. The usual Workers pattern for that is a managed browser binding. Turns out celld doesn’t ship that primitive, and it’s not on the roadmap. 😭

That got me curious: how much of the web actually disappears without JavaScript in 2026? How far could a Workers runtime get without a browser at all? So I fetched 100 real URLs to see what mattered more — bypassing anti-bot measures, or getting page JS to run. It was the former, by a wide margin. So then I did the same fetch from inside a celld Worker, HTTP only, no browser, no page JS — reaching for Web Unlocker whenever I hit a Cloudflare/WAF challenge or a 403 — to see what happened.

Bare fetch from that celld Worker returned usable data only ~30% of the time. With Web Unlocker, still no JavaScript execution, that jumped to ~72%. Turns out, you can get a lot out of celld — a browserless, JavaScript-less (page JS, that is) Workers equivalent on the modern web — if you’re smart about what’s actually blocking you.

Let’s find out how.

How do I obtain accurate data from difficult-to-crawl websites?

There are plenty of studies out there that measure JavaScript as a barrier, but I found that they only measure landing pages. In most cases, the data you actually want lives on an inner page instead — a listing, a product, an article, whatever.

So I froze an equal(ish) mix of 100 public inner pages spread across 13 sectors: travel/lodging, real estate, restaurants, airlines, pharmaceuticals, sports, TV, music, furniture, apparel, food, retail, and mass media. Anything reachable without a login.

Anything reachable without a login. URLs like these:

https://www.booking.com/hotel/us/the-plaza.html  
https://www.zillow.com/homedetails/15-Central-Park-W-New-York-NY-10023/97516205_zpid/  
https://www.ikea.com/us/en/p/billy-bookcase-white-20522046/

As I said, to establish the challenge pages/403 vs. JavaScript gap baseline, I first fetched each URL three ways.

• Stage 1 is a plain HTTP GET, no JavaScript at all.
• Stage 2 is that same GET through Web Unlocker, a commercial anti-bot bypass, still with no rendering and no page JavaScript running.
• Stage 3 is the opposite end of the spectrum — full local Playwright.

100 frozen inner URLs  
      │  
      ▼  
Acquisition ──► http_get()            Stage 1 — HTTP GET only  
             ──► unlocker_get()       Stage 2 — HTTP GET + Web Unlocker, no render  
             ──► playwright_get()     Stage 3 — local Chromium (to run page JS)  
      │  
      ▼  
Extract(html, schema)                 same extractor, every run  
      │  
      ▼  
Required fields match frozen identity?  pass / fail, plus a mismatch class

These runs are independent of each other i.e. not a ladder. Web Unlocker without render is the access stack only: it can solve challenges and bypass CAPTCHA automatically, but does not run the listing’s page JavaScript. Local Playwright is the opposite stack: a real browser that can run HTML/DOM/page JavaScript, but has nothing to guarantee access.

Why an HTTP 200 Does Not Mean Success

An HTTP 200 only tells you that the server responded successfully. It does not (and cannot) tell you that the data you’re actually after was successfully collected. Just a 200 OK could mean your fetched HTML was:

• An empty React shell waiting for JavaScript to run
• A Cloudflare or AWS WAF challenge page instead of the page you thought you were getting
• Perfectly valid website markup, but with the actual record missing because JavaScript was supposed to fetch it behind a loading spinner.

All of these would be fatal for my hobby project. If my use case ever involved RAG, it would be even worse — the document would look complete to every single one of my downstream systems. My Retrieval layer couldn’t return information that was never collected, so the Generation layer would have to fill the gap from whatever context it could find (say hello to ALL the hallucinations) — but corpus metrics would still just say everything was healthy.

So you can’t score status codes, and you can’t reliably use body size in bytes as a proxy for success. You need to ask a smarter question:

Does this page actually contain the record I came here to collect?

A Schema-Based Approach For Judging Field Completeness

First, I built a schema per page type, and separately, a seed row per URL. Those are two different things, so let me explain them:

• The schema is a checklist of which fields a page type needs to show — name, location, rating, description, whatever that job actually calls for.
• The seed is the correct value for those fields on one specific URL — the exact name, city, and description I expect to see, written down before I fetched anything.

A page passes only when both line up just right — the right fields, with the right values.

“Page type” just means the job the URL is doing. All 100 URLs, across 13 industries, can be boiled down to just two kinds of jobs:

1. Looking for a detail — a specific listing someone would click through to: a hotel, a property, an SKU, a drug, a film, an article.

https://www.booking.com/hotel/us/the-plaza.html  
https://www.zillow.com/homedetails/15-Central-Park-W-New-York-NY-10023/97516205_zpid/  
https://www.ikea.com/us/en/p/billy-bookcase-white-20522046/  
https://www.drugs.com/ibuprofen.html  
https://www.imdb.com/title/tt0468569/  
https://www.bbc.com/news/world-51839928

2. Looking for a results list — an index or a set of results for some query.

https://www.booking.com/searchresults.html?ss=New+York  
https://www.zillow.com/new-york-ny/  
https://www.tripadvisor.com/Restaurants-g60763-New_York_City_New_York.html  
https://www.kayak.com/flights/NYC-LON

So for the schema, required fields must follow the job, not the brand — lodging and restaurant pages need a location and a rating, for example:

const DETAIL = {  
  travel_lodging: ["name", "location", "rating", "description"],  
  restaurants:    ["name", "location", "rating", "description"],  
  real_estate:    ["name", "location", "description"],  
  airlines:       ["name", "location", "description"],  
  // Similarly...  
  // pharmaceuticals, sports, tv_movies, music,  
  // furniture, apparel, food_beverages, retail, mass_media  
  default:        ["name", "description"],  
};  
const RESULTS = ["destination", "listing_name", "listing_signal"];  
// listing_signal is vertical-aware: rating or review count for  
// lodging/restaurants, result count for a plain index, price or  
// duration for a flight search — "some signal beyond a bare name."

The same logic extends to every other sector.

Now the seed. Every one of the 100 URLs has a row, written before any fetch: the URL, its job (which schema it gets), the name I expect to find, and the place I expect to find it. I save it to a seed.tsv

url,publisher,page_kind,geo,label,vertical  
https://www.discogs.com/master/21491-Radiohead-OK-Computer,discogs,album_detail,GLOBAL,OK Computer,music  
https://www.bbc.com/sport/football/teams/manchester-united,bbc,team_page,GB-LON,Manchester United,sports  
https://www.mlb.com/player/shohei-ohtani-660271,mlb,player_detail,GLOBAL,Shohei Ohtani,sports  
https://www.imdb.com/title/tt0468569/,imdb,title_detail,GLOBAL,The Dark Knight,tv_movies

That row is frozen — it’s essentially the answer key, and it was locked in before I ran a single request. Schema choice and identity are both decided in advance; nothing here is picked by looking at what came back on the page.

You could save yourself some trouble by using an LLM to generate these pairings. Probably more accurate with a frontier model — but it would need raw HTML, and that blows through a context window fast across 100 URLs. Not tenable at this scale.

Each HTML body I collect is one observation of that same URL. I run the identical extractor on every one. It only pulls candidate strings — <title>, Open Graph, JSON-LD (name, address, description, ratingValue), and <h1>. A pass requires two things at once: every field the schema requires has to be present, and its value has to match the seed — a name has to hit the seeded alias for that listing, not just occupy a name-shaped slot. A JSON-LD name for some other listing doesn't count. A challenge page that just so happens to say "Plaza" in the footer doesn't count.

That’s why the seed is adversarial where it matters most. Eleven of the lodging rows are near-duplicate “Plaza” hotels spread across four sources:

The Plaza New York  
Hotel Riu Plaza Times Square  
Broadway Plaza Hotel  
Hotel Plaza Athénée  
Best Western Plus Plaza Hotel

A matcher that passes on a bare substring hit for “Plaza” will fail immediately and visibly.

Note that this can still false-negative when locale or markup drifts past the aliases I froze. If you absolutely need to fix these drifts outside your control (selectors on the page changing, locales relabeling certain fields, multiple DOM variants) — I found Scraper Studio incredibly useful: you describe the fields in natural language, it writes the extractor, self-corrects using AI, and runs on Bright Data’s unblocking network.

So what results did we get?

What Matters More — Bypassing CAPTCHA/Challenges or Enabling JavaScript?

Stage 1: Plain fetch (31% Useful Data)

This is the cheapest baseline — what a Node.js script (and later, a celld Worker) does with a bare fetch and nothing else:

async function httpGet(url) {  
  const res = await fetch(url, {  
    headers: {  
      "User-Agent": UA,  
      Accept: "text/html,application/xhtml+xml",  
    },  
    redirect: "follow",  
    signal: AbortSignal.timeout(30_000),  
  });  
  return { status: res.status, html: await res.text() };  
}

With this simple approach, 31 / 100 URLs were field-complete. As expected, not great. If this already gives you everything you need — congratulations. Stop right here. You’ve saved yourself a ton of time, effort, and possibly money.

Stage 2: Web Unlocker, No Rendering (73% Useful Data)

This is where we judge whether those 69 failures in Stage 1 were due to a genuine JS requirement, or just a Cloudflare/WAF challenge gate. Web Unlocker rotates proxies, sets fingerprints, and solves CAPTCHAs automatically.

Web Unlocker has two access methods that they say return identical results: a native HTTPS proxy, and a REST POST to https://api.brightdata.com/request. For this baseline I used the native proxy, and did NOT add -render to the auth string, which would have forced page JavaScript rendering. This run is only Unlocker as an access tool, so we don't want headless Chrome. Unlocker may still run challenge JavaScript on their infra — fingerprints, CAPTCHAs, Cloudflare/WAF probes — but that is not JavaScript on our target page hydrating **#app**.

import { ProxyAgent, fetch as proxyFetch } from "undici";  
const AUTH = process.env.BRIGHT_DATA_UNLOCKER_AUTH; // USER:PASS, no `-render`  
const dispatcher = new ProxyAgent({  
  uri: `@brd.superproxy.io:44445`">http://${AUTH}@brd.superproxy.io:44445`,  
  requestTls: { rejectUnauthorized: false },  
  proxyTls: { rejectUnauthorized: false },  
});  
const res = await proxyFetch(url, {  
  dispatcher,  
  headers: { "User-Agent": UA, Accept: "text/html,application/xhtml+xml" },  
  redirect: "follow",  
});  
return res.text();

If you’re using this, you’ll need your Unlocker username and password formatted as that ${AUTH} string. Sign up here to get them. New pay-as-you-go accounts get 5,000 free credits per month, no credit card required, with a hard stop when you run out.

Your username looks like brd-customer-<id>-zone-<zone_name> — zone name only, no -render suffix. The password is your zone password from the dashboard. Drop both into .env as a USER:PASS auth string:

BRIGHT_DATA_UNLOCKER_AUTH=brd-customer-XXXXX-zone-web_unlocker:PASSWORD  
BRIGHT_DATA_UNLOCKER_PROXY_HOST=brd.superproxy.io  
BRIGHT_DATA_UNLOCKER_PROXY_PORT=44445

Using the Web Unlocker, field completeness jumped from 31 to 73/100 — a whopping +42 jump from bypassing anti-bot measures alone. I went in expecting the JS gap to dominate, since most of these are React or Vue front ends, so seeing access outweigh it by nearly 2:1 was quite the surprise 👀.

However, 27 URLs still returned only an empty #app shell.We can be reasonably certain that those were genuine JavaScript failures: opting out of render didn't fix them, and as we'll see next, nothing short of running the page's JS in a real browser will.

Stage 3: Local Playwright (54% Useful Data)

This run is independent of Stage 2. We’re using your very regular, run-of-the-mill local Playwright script — this guarantees on-page JavaScript runs, but will do nothing to ensure access to the page in the first place.

const browser = await chromium.launch({ headless: true });  
const page = await browser.newPage();  
await page.goto(url, { waitUntil: "domcontentloaded" });  
await page.waitForLoadState("networkidle", { timeout: 10_000 }).catch(() => {});  

const html = await page.content();

Surprisingly, Playwright pushed field-completeness only to 54 / 100 — a +23 lift from just allowing JavaScript to run, well short of Stage 2’s 73 / 100, which never ran page JavaScript at all.

So we got fewer usable records by enabling JavaScript and rendering via a real browser, than we did by simply fixing access and making HTTP fetch calls. It’s quite the non-intuitive result.

Obviously, our 100 URL spread, while diverse, is not a representation of the internet — but I think we definitely have enough to say that you should only take the Playwright/Puppeteer/Selenium route when you’re certain JavaScript is the blocker — i.e. when you genuinely get an empty shell on the URLs you want. Don’t make it the default.

For completeness’ sake, I also used Bright Data’s hosted Chromium via their Browser API, connecting over CDP. This yielded 98 / 100 — our best result yet, but ultimately irrelevant for our needs — celld cannot use browsers anyway, local or hosted. Regardless, this result does show that the 27 Unlocker failures were, in fact, because of JavaScript. Given access and a browser, almost every URL was collected.

Now that I had empirical proof that I could get better results in the long run by ensuring access via Web Unlocker than I could by ensuring the presence of a real DOM + JavaScript with a real browser, I ran celld locally via rclone, and reran the HTTP GET layer in it.

How Do I Run celld Locally On My PC?

So celld has no local-filesystem mode — each of its nodes needs an S3-compatible bucket with conditional writes. But you can run it locally on your PC using rclone — a package that can map a local filesystem directory to a virtual S3 bucket.

I gave rclone access to an isolated folder on my PC and served it as S3, then hooked it up to celld at that endpoint as per their docs:

rclone serve s3 --addr 127.0.0.1:9090 --auth-key celld,celld ~/celld-s3  
export AWS_ACCESS_KEY_ID=celld # these can be whatever  
export AWS_SECRET_ACCESS_KEY=celld # these can be whatever  
export CELLD_STORAGE_PROBE=0   # rclone accepts If-Match and does not enforce it  
celld deploy . --bucket s3://celld --endpoint http://127.0.0.1:9090  
celld        --bucket s3://celld --endpoint http://127.0.0.1:9090

The celld bucket is still required because celld stores the deployment and the node lease there even when the app has no cells.

This part is a bit tricky — no Windows binary exists for celld yet, so I’m running it under WSL. Of course, rclone serve s3 is highly experimental — but that's fine, you're never going to run celld with local filesystem-based storage except for testing and/or experimentation, anyway.

Just a quick note since celld has been blowing up recently because of the Durable Object angle — our celld app is an ordinary Wrangler Worker (a Cloudflare Worker equivalent, essentially) that celld deploy accepts: wrangler.jsonc and a fetch handler. It is not a Durable Object.

Stage 1 on celld is the same vanilla HTTP GET from earlier, just running inside celld this time:

export default {  
  async fetch(request) {  
    const url = new URL(request.url).searchParams.get("target");  
    const res = await fetch(url, {  
      headers: {  
        "User-Agent": UA,  
        Accept: "text/html,application/xhtml+xml",  
      },  
      redirect: "follow",  
      signal: AbortSignal.timeout(30_000),  
    });  
    return new Response(await res.text(), {  
      headers: { "content-type": "text/html; charset=utf-8" },  
      status: res.status,  
    });  
  },  
};

Easy enough. Run it, collect and analyze the HTML it gets. Just as we did before.

Something to note — celld is not Node.js. It is a daemon that runs Wrangler bundles. Browsers aren’t the only thing out of scope — so are TCP sockets that actually connect.

So for Stage 2, we cannot use Bright Data’s Web Unlocker via the proxy layer inside celld — we have to use its API instead. Genuinely good use case for having TWO access methods, now that I look back on it.

Stage 2 — our Web Unlocker layer on celld — is Bright Data’s direct API instead of their native proxy: we make a POST request to https://api.brightdata.com/request, with a bearer token, a zone name, the target URL, and format: "raw". As discussed before, do not set render: "true"

export default {  
  async fetch(request, env) {  
    const url = new URL(request.url).searchParams.get("target");  
    const res = await fetch("https://api.brightdata.com/request", {  
      method: "POST",  
      headers: {  
        Authorization: `Bearer ${env.BD_API_KEY}`,  
        "Content-Type": "application/json",  
      },  
      body: JSON.stringify({  
        zone: env.BD_ZONE,  
        url,  
        format: "raw", // no render, no JavaScript. Access only.  
      }),  
    });  
    return new Response(await res.text(), {  
      headers: { "content-type": "text/html; charset=utf-8" },  
    });  
  },  
};

The results stayed near-identical to our earlier Node.js script:

A different TLS stack didn’t seem to have changed the story much: bare fetch from the Worker sees the same web Node does, give or take a couple of pages. Unlocker too: 73 on the Windows proxy, 72 through the isolate's API route. Pretty much within margin of error.

If you’re building on celld, the priority isn’t “wait for browser support.” It’s “solve access first” — that’s most of the gap, and it’s solvable today with what celld already ships on the Worker surface: outbound fetch, pointed at an Unlocker API.

The Takeaway

A larger portion of the web is actually hidden from you by Cloudflare/AWS WAF challenge pages, 403 errors, anti-bot measures, and so on, than it is by JavaScript.

Just solving that access problem boosted our usable data from 31 to 73 (n=100) while jumping through hoops to ensure JavaScript on a page always ran could only boost it from 31 to 54. Ensuring access to much of the web is clearly the harder challenge — and the bigger gain.

Butcelld can already do it out of the box — an outbound fetch call to the Bright Data’s Web Unlocker API is all it takes.

Comments

Loading comments…