Why I Wrote a Web Scraper to Track Live Poker Tournament Payouts (And What It Revealed About Variance)

I started this project because I lost a $200 buy-in in 45 minutes and wanted to know if that was normal or if I just play badly. Turns out both things can be true at once. But the real story here isn't tilt therapy. It's the scraper.

Over three weekends I built a Python scraper paired with a Next.js dashboard that pulls live tournament payout structures from public results pages, normalizes them by buy-in tier, and plots variance across field sizes. No API, no partnership, just requests, BeautifulSoup, and a lot of regex that I'm not proud of. The dashboard now tracks something like 340 tournaments across four buy-in bands, and the shape of the data surprised me.

This post walks through the build. Stack, scraping logic, the schema I landed on after rewriting it twice, and the chart that actually made me rethink my own bankroll rules. Raw payout numbers tell you how top-heavy a structure is. They don't tell you what to do with that information once you're sitting at a table deciding whether to shove a flush draw for 40% of your stack. TThat gap between data and decision is real, and it's not one a scraper closes on its own. I'll get to why.

The Stack, and Why I Didn't Overthink It

Python for the scraper. Next.js for the front end. A Postgres instance on a $6 droplet doing all the heavy lifting.

I considered Scrapy. Went with plain requests plus BeautifulSoup instead, mostly because the target pages weren't JavaScript-rendered and I didn't want to spin up Playwright for something this small. Sometimes the boring tool wins.

The scraping logic itself splits into three functions: fetch the results page, parse the payout table, normalize the buy-in against a currency field that three different sites format inconsistently. That normalization step ate an entire Saturday. One site listed buy-ins as "$500+50" (buy-in plus fee), another just listed "$550," and a third used a currency symbol that broke my regex until I added a Unicode-aware match. Small thing. Cost me four hours.

```python import requests from bs4 import BeautifulSoup import re

def fetch_payout_table(url): resp = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=10) soup = BeautifulSoup(resp.text, "html.parser") rows = soup.select("table.payouts tr") return [parse_row(r) for r in rows if r]

def normalize_buyin(raw): match = re.search(r"\$?([\d,]+)(?:\+([\d,]+))?", raw) if not match: return None base = int(match.group(1).replace(",", "")) fee = int(match.group(2).replace(",", "")) if match.group(2) else 0 return base + fee ```

Respecting robots.txt and rate limits mattered here, not as a legal checkbox but because getting blocked mid-scrape means starting the whole run over. A recent academic paper on the ethics and legality of research scraping lays out the considerations better than I can, and I ended up following its throttling recommendations almost exactly: one request every 2 to 4 seconds, randomized, with a proper user agent string rather than pretending to be a browser I wasn't.

The Schema Problem Nobody Warns You About

Here's what I didn't expect. The hard part wasn't scraping. It was deciding what a "payout tier" even means when tournament structures vary this much.

Some events pay 15% of the field. Others pay 10%. A $200 buy-in event with 800 entrants and a $1,000 buy-in event with 200 entrants aren't comparable on raw dollar figures, so I built a normalized "multiplier of buy-in" column for every finishing position. First place divided by buy-in. Final table average divided by buy-in. Min-cash divided by buy-in.

That's the column that made the chart interesting.

What the Variance Chart Actually Showed

Across the 340 tournaments in my dataset, min-cash payouts averaged 1.4x to 1.8x the buy-in depending on tier. Sounds fine on paper. Barely worth the four hours of variance and dead time it takes to get there.

But first-place multipliers told a completely different story. In the $100 to $250 tier, winners averaged 140x buy-in. In the $1,000+ tier, that dropped to roughly 90x. Bigger buy-ins, flatter payout curves, relative to the money on the table. Makes sense once you think about it (softer structures at low stakes tend to overpay the top spots to keep recreational players hooked) but I hadn't seen it laid out with actual numbers before.

The Las Vegas Review-Journal's coverage of a recent WSOP Main Event with over 9,000 entrants and a $10 million top prize is a good real-world anchor for this. That's a field size and payout scale my scraper's low-stakes sample can't touch, but the shape holds. Massive fields flatten the payout curve even as the total pool grows.

I plotted this in the Next.js dashboard using Recharts, a stacked bar showing multiplier-of-buy-in against finishing position, faceted by buy-in tier. Nothing fancy. The insight was in the normalization, not the chart library.

Where the Data Runs Out and Judgment Has to Start

This is the part that actually changed how I play, not just how I code.

A scraper can tell you that a $300 tournament pays min-cash at 1.6x buy-in on average. It cannot tell you whether shoving 60 big blinds into a three-bet pot in level 14 is correct given your specific stack, the bubble dynamics, or the fact that the guy across from you has three-bet nine times in two hours. That's strategy, not statistics. My dataset is descriptive. It describes what happened across a sample. It says nothing about what you should do next Tuesday at a $150 event with 11 big blinds and a short stack to your left.

My scraper can tell you the average min-cash multiplier for a $300 field. It cannot tell you the right shove range from the button with 11 big blinds, or when a three-bet is a bluff versus a value hand from a specific opponent. That's what sites built specifically to teach hand-reading and range construction are for, and it's a different skill set than the one I spent three weekends automating. Pokerology is the kind of resource I mean: strategy content aimed at the exact decision point where my data stops being useful. A paper I came across afterward, a heuristic framework for adaptive poker AI, makes a similar point from the opposite direction: even models built to exploit opponents need a strategic layer on top of raw statistical inference, not instead of it.

Bankroll management is where this bit me hardest. I'd been playing $300 buy-ins off a bankroll that, by any sane rule, should have kept me at $150 or under. The variance data made that obvious in a way that gut feeling never had. Seeing the min-cash rate at 1.6x and realizing how often I was landing below that line, not above it, was a genuinely uncomfortable Saturday afternoon with a spreadsheet.

Building It Yourself

If you want to build something similar, the Real Python guide to web scraping is still the cleanest starting point I've found for the actual HTML-parsing mechanics, and I referenced it more than once when my selectors broke on a site redesign mid-project.

The rough build order that worked for me:

• Pick 3 to 5 public results sources and check each one's robots.txt before writing a single line of scraping code.
• Build the fetch and parse functions first, test against saved HTML snapshots so you're not hammering the live site while debugging.
• Design your normalization schema before you scrape at scale. I rebuilt mine twice because I skipped this step.
• Store raw and normalized values separately. You'll want to re-derive metrics later without re-scraping.
• Wire up the Next.js front end last. The data model is the hard part. The charts are the easy part.

One thing I'd do differently: I'd add a data validation layer earlier. Twice I ran overnight scrapes that silently failed on a subset of pages because of a malformed table, and I didn't catch it until the variance numbers looked weirdly clean the next morning.

Why This Matters Beyond My Side Project

Forbes' coverage of a recent WSOP Main Event points out just how much the series has grown as a spectacle and a data source. That growth means more public payout data than ever, which means more room for exactly this kind of project. If you're a developer who also plays, there's a genuinely underbuilt niche here. Most poker data tools are built for pros tracking their own hand histories, not for structural analysis across the wider tournament ecosystem.

I'd like to extend the scraper to pull hand-for-hand final table data next, which is messier because a lot of it lives in unstructured PDF reports rather than clean HTML tables. That's a problem for another weekend.

If you're building your own scraping projects and want a broader look at the Python and web scraping tutorials on Stackademic, that's a reasonable next stop for the mechanics side of this. And if data engineering as a discipline interests you more generally, the structured learning paths on Stackademic cover a lot of adjacent ground, from API design to data pipelines, that this project touched on without ever being the main point.

FAQ

Is it legal to scrape public poker tournament results pages? Generally yes, if the data is publicly accessible and you respect robots.txt and rate limits. Legal risk rises with ToS violations or scraping behind logins. Check the specific site's terms and consider caching aggressively to minimize request volume.

Why use Python instead of just doing this in Next.js with server-side fetch? Python's parsing libraries (BeautifulSoup, lxml) handle messy HTML better than anything in the JS ecosystem I've tried. Next.js is great for the dashboard, not for the scraping layer itself.

Does higher variance mean a tournament is rigged or unfair? No. Variance in payout multipliers reflects structure design (how top-heavy the prize pool is), not fairness. Softer, low-stakes fields often pay first place disproportionately to keep recreational players engaged, which is a design choice, not a red flag.

Can this kind of scraper replace a poker strategy course or coach? No, and it was never meant to. It answers descriptive questions (what happened across a sample) not prescriptive ones (what should you do in a specific hand). Strategy resources and coaching fill a different, necessary gap.

What's the hardest part of a project like this for someone starting out? Data normalization, not scraping. Getting HTML off a page is mechanical. Deciding how to compare a $200 buy-in event against a $2,000 one in a way that's actually meaningful takes more thought than the code itself.

Building Data Tools for Games You Actually Play

The best side projects come from friction you feel yourself, not problems you read about. I built this because I wanted a number instead of a gut feeling, and got one. It's not a betting system, and I'm not pretending the chart tells you what to do at the table. What it does is separate the two problems cleanly: here's what the structures actually pay, and here's a completely different question about what you do with that stack tonight.

Gambling involves risk. Please play responsibly and only wager what you can afford to lose. If you feel gambling is becoming a problem, visit BeGambleAware.org or call 1-800-GAMBLER.

Comments

Loading comments…