RTP data is hiding in plain sight. Every licensed online pokies platform publishes return-to-player percentages somewhere on their site. Buried in game info modals, help pages, or paytables. And almost nobody aggregates it. That gap is exactly what makes it a great real-world scraping project. The data is structured enough to be learnable, messy enough to be realistic, and publicly available enough that you won’t need to fight an API paywall on day one.
Before writing a single line of pandas, it’s worth understanding what the data actually represents. A guide to online pokies real money from The Sun Papers breaks down RTP ranges, volatility tiers, and how platform operators structure their game libraries, which maps cleanly to the columns you’ll build your DataFrame around. Read it once before you open your IDE. Knowing the domain saves you from building the wrong schema.
This tutorial walks through a complete pipeline: fetch HTML with `requests`, parse it with `BeautifulSoup`, load the results into a `pandas` DataFrame, clean the percentage strings, and run a basic statistical summary. By the end you’ll have a reusable script and a rough template you can point at any pokies catalogue page.
What You’re Actually Measuring
RTP stands for return-to-player. A 96.5% RTP means the game returns $96.50 for every $100 wagered over a statistically significant number of spins. The operative phrase is “over a statistically significant number”. Short sessions deviate wildly from the theoretical figure. That’s volatility, the second column in your dataset.
Volatility (or variance) describes how that 96.5% is distributed. High-volatility slots pay infrequently but large. Low-volatility slots pay small amounts often. Both can have identical RTPs. From a data perspective, RTP without volatility is half the picture.
The columns you want in your final DataFrame:
- `game_name`. String
- `provider`. String (NetEnt, Pragmatic Play, IGT, etc.)
- `rtp_pct`. Float64
- `volatility`. Categorical: Low / Medium / High
- `max_win_x`. Float64 (max win as a multiplier of stake)
Not every source publishes all five. That’s fine. You’ll handle missing values in the cleaning step.
Environment Setup
Python 3.12 or later. The pipeline runs on 3.11 too, but Python 3.15 is currently in feature-freeze beta and its lazy import system will make the initial `import pandas` noticeably snappier once it ships. Worth keeping in mind if you plan to scale this.
Install your dependencies:
“`bash pip install requests beautifulsoup4 pandas lxml “`
`lxml` is the faster HTML parser. `html.parser` (stdlib) works but chokes on malformed markup more often, and pokies pages are frequently auto-generated with inconsistent tag nesting.
Step 1. Fetch the Page
“`python import requests from bs4 import BeautifulSoup
HEADERS = { “User-Agent”: ( “Mozilla/5.0 (Windows NT 10.0; Win64; x64) ” “AppleWebKit/537.36 (KHTML, like Gecko) ” “Chrome/125.0.0.0 Safari/537.36” ) }
def fetch_page(url: str) -> BeautifulSoup: response = requests.get(url, headers=HEADERS, timeout=10) response.raise_for_status() return BeautifulSoup(response.text, “lxml”) “`
A few things worth noting. The `User-Agent` string matters. Requests with the default Python UA get blocked by Cloudflare on roughly 60% of gaming-adjacent sites. The one above mimics Chrome 125 on Windows and gets through most static pages without triggering a challenge. The `timeout=10` prevents the script hanging indefinitely on a slow server. And `raise_for_status()` throws an `HTTPError` on 4xx/5xx responses rather than silently returning broken HTML. Catch it upstream if you’re looping over many URLs.
For a deeper look at what’s happening under the hood when `requests` fetches a page, Real Python’s guide to building a web scraper with Beautiful Soup covers the full request/parse cycle in detail.
Step 2. Parse the Game Table
This is where projects diverge. Every site structures its game metadata differently. Some put RTP in a `<td>` inside a `<table>`. Others use `<div>` cards with `data-rtp` attributes. A few bury it in a `<script>` tag as JSON.
Here’s a parser that handles the most common pattern. A `<table>` with headers:
“`python def parse_rtp_table(soup: BeautifulSoup) -> list[dict]: records = [] table = soup.find(“table”, {“class”: lambda c: c and “game” in c.lower()}) if not table: return records
headers = [th.get_text(strip=True).lower() for th in table.find_all(“th”)]
for row in table.find_all(“tr”)[1:]: # skip header row cells = row.find_all(“td”) if len(cells) != len(headers): continue record = {headers[i]: cells[i].get_text(strip=True) for i in range(len(headers))} records.append(record)
return records “`
If you hit a site that uses `data-*` attributes instead of table rows, swap the `find_all(“tr”)` loop for:
“`python cards = soup.find_all(“div”, attrs={“data-rtp”: True}) records = [{“game_name”: c.get(“data-title”, “”), “rtp_pct”: c.get(“data-rtp”, “”)} for c in cards] “`
Neither approach is universal. You’ll need to inspect the target site’s DOM in DevTools before committing to either pattern. That five-minute inspection step saves hours of debugging.
Step 3. Load Into pandas and Clean
“`python import pandas as pd import re
def build_dataframe(records: list[dict]) -> pd.DataFrame: df = pd.DataFrame(records)
Normalise column names
df.columns = ( df.columns .str.strip() .str.lower() .str.replace(r”[^a-z0-9]+”, “_”, regex=True) )
Convert RTP strings like ‘96.5%’ or ‘96.50 %’ to float
if “rtp” in df.columns: df[“rtp_pct”] = ( df[“rtp”] .str.extract(r”(d+.?d*)”) .astype(float) ) df.drop(columns=[“rtp”], inplace=True)
Coerce volatility to a proper categorical
if “volatility” in df.columns: valid = [“low”, “medium”, “high”] df[“volatility”] = ( df[“volatility”] .str.lower() .where(df[“volatility”].str.lower().isin(valid)) ) df[“volatility”] = pd.Categorical( df[“volatility”], categories=valid, ordered=True )
Drop rows where RTP is missing or implausible
df = df.dropna(subset=[“rtp_pct”]) df = df[(df[“rtp_pct”] > 70) & (df[“rtp_pct”] <= 100)]
return df “`
The RTP filter `> 70` catches garbage entries. Occasionally a site publishes a “bonus contribution” percentage in the same column and you end up with a row claiming 20% RTP. That’s not a pokie, that’s a parsing error. The `<= 100` ceiling handles the rare case where a site formats RTP as a ratio (1.965) instead of a percentage.
For a thorough breakdown of the pandas methods used here. `str.extract`, `pd.Categorical`, `dropna`. Towards Data Science has a solid walkthrough of data analysis with pandas that covers exactly this kind of cleaning workflow.
Step 4. Analyse the Distribution
“`python def summarise(df: pd.DataFrame) -> None: print(“=== RTP Summary ===”) print(df[“rtp_pct”].describe().round(2)) print()
if “volatility” in df.columns: print(“=== Mean RTP by Volatility ===”) print( df.groupby(“volatility”, observed=True)[“rtp_pct”] .agg([“mean”, “count”]) .round(2) ) print()
if “provider” in df.columns: print(“=== Top 10 Providers by Mean RTP ===”) print( df.groupby(“provider”)[“rtp_pct”] .mean() .sort_values(ascending=False) .head(10) .round(2) ) “`
On a catalogue of 400+ games, a typical output looks something like this:
“` === RTP Summary === count 412.00 mean 96.21 std 1.43 min 92.00 25% 95.50 50% 96.20 75% 97.00 max 99.07 “`
The standard deviation around 1.4% is narrower than most people expect. The difference between the 25th and 75th percentile. 95.5% to 97%. Is where almost all catalogue variation lives. A game at 94% isn’t dramatically worse than one at 97% in expected-value terms across a typical session, but it’s meaningfully worse over tens of thousands of spins. That’s the insight you’d never surface just by reading individual game pages.
Step 5. Putting It Together
“`python if __name__ == “__main__”: TARGET_URL = “https://example-pokies-catalogue.com/games” # swap in your target
soup = fetch_page(TARGET_URL) records = parse_rtp_table(soup)
if not records: print(“No records parsed — inspect the page DOM and update the parser.”) else: df = build_dataframe(records) print(f”Loaded {len(df)} games after cleaning.”) summarise(df)
Optional: save to CSV for further analysis
df.to_csv(“pokies_rtp.csv”, index=False) print(“Saved to pokies_rtp.csv”) “`
From here you can pipe the CSV into a Jupyter notebook, load it into a Streamlit dashboard, or push it to a database. If you’re working with hundreds of catalogue pages and performance becomes a concern, the pipeline translates well to Polars with minimal refactoring. The API is intentionally pandas-adjacent.
Where This Breaks (and What to Do)
Four failure modes show up consistently on pokies pages.
JavaScript-rendered content. If `BeautifulSoup` returns an empty table even though the browser shows data, the page is rendering client-side. Swap `requests` for `playwright` or `selenium` to get the post-render HTML. This adds complexity but is unavoidable on modern single-page app builds.
Rate limiting. Hit a site 50 times in 30 seconds and you’ll see a 429. Add `time.sleep(1.5)` between requests and randomise the delay slightly: `time.sleep(random.uniform(1.0, 2.5))`. Polite scraping.
Dynamic class names. Some sites generate CSS class names at build time (`game-card-a3f7b` today, `game-card-9c2d1` tomorrow). Don’t rely on class names as primary selectors. Prefer structural selectors or `data-*` attributes where available.
RTP not published. Plenty of platforms don’t expose RTP at the catalogue level, only inside individual game detail pages. That means one HTTP request per game rather than one per page. Still doable, just slower and more polite-scraping-critical.
FAQ
Is it legal to scrape publicly available RTP data from pokies sites? Generally yes, if you’re accessing publicly visible pages without bypassing authentication. Most jurisdictions treat publicly accessible HTML as fair game for personal or research use. Check the site’s Terms of Service before large-scale crawls. Some explicitly prohibit automated access, and a Cease and Desist is an annoying interruption to a side project.
Why use `requests` and `BeautifulSoup` rather than `Scrapy`? `Scrapy` is better for large-scale, multi-domain crawls where you need async fetching and a built-in pipeline. For a single-site RTP scraper with under a few thousand records, `requests` and `BeautifulSoup` are faster to write and easier to debug. Scale up to `Scrapy` if your catalogue exceeds roughly 5,000 game pages or if you’re running regular scheduled crawls.
How do I handle sites that require login to see RTP data? Use `requests.Session()` to persist cookies across requests and send a POST to the login endpoint first. Extract the CSRF token from the login page HTML before submitting credentials. This works on standard form-based auth. OAuth or SSO-gated content needs a different approach.
My RTP column is coming back as NaN for half the rows. What’s wrong? Almost always a formatting inconsistency. Some rows use `96.5%`, others use `96,5%` (comma decimal separator), others just `96`. The `str.extract(r”(d+.?d*)”)` regex won’t catch the comma variant. Add a `.str.replace(“,”, “.”)` before the extract step.
Can I use this pipeline to compare RTPs across multiple casinos? Absolutely. Wrap `fetch_page` and `parse_rtp_table` in a loop over a list of URLs, add a `source_url` column to each DataFrame before concatenation, then use `pd.concat()` to merge them. `df.groupby(“source_url”)[“rtp_pct”].mean()` gives you a cross-operator comparison in two lines.
From Script to Real Insight
The industry is paying attention to this kind of pipeline. In June 2026, Vyking Ventures backed PlayAIO specifically to accelerate data and AI analytics within iGaming, which signals that Python-powered data work is now a boardroom-level priority in this space, not just a hobbyist exercise. What you’re building here is a smaller version of exactly what those platforms are funding at scale.
The script above is a starting point. The parsing function will need tailoring to your target site. The cleaning thresholds might need adjusting. But the architecture. Fetch, parse, clean, analyse. Holds for any structured data source, pokies catalogue or otherwise.
Gambling involves risk. Play responsibly and only wager what you can afford to lose. If gambling is becoming a problem, visit BeGambleAware.org or call 1-800-GAMBLER.