IL9Cast is a pretty lean operation — a single Python app running on Railway that does everything from fetching market data to serving the website. Here's how it all fits together.
The Cloud Setup
We run on Railway, which is a platform-as-a-service that deploys straight from GitHub. Every time we push code to the main branch, Railway automatically builds and deploys a new version using Nixpacks (their build system). The whole deploy cycle takes about 30 seconds.
The app runs behind Gunicorn, a production-grade Python WSGI server. We use the --preload flag, which is important — it loads the app once in memory before forking workers, so our background data collector only starts a single thread instead of duplicating itself across workers. Without that flag, you'd get multiple scrapers all writing to the same file at the same time, which is a recipe for corrupted data.
Here's the key configuration from railway.toml:
startCommand = "gunicorn app:app --preload"
healthcheckPath = "/"
restartPolicyType = "ON_FAILURE"
restartPolicyMaxRetries = 10
That restart policy is our safety net. If the app crashes — maybe a dependency breaks, maybe Railway has a hiccup — it'll automatically restart up to 10 times before giving up. In practice, it almost never needs more than one.
Persistent Storage (the tricky part)
Railway containers are ephemeral by default — when the app redeploys, everything on the filesystem gets wiped. That's fine for code, but our historical data needs to survive between deploys. So we use a Railway persistent volume mounted at /app/data.
When the app boots up, it runs a path resolution function that checks, in order: /data, /app/data, then falls back to a local data/ directory. This means the same code works on Railway (where the volume lives at /app/data) and on a developer's laptop (where it just uses the local folder). No environment variables, no conditional imports — just a directory check.
Why JSONL (and not a database)
You might wonder why we're storing data in a flat file instead of PostgreSQL or SQLite. Honestly? For our use case, JSONL is better.
JSONL (JSON Lines) means one JSON object per line. Every 3 minutes, the scraper appends a single line to the file — that's it. No connection pools, no schema migrations, no ORM overhead. The file is human-readable (you can literally tail it to see the latest data), and if a line gets corrupted, every other line is still perfectly valid. Try saying that about a SQLite database after a partial write.
The append operation is actually a bit more careful than a simple file write:
- Write to a temp file first — the new snapshot goes to
historical_snapshots.jsonl.tmp
- Copy existing content + new line — both old data and the new snapshot get written to the temp file
- Atomic replace —
os.replace() swaps the temp file into place in a single filesystem operation. If the app crashes mid-write, either the old file or the new file exists — never a half-written mess
At 480 snapshots per day (~3 KB each), the file grows at roughly 1.4 MB/day. After a full election cycle, we're looking at maybe 70 MB total. That's nothing — your phone has thousands of times more storage. A database would be overkill here.
The Data Collection Loop
Every 3 minutes, a background thread wakes up and runs our collect_market_data() function. Here's what happens in those few hundred milliseconds:
- 1. Fetch Manifold — HTTP GET to their public API. Returns JSON with each candidate's probability (0.0–1.0 scale, which we multiply by 100). Timeout: 10 seconds.
- 2. Fetch Kalshi — HTTP GET to their trade API. Returns an array of markets, each with
last_price, yes_bid, and yes_ask. Same 10-second timeout.
- 3. Normalize names — Manifold calls her "Kat Abughazaleh" and Kalshi calls her "Katheryn Abughazaleh." We strip prefixes, suffixes, and known variations to get a canonical key for matching.
- 4. Aggregate — Apply the 40/42/12/6 weighted formula (see the Markets Aggregation section above).
- 5. Soft normalize — Nudge probabilities 30% toward summing to 100%, so the chart doesn't show the race at 130% or 80% total.
- 6. Spike dampen — Compare to the previous snapshot. If any candidate moved more than ±3 percentage points, clamp the change. This prevents chart artifacts from thin-market Kalshi trades.
- 7. Save — Atomically append to the JSONL file.
If both APIs fail (say Manifold is down and Kalshi returns an error), we skip the snapshot entirely. Bad data is worse than missing data — the chart just won't have a point for that 3-minute window, and the gap detection handles it gracefully.
The Scheduler Problem
Running a background task alongside a web server sounds simple, but there's a subtle gotcha. When you run Flask locally, you get one process — easy. But Gunicorn can spawn multiple worker processes, and each one would try to run its own scheduler. That means two workers = two scrapers = double the data (and double the API calls).
We solve this two ways depending on the environment:
- Local development: Uses APScheduler's
BackgroundScheduler, which runs the job in a background thread within the single Flask process.
- Production (Gunicorn): Detects Gunicorn via
sys.argv[0] and instead spins up a plain threading.Thread in daemon mode. The --preload flag ensures this thread is created once in the master process before workers fork, so only one thread ever exists.
Chart Data Pipeline
When you load the Markets page, your browser hits /api/snapshots/chart?period=1d (or 7d or all). Here's what the server does before sending data back:
- Cache check — We keep a 60-second in-memory cache. If the same period was requested within the last minute, we serve the cached version instantly.
- Load & filter — Read all snapshots from the JSONL file, parse timestamps, sort chronologically, and filter to the requested time window.
- Gap detection — Scan consecutive timestamps for gaps > 2 hours. These become the dashed-line segments on the chart — they represent real outages (Railway restarts, AWS issues), not normal 3-minute intervals.
- EMA smoothing — Run an exponential moving average (alpha = 0.15) across each candidate's probability series. This is the biggest smoothing step — each data point becomes 15% raw value + 85% previous smoothed value, which kills jitter while preserving genuine trends.
- RDP simplification — The Ramer-Douglas-Peucker algorithm finds which points you can remove without changing the visual shape of the line (within an epsilon tolerance of 0.5 percentage points). A week of data at 3-minute intervals = ~3,360 points per candidate. After RDP, that drops to maybe 200–400 points. Your browser thanks us.
The RDP algorithm is actually kind of elegant. Imagine drawing a straight line from the first data point to the last. Now find whichever intermediate point is farthest from that line. If it's farther than epsilon, that point matters — keep it, and recursively check both halves. If it's closer than epsilon, the whole segment is "flat enough" to represent with just the endpoints. It's O(n log n) on average and perfectly preserves peaks, valleys, and inflection points while throwing away the boring flat stretches.
The Frontend Rendering
The chart itself is rendered with Chart.js using a time-scaled x-axis. The data comes in as {x: timestamp, y: probability} pairs, and Chart.js handles the rest. A few important settings:
- Monotone cubic interpolation — This is the
cubicInterpolationMode: 'monotone' setting. Regular cubic splines can "overshoot" — if a candidate goes from 60% to 62%, a regular spline might draw a curve that briefly dips to 59% between the points. Monotone splines guarantee the curve never exceeds the actual data values. No fake dips, no fake peaks.
- Tension 0.5 — Controls how curvy the lines are. At 0 you get straight segments (ugly). At 1 you get maximally curvy (too smooth, hides real movement). 0.5 is the sweet spot.
- Central Time display — All timestamps are stored in UTC but displayed in Central Time (America/Chicago) using
Intl.DateTimeFormat. This handles daylight saving transitions automatically — no hardcoded offsets.
- Segment styling for gaps — Chart.js lets you style individual line segments. For each segment, we check if the two endpoints span a known gap period. If they do, the segment gets dashed and faded. This is done per-frame via a callback, so it works even when you zoom or pan.
What Could Go Wrong (and what we do about it)
A few things have bitten us before, so we built defenses:
- API timeouts: Both Manifold and Kalshi calls have 10-second timeouts. If they're slow, we fail fast instead of blocking the scheduler thread.
- Partial API failure: If only one API goes down, we still collect what we can, but spike dampening prevents the sudden weight shift from creating chart artifacts.
- Railway restarts: The persistent volume survives container restarts. When the app boots, it checks for existing data and picks up where it left off.
- Corrupt JSONL lines: The reader skips unparseable lines and logs a warning. One bad line doesn't take down the whole dataset — that's the beauty of line-delimited formats.
- Duplicate schedulers: The Gunicorn
--preload + sys.argv detection ensures exactly one scraper thread exists in production.
Dependencies
The whole app runs on five Python packages:
Flask 2.3.2 — web framework
Werkzeug 2.3.6 — WSGI utilities (Flask dependency)
Requests 2.31.0 — HTTP client for API calls
Gunicorn 21.2.0 — production WSGI server
APScheduler 3.10.4 — background job scheduling (dev mode)
No NumPy, no Pandas, no heavyweight data libraries. The EMA, RDP, and aggregation math are all hand-written in ~100 lines of plain Python. For a project that processes a few hundred data points, there's no reason to import a 30 MB library.