← Back to Dashboards

⛳ GOLF — Bonus Scraper System

Web scraper for bonus data extraction with SQLite storage, proxy rotation, FTS search, and two React viewer apps

Python React SQLite FTS5 Proxy Rotation CSV Export
📁
1,538
Total Files
📝
139K
Lines of Code
🗄️
29,385
Database Records
📊
4
DB Tables
🧪
24
Test Files
🖥️
2
Viewer Apps
📋
26
Doc Files
📦
138
JS Lines (Viewers)

Database Table Sizes (Records)

Code Breakdown (Lines)

🔄 Data Pipeline Flow
⚙️
Config
57 lines
→
🕷️
Scraper
278 lines
→
🌐
Network
123 lines
→
🗄️
Database
170 lines
→
📤
Export
CSV / FTS
→
👁️
Viewer
React Apps
🗄️ Database Schema — base.db
b
Bonuses
313
records
l
Links / Logs
26,833
records
s
Sources / Sessions
934
records
t
Targets
1,305
records
🏗️ Scraper Architecture
⚙️
Configuration
config.py
57 lines
Central config for delays, proxy lists, user agents, max withdrawals, and scraper parameters.
Delays Proxies User Agents Limits
🌐
Network Layer
network.py
123 lines
HTTP requests with proxy rotation, AbuseIPDB checking, and request throttling.
Proxy Rotation AbuseIPDB Throttling
🗄️
Database Layer
db.py
170 lines
SQLite with FTS5 full-text search, CRUD operations across 4 tables, and data integrity checks.
SQLite FTS5 Search CRUD Integrity
🕷️
Core Scraper
scraper.py + main.py
314 lines
Orchestrates scraping flow, parses bonus data from multiple sources (Dayne, Sarah).
Parsing Multi-source Orchestration
🖥️ Viewer Applications
🏆
Dayne Bonuses Viewer
React · 138 lines JS
Bonus Data Display FTS5 Search Filtering Responsive Dark Theme Quick Stats
📊
Sarah Bonuses Viewer
React · 138 lines JS
Bonus Data Display FTS5 Search Date Sorting Responsive Dark Theme Export Ready
🔧 Configuration System
🕐 Timing
  • Configurable request delays
  • Throttling between proxies
  • Session timeouts
  • Retry backoff intervals
🔄 Proxy Rotation
  • Multiple proxy list support
  • Automatic failover
  • AbuseIPDB verification
  • Rotation strategies
📋 Data Limits
  • Max withdrawals cap
  • Pagination depth
  • Source-specific limits
  • Duplicate detection
🎭 Critique System
AI Code Review Panel
45 characters across 16 domains
45
Critique Characters
16
Review Domains
Multi-Source
Scraper Coverage
📅 Activity Timeline
Recent Session
Active Scraping Operations
Multiple log files documenting ongoing scraping activity
📊 29,385 total records
Data Collection
Bonus Data Extraction
Dayne and Sarah bonus sources scraped and indexed
🎯 313 bonus records
Database Growth
Link & Target Indexing
26,833 links and 1,305 targets catalogued with FTS5 indexing
🔗 28,138 link/target records
Export & Analysis
CSV Exports Generated
Compiled URLs, mapped URLs, and bonus data exported for analysis
📤 Multiple CSV files
Development
Core System Built
745 lines of core Python across 7 files with proxy rotation and FTS5 search
🐍 745 lines core Python
UI Development
React Viewer Apps
Dayne and Sarah bonus viewer apps built with FTS5 search integration
⚛️ 2 React apps
Quality
Testing & Documentation
24 test files and 26 documentation files maintaining code quality
📝 50 test/doc files

Core Python Breakdown (Lines)

Project Composition (Files)