使用 GPT 从 Reddit 中搜集营销痛点信息 — 专为推动 Cronlytic 的增长而打造。
Built to uncover hidden marketing signals on Reddit — and help power smarter growth for Cronlytic.com Click the thumbnail above to watch a full explainer — why I built this tool, how it works, and how you can use it to automate Reddit lead generation using GPT-4.
A Python application that scrapes Reddit for potential marketing leads, analyzes them with GPT models, and identifies high-value opportunities. Includes an interactive Streamlit dashboard for browsing and filtering results.
This tool uses a combination of Reddit's API and AI models (OpenAI or Anthropic) to:
The application maintains a balance between focused and exploratory subreddits, intelligently refreshing the exploratory list based on discoveries. This exploration process happens automatically as part of the main workflow.
config.yaml)Clone the repository:
git clone https://github.com/Mohamedsaleh14/Reddit_Scrapper.git
cd Reddit_ScrapperCreate a virtual environment:
python3 -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activateInstall dependencies:
pip install -r requirements.txtSet up environment variables by copying .env.template to .env:
cp .env.template .envEdit .env and add your API credentials:
REDDIT_CLIENT_ID=your_client_id
REDDIT_CLIENT_SECRET=your_client_secret
REDDIT_USER_AGENT=script:cronlytic-reddit-scraper:v1.0 (by /u/yourusername)
OPENAI_API_KEY=your_openai_api_key
ANTHROPIC_API_KEY=your_anthropic_api_keyConfigure the application by editing config/config.yaml. Key settings include:
openai or anthropic as the batch processing backendTo run the pipeline once:
python3 main.pyThis will:
To run the pipeline daily at the configured time (TODO, Fix scheduler):
python3 scheduler/daily_scheduler.pyAfter running the pipeline at least once, you can explore the results using the interactive Streamlit dashboard:
./run_gui.shOr run it directly:
streamlit run gui/gui.py --server.port 8501 --server.address localhostThe dashboard provides:
Results are stored in a SQLite database at data/db.sqlite. Besides the GUI, you can query it directly:
-- Today's top leads
SELECT * FROM posts
WHERE processed_at >= DATE('now')
ORDER BY roi_weight DESC, relevance_score DESC
LIMIT 10;
-- Posts with specific tag
SELECT * FROM posts
WHERE tags LIKE '%serverless%'
ORDER BY processed_at DESC;…The application includes several safeguards to control API costs:
config.yaml)| Feature | Status | Notes |
|---|---|---|
| Reddit Scraping (Posts & Comments) | ✅ Done | Age-filtered, deduplicated, tracked via history table |
| Primary & Exploratory Subreddit Logic | ✅ Done | With refreshable exploratory_subreddits.json |
| GPT Filtering | ✅ Done | Via batch API, scoring + threshold-based selection |
| GPT Insight Extraction | ✅ Done | With batch API, structured JSON, ROI + tags |
| SQLite Local DB Storage | ✅ Done | Full schema, type handling (post/comment) |
| Rate Limiting | ✅ Done | Real limiter applied to avoid Reddit bans |
| Budget Control | ✅ Done | Tracks monthly cost, blocks over-budget batches |
| Daily Runner Pipeline | ✅ Done | Logs step-by-step, fail-safe batch handling |
| Anthropic Batch API Provider | ✅ Done | Full alternative to OpenAI with config-based switching |
| Parallel Batch Processing | ✅ Done | Token-aware scheduling, partial result recovery |
| Technical Depth Scoring | ✅ Done | Measures engineering complexity and defensibility |
| Implementability Scoring | ✅ Done | Feasibility assessment with willingness-to-pay signals |
| OpenAI Storage Cleanup | ✅ Done | Auto-cleans accumulated batch files before each run |
| Cached Summaries → GPT Discovery | ✅ Done | Based on post text, fallback if prompt fails |
| Comment scraping toggle | ✅ Done | Controlled via config key (include_comments) |
| Retry on GPT Batch Failures | ✅ Done | With exponential backoff and item-level retry |
| Streamlit GUI Dashboard | ✅ Done | Filter, sort, browse, and analyze results visually |
| Feature | Status | Suggestion |
|---|---|---|
| Parallel subreddit fetching | Manual (sequential) | Consider async/threaded fetch in future |
| Tagged CSV Export / CLI | Missing | Useful for non-technical review/debug |
| Multi-language / non-English handling | Not supported | Detect & skip or flag for English-only use |
| Unit tests / mocks | Not present | Add test coverage for scoring and DB logic |
Thanks to the following people who have contributed to this project:
| Contributor | Contributions |
|---|---|
| @Mohamedsaleh14 | Creator & maintainer |
| @Dieterbe | Bug fixes, prompt system refactoring, enhanced logging, GUI, batch optimization, and many quality-of-life improvements |
| Claude Code | AI pair programmer — code implementation, issue triage, and PR integration |
This tool was created as part of the growth strategy for Cronlytic.com — a serverless cron job scheduler designed for developers, indie hackers, and SaaS teams.
If you're building something and want to:
Check out Cronlytic — and let us know what you'd love to see.
This project is open source for personal and non-commercial use only. Commercial use (including hosting it as a backend or integrating into products) requires prior approval.
See the LICENSE file for full terms.
This project uses open source libraries, which are governed by their own licenses:
Use of this project must also comply with these third-party licenses and terms.
暂无开放 Issues,或尚未同步最近议题。