osint-framework

开源

OSINT 框架: 基于 Python 的开源智能 CLI 框架,类似于 Metasploit。

200 stars0 点赞0 次浏览

工具介绍

OSINT 框架: 基于 Python 的开源智能 CLI 框架,类似于 Metasploit。

Overview

OSIF v2.0 is a complete rewrite of the original CLI tool into a full-stack, API-first investigation platform. It is designed for security researchers, penetration testers, and PI/skip-tracing workflows.

What's new in v2:

FastAPI async backend replacing the Flask CLI
PostgreSQL + SQLAlchemy 2.0 for persistent case and graph storage
arq + Redis task queue for parallel async scan execution
Vue 3 frontend with Cytoscape.js graph visualization
MinIO S3-compatible object storage for evidence files
Case knowledge graph — manual entities, relationships, provenance tracking
11+ OSINT modules with caching, rate limiting, and auto-correlation

Screenshots

Investigation Graph

Scan Execution

Timeline

Integrations

Architecture

…

Services

Service	Image	Port (host)	Purpose
`frontend`	Custom (Nginx + Vue)	3000	Web UI — investigation dashboard
`backend`	Custom (FastAPI)	6000	REST API + WebSocket
`worker`	Custom (arq)	—	Async scan task execution
`console`	Custom (CLI)	—	Metasploit-style REPL (`./osif`)
`postgres`	postgres:16-alpine	5434	Primary database (cases, graph, scans)
`redis`	redis:7-alpine	6381	Task queue + caching + rate limiting
`minio`	minio/minio:latest	9100 / 9101	Evidence file storage (S3-compatible)
`minio-init`	minio/mc:latest	—	One-shot bucket provisioning

Quick Start

Prerequisites

Docker 24+
Docker Compose v2 (bundled with Docker Desktop)

1. Clone the repository

bash

git clone https://github.com/fr4nc1stein/osint-framework osif
cd osif

2. Configure environment

bash

cp backend/.env.example .env

Edit .env and add any API keys you have (all are optional — free-tier modules work without keys):

env

# OSINT API Keys (optional)
SHODAN_API_KEY=
VT_API=
ABUSEIPDB_API_KEY=
TOMBA_API_KEY=
TOMBA_SECRET_KEY=
HUNTER_API_KEY=

3. Start all services

bash

docker-compose -f docker-compose.dev.yml up -d --build

The backend entrypoint automatically:

Waits for PostgreSQL and Redis to be healthy
Runs all Alembic migrations (alembic upgrade head)
Provisions the MinIO osif-evidence bucket
Starts the FastAPI server on port 6000

4. Verify everything is running

bash

docker-compose -f docker-compose.dev.yml ps

Expected output:

NAME                STATUS              PORTS
osif_postgres       Up (healthy)        0.0.0.0:5434->5432/tcp
osif_redis          Up (healthy)        0.0.0.0:6381->6379/tcp
osif_minio          Up (healthy)        0.0.0.0:9100->9000/tcp, 0.0.0.0:9101->9001/tcp
osif_minio_init     Exited (0)
osif_backend        Up                  0.0.0.0:6000->6000/tcp
osif_worker         Up
osif_frontend       Up                  0.0.0.0:3000->80/tcp
osif_console        Up

5. Open the UI

Interface	URL
Investigation Dashboard	http://localhost:3000
API (Swagger docs)	http://localhost:6000/api/docs
MinIO Console	http://localhost:9101

Database Tables

Migrations run automatically on startup. Current schema (Phase 3.4):

cases               — Investigation containers
scans               — OSINT scan jobs
indicators          — Graph nodes (discovered data)
edges               — Graph edges (discovered relationships)
scan_findings       — Scan ↔ edge association
scan_templates      — Reusable module presets
case_entities       — Manual graph nodes (PI workspace)
case_relationships  — Manual graph edges
case_evidence       — Evidence file attachments
case_evidence_links — Evidence ↔ node association
alembic_version     — Migration state

OSINT Modules

11 modules ship out of the box. All modules run in parallel via the arq worker.

#	Module ID	Category	API Key	Cache	Rate Limit
1	`dns_records`	domain	—	✅	—
2	`subdomain_enum`	domain	—	✅	—
3	`whois_lookup`	domain	—	✅	—
4	`email_hunter`	domain	✅	✅	tomba
5	`virustotal_domain`	domain	✅	✅	4 req/min
6	`urlscan_lookup`	domain	—	✅	1 req/2s
7	`ip_geolocation`	ip	—	✅	—
8	`abuseipdb`	ip	✅	✅	1000/day
9	`shodan_lookup`	ip	✅	✅	1 req/s
10	`email_domain`	email	—	✅	—
11	`hibp_breach`	email	—	✅	—

Configuration

Environment Variables

All configuration is managed via .env at the project root. The file is loaded by both backend and worker services via env_file.

…

For production: Change SECRET_KEY, MINIO_ROOT_PASSWORD, and the database password. Set DEBUG=false.

MinIO Credentials

Default MinIO credentials (change for production):

Setting	Default
Root user	`osif_minio`
Root password	`osif_minio_password`
Bucket	`osif-evidence`
Console URL	http://localhost:9101

API Reference

Base URL: http://localhost:6000

Health

GET  /health        — Liveness probe
GET  /ready         — Readiness probe (DB + Redis)

Cases

POST   /api/v1/cases                   — Create case
GET    /api/v1/cases                   — List cases
GET    /api/v1/cases/{id}              — Get case
PUT    /api/v1/cases/{id}              — Update case
DELETE /api/v1/cases/{id}              — Delete case

Scans

POST   /api/v1/scans                   — Launch scan (enqueues to arq)
GET    /api/v1/scans                   — List scans
GET    /api/v1/scans/{id}              — Get scan + progress
GET    /api/v1/scans/{id}/graph        — Get scan graph (nodes + edges)

Case Graph (PI Workspace)

…

Evidence

GET    /api/v1/cases/{id}/evidence     — List evidence
POST   /api/v1/cases/{id}/evidence     — Link URL/note as evidence
POST   /api/v1/cases/{id}/evidence/upload — Upload file (stored in MinIO)
DELETE /api/v1/cases/{id}/evidence/{eid}  — Delete evidence

Modules & Templates

GET    /api/v1/modules                 — List all OSINT modules
GET    /api/v1/modules/{id}            — Get module details
POST   /api/v1/templates               — Create scan template
GET    /api/v1/templates               — List templates
GET    /api/v1/templates/{id}          — Get template
DELETE /api/v1/templates/{id}          — Delete template

Export

GET    /api/v1/export/{scan_id}/json   — Export scan as JSON
GET    /api/v1/export/{scan_id}/csv    — Export as CSV (edges or nodes)
GET    /api/v1/export/{scan_id}/graphml — Export as GraphML (Gephi/Cytoscape)

WebSocket

WS     /ws/scan/{scan_id}             — Real-time scan events

Event types: module_start, discovery, module_complete, module_error, scan_complete, auto_linker_complete

Interactive docs: http://localhost:6000/api/docs

Scan Execution Flow

POST /api/v1/scans
        ↓
   Tasks enqueued (one arq job per module)
        ↓
   For each module (parallel):
     1. Check rate limit (Redis)
     2. Check result cache (Redis)
     3. Execute module against target
     4. Cache results
     5. Build graph (upsert indicators + edges)
     6. Publish WebSocket events
        ↓
   All modules complete
        ↓
   Auto-linker runs:
     - Finds corroborations (same relationship from 2+ modules)
     - Infers transitive relationships
     - Publishes correlation stats
        ↓
   Scan status → completed

CLI Console

The console service gives you the original Metasploit-style REPL, now API-connected:

bash

docker exec -it osif_console ./osif

…

Shodan Attack Surface

Email Hunter (Hunter.io & Tomba)

GeoWifi Hunter

Development

Local backend + frontend dev server

bash

# Start infrastructure only
docker-compose -f docker-compose.dev.yml up -d postgres redis minio minio-init

# Run backend with hot reload
cd backend
pip install -r requirements.txt
alembic upgrade head
uvicorn app.main:app --reload --port 6000

# Run arq worker
arq app.tasks.worker.WorkerSettings

# Run frontend dev server (in a separate terminal)
cd frontend
npm install
npm run dev
# → http://localhost:5173

Rebuild after code changes

bash

# Rebuild and restart all services
docker-compose -f docker-compose.dev.yml up -d --build

# Or restart a single service
docker-compose -f docker-compose.dev.yml up -d --build backend

View logs

bash

# All services
docker-compose -f docker-compose.dev.yml logs -f

# Specific services
docker-compose -f docker-compose.dev.yml logs -f backend worker

Database access

bash

docker-compose -f docker-compose.dev.yml exec postgres psql -U osif

Run a new migration

bash

# From inside the backend container
docker-compose -f docker-compose.dev.yml exec backend \
  alembic revision --autogenerate -m "describe your change"

# Apply
docker-compose -f docker-compose.dev.yml exec backend \
  alembic upgrade head

Stop and clean up

bash

# Stop all services (keep volumes)
docker-compose -f docker-compose.dev.yml down

# Stop and remove all data volumes
docker-compose -f docker-compose.dev.yml down -v

What's Complete

Phase	Feature Area	Status
1	FastAPI backend, PostgreSQL, 6 modules, Docker	✅
2	arq task queue, WebSocket, graph service, 8 modules	✅
3	Auto-linker, Redis caching, rate limiting, 11 modules, scan templates, export	✅
3.1	Case management UI, case-scoped graph	✅
3.2	Encrypted integration credentials, AI settings, first-run setup	✅
3.3	Timeline, notes, reports	✅
3.4-A	Manual entities and relationships (PI workspace)	✅
3.4-B	Evidence attachments (MinIO file storage)	✅

Troubleshooting

Services not starting

bash

# Check health status
docker-compose -f docker-compose.dev.yml ps

# Check startup logs
docker-compose -f docker-compose.dev.yml logs backend
docker-compose -f docker-compose.dev.yml logs minio

Database migration errors

bash

# Check current migration state
docker-compose -f docker-compose.dev.yml exec backend alembic current

# Apply pending migrations manually
docker-compose -f docker-compose.dev.yml exec backend alembic upgrade head

MinIO bucket not created

bash

# Re-run the init container
docker-compose -f docker-compose.dev.yml run --rm minio-init

Port conflicts

Default host ports used:

Port	Service
3000	Frontend (Nginx)
5434	PostgreSQL
6000	Backend API
6381	Redis
9100	MinIO API
9101	MinIO Console

Change host-side ports in docker-compose.dev.yml if any conflict with existing services.

Module not discovered

bash

# List all discovered modules
curl http://localhost:6000/api/v1/modules | jq '[.[] | .module_id]'

# Check backend logs for import errors
d

Issues· 6 开放

查看全部 Issues 在 GitHub 打开

暂无开放 Issues，或尚未同步最近议题。

> 标签

Python

暂无评论，来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日

最后更新2026年9月18日

分类编程语言

定价开源

> 相关工具

TypeScript

JavaScript 的超集，为前端与全栈提供静态类型

Python

通用编程语言，广泛用于 Web、数据与 AI

Google 推出的简洁高效系统语言

osint-framework

工具介绍

Overview

Screenshots

Investigation Graph

Scan Execution

Timeline

Integrations

Architecture

Services

Quick Start

Prerequisites

1. Clone the repository

2. Configure environment

3. Start all services

4. Verify everything is running

5. Open the UI

Database Tables

OSINT Modules

Configuration

Environment Variables

MinIO Credentials

API Reference

Health

Cases

Scans

Case Graph (PI Workspace)

Evidence

Modules & Templates

Export

WebSocket

Scan Execution Flow

CLI Console

Shodan Attack Surface

Email Hunter (Hunter.io & Tomba)

GeoWifi Hunter

Development

Local backend + frontend dev server

Rebuild after code changes

View logs

Database access

Run a new migration

Stop and clean up

What's Complete

Troubleshooting

Services not starting

Database migration errors

MinIO bucket not created

Port conflicts

Module not discovered

Issues· 6 开放

> 标签

> 讨论区

> 工具信息

> 相关工具