[BUG] json-export silently drops findings: dedupe hash omits port

Is there an existing issue for this?

• I have searched the existing issues.

Current Behavior

-je / -jsonl-export / -sarif-export / -markdown-export / issue trackers drop findings that -jsonl / -o report, non-deterministically.

There are two independent issues here:

A is the data loss. B is only why it's spotty rather than consistently broken.

Defect A — dedupe key ignores the port

pkg/reporting/dedupe/dedupe.go:75-102 hashes TemplateID | MatcherName | ExtractorName | Type | Host | Matched | ExtractedResults | Metadata. ResultEvent.Host is the hostname only (no port). So when two targets on the same host redirect to a common origin, Matched converges and both events hash identically — one is discarded.

http://host:80 + https://host:443, where :80 → 301 → :443:

-jsonl has 2 rows; -je has 1. This part is deterministic and always wrong: the two events are genuinely two findings on two different ports, and the key cannot tell them apart.

Note that a template whose extracted-results happen to differ between the two ports is not affected, because those bytes are in the hash. In our data missing-cookie-samesite-strict survived on both ports only because it extracted a different session cookie each time — which is what confirmed the key for us.

Defect B — Index is check-then-act, unlocked

Storage.Index does storage.Has(hash) then storage.Put(hash) (dedupe.go:104-112), and ReportingClient.CreateIssue (reporting.go:289-307) calls it without a lock from concurrent template goroutines. So whichever event lands first wins, and if both land inside the window neither is culled.

This is what makes the symptom intermittent. The :80 target carries an extra redirect hop, which usually spaces the two events far enough apart for the second to be culled; when they land together, both survive.

Live, 10 consecutive identical runs, one template (gitlab-detect), two targets (http://gitlab.example.com:80, https://gitlab.example.com:443):

run  1: jsonl=2  json-export=2  ports=[80,443]  <- both kept (events landed together)
run  2: jsonl=2  json-export=1  ports=[80]      <- :443 dropped
run  3: jsonl=1  json-export=1  ports=[443]     <- only 1 event generated (unrelated)
run  4: jsonl=2  json-export=1  ports=[80]      <- :443 dropped
run  5: jsonl=2  json-export=1  ports=[443]     <- :80  dropped
run  6: jsonl=2  json-export=1  ports=[443]     <- :80  dropped
run  7: jsonl=2  json-export=1  ports=[443]     <- :80  dropped
run  8: jsonl=2  json-export=1  ports=[80]      <- :443 dropped
run  9: jsonl=2  json-export=1  ports=[443]     <- :80  dropped
run 10: jsonl=2  json-export=1  ports=[80]      <- :443 dropped

Of the 9 runs that produced two events: 8 lost one of them in -je. Which one survived was an even split — :443 4 times, :80 4 times. -jsonl was correct in all 10.

Concurrency proven in isolation — two identical events, Index called from two goroutines, 200 trials:

both reported unique (dedupe leaked): 198/200
one dropped:                            2/200

This is almost certainly the cause of #4371 (-json-export is not saving all the detected findings, closed as abandoned, no root cause).

Expected Behavior

-je and -jsonl contain the same findings. Two distinct scanned origins (:80 and :443) are two findings, not one, regardless of where they redirect to (fixes A). Output is identical across identical runs (fixes B).

Steps To Reproduce

# two ports on one host; :8080 redirects to :8081
cat > server.py <<'EOF'
import http.server, threading
class Redir(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(301)
        self.send_header("Location", "http://127.0.0.1:8081" + self.path)
        self.send_header("Content-Length", "0"); self.end_headers()
    def log_message(self, *a): pass
class App(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        b = b"REPRO_CANARY_TOKEN"
        self.send_response(200); self.send_header("Content-Length", str(len(b)))
        self.end_headers(); self.wfile.write(b)
    def log_message(self, *a): pass
threading.Thread(target=http.server.ThreadingHTTPServer(("127.0.0.1",8080), Redir).serve_forever, daemon=True).start()
http.server.ThreadingHTTPServer(("127.0.0.1",8081), App).serve_forever()
EOF
python3 server.py &

cat > repro.yaml <<'EOF'
id: dedupe-repro
info:
  name: dedupe repro
  author: repro
  severity: info
http:
  - method: GET
    path:
      - "{{BaseURL}}/canary"
    redirects: true
    max-redirects: 2
    matchers:
      - type: word
        words:
          - "REPRO_CANARY_TOKEN"
EOF

printf 'http://127.0.0.1:8080\nhttp://127.0.0.1:8081\n' > targets.txt

for i in 1 2 3 4 5; do
  nuclei -list targets.txt -t repro.yaml -jsonl -o o.jsonl -je e.json -silent -duc >/dev/null 2>&1
  echo "run $i: jsonl=$(wc -l < o.jsonl) json-export=$(jq length e.json)"
done

Relevant log output

# -jsonl  (2 rows, correct)
{"template-id":"dedupe-repro","host":"127.0.0.1","port":"8081","url":"http://127.0.0.1:8081","matched-at":"http://127.0.0.1:8081/canary"}
{"template-id":"dedupe-repro","host":"127.0.0.1","port":"8080","url":"http://127.0.0.1:8080","matched-at":"http://127.0.0.1:8081/canary"}

# -je  (1 row, :8080 silently dropped)
[{"template-id":"dedupe-repro","host":"127.0.0.1","port":"8081","url":"http://127.0.0.1:8081","matched-at":"http://127.0.0.1:8081/canary"}]

Environment

nuclei: v3.9.0 (also present on dev @ ffbb05b — dedupe.go unchanged)
OS: macOS 15 (darwin/arm64)
Go: 1.27.0

Anything else?

Suggested fix — pkg/reporting/dedupe/dedupe.go. The three hasher.Write additions are defect A; the mutex is defect B:

@@ type Storage struct
 	temporary string
 	storage   *leveldb.DB
+	mu        sync.Mutex
 
@@ func (s *Storage) Index
 	if result.Host != "" {
 		_, _ = hasher.Write(conversion.Bytes(result.Host))
 	}
+	if result.Port != "" {
+		_, _ = hasher.Write(conversion.Bytes(result.Port))
+	}
+	if result.Scheme != "" {
+		_, _ = hasher.Write(conversion.Bytes(result.Scheme))
+	}
+	if result.URL != "" {
+		_, _ = hasher.Write(conversion.Bytes(result.URL))
+	}
 	if result.Matched != "" {
 		_, _ = hasher.Write(conversion.Bytes(result.Matched))
 	}
@@
 	hash := hasher.Sum(nil)
 
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
 	exists, err := s.storage.Has(hash, nil)

Verified against the patch: the concurrency probe goes 198/200 leaked → 200/200 correctly unique, and the repro above gives jsonl=2 / json-export=2.

Two smaller issues in the same function, if worth folding in:

• No field separators. Fields are concatenated straight into the hash, so (TemplateID="ab", MatcherName="c") and (TemplateID="a", MatcherName="bc") collide. A delimiter byte between writes fixes it.
• Metadata is hashed in Go map order (dedupe.go:98), which is randomised, so events carrying >1 payload value (fuzzing / brute-force templates — Metadata = OperatorsResult.PayloadValues) hash differently each time and escape dedupe. Measured: 500 byte-identical events with 6 metadata keys → Index reported 6 of them unique. Sorting keys before hashing fixes it.