Refactoring NLTK's Import Structure into a Strict DAG

The Problem

Currently, NLTK’s internal module dependencies form a highly dense and entangled graph. Because we have historically operated with a permissive import structure, modules frequently cross boundaries in ways that create cycles and bidirectional dependencies (e.g., base utilities importing from high-level applications).

This tangled graph makes the codebase harder to maintain, increases the risk of regressions during targeted updates, and slows down initial import times.

The Proposed Solution

Our primary goal is to untangle this structure and transform NLTK's module dependencies into a true Directed Acyclic Graph (DAG).

I propose we achieve this by shifting to a strictly restrictive import policy. By defining clear boundaries between our core layers and enforcing them in the CI pipeline using import-linter, we can guarantee the graph remains acyclic.

Because NLTK has a well-established history of accepting necessary breakage for long-term health—especially heading into major version bumps—we have the opportunity to execute a true structural rewrite rather than a multi-year deprecation cycle.

However, a monolithic "rewrite the world" PR would be unreviewable and cause severe merge conflict fatigue. Instead, I propose we execute this in four aggressive, rapid-fire phases to keep PRs scoped entirely to file movements and import shifts.

The Phased Roadmap

• Phase 1: Purify the Base (nltk.util, nltk.data) Strip the absolute bottom of the hierarchy of any upward dependencies. Base utilities and core data loading mechanisms must not import from higher-level processing modules.
• Phase 2: Isolate the Data Layer (nltk.corpus and readers) Ensure that corpus readers strictly import from the base layer. This prevents low-level file-reading mechanisms from pulling in heavy NLP components.
• Phase 3: Linearize the Processing Pipeline (nltk.tokenize, nltk.tag, nltk.parse) Establish a strict directional flow: parsers can import taggers, and taggers can import tokenizers, but never the reverse. Any shared logic creating cycles here will be extracted into independent modules.
• Phase 4: Top-Level Apps and Total Lockdown (nltk.app, nltk.metrics) Restructure the highest-level interactive tools. Once complete, we finalize the import-linter contract in our YAML workflows to fail any future PR that violates this acyclic architecture.

Discussion

To minimize the synchronization tax and branch drift on main, we would need to review and merge these phases in relatively quick succession. Do we have the bandwidth to target this for a future major release?