Core SDK: cross-tenant token-cost collisions, dropped litellm success-callbacks, and NoneType-masked import failures

Summary

In-depth audit of src/praisonai-agents/praisonaiagents (the Core SDK layer) against this project's own stated MUSTs — "multi-agent + async safe by default" and "production-ready... safe by default". All three findings below were confirmed by reading the source at the cited lines and by running a standalone reproduction against the actual current code (commit 43a106d), not inferred from description alone. Repro scripts are included in each section.

I searched existing issues first. This repo has an extensive history of similar audits (e.g. #3977, #5025, #4937, #1482, #1159). None of them cover the three mechanisms below — I specifically checked and ruled out overlap (see Validation notes).

This report excludes documentation, tests, coverage, file size, and generic performance tuning — only behavioral/architectural gaps with concrete failure scenarios.

1. Process-wide TokenCollector aggregates cost/usage by bare agent name — two unrelated agents with the same name merge irreversibly into one cost bucket

Where: praisonaiagents/telemetry/token_collector.py:69-85 (SessionTokenMetrics.add_interaction), consumed via the module-level singleton _token_collector (token_collector.py:193), and "scoped" for reporting in praisonaiagents/agents/agents.py:3189-3271.

# telemetry/token_collector.py:69-85
def add_interaction(self, model: str, agent: Optional[str], metrics: TokenMetrics):
    self.total_interactions += 1
    self.total_metrics = self.total_metrics + metrics
    if model not in self.metrics_by_model:
        self.metrics_by_model[model] = TokenMetrics()
    self.metrics_by_model[model] = self.metrics_by_model[model] + metrics
    if agent:
        if agent not in self.metrics_by_agent:
            self.metrics_by_agent[agent] = TokenMetrics()
        self.metrics_by_agent[agent] = self.metrics_by_agent[agent] + metrics   # keyed by bare name

agents/agents.py later tries to scope get_token_usage_summary()/get_detailed_token_report() to "this instance's own agents" (_own_agent_names() at agents.py:3189-3195), but it can only filter the by_agent dict that already exists — it cannot un-merge metrics that were summed into one bucket the moment two agents shared a name (agents.py:3235-3237 — scoped_by_agent = {name: metrics for name, metrics in by_agent.items() if name in own_names} still returns the merged totals for that name).

Every unnamed Agent() defaults to the same name ("Agent"), so this isn't an edge case — it's the default. Any multi-tenant server that creates one PraisonAIAgents/Agent per request (a documented, common deployment pattern) will silently merge different tenants' token counts and cost whenever two agents share a name.

Reproduction (ran against current code):

from praisonaiagents.telemetry.token_collector import get_token_collector, TokenMetrics

collector = get_token_collector()
collector.reset()

# Tenant A's agent (default name "Agent")
collector.track_tokens('gpt-4o', 'Agent', TokenMetrics(input_tokens=100, output_tokens=50))
# Tenant B's *unrelated* agent, same default name, running concurrently
collector.track_tokens('gpt-4o', 'Agent', TokenMetrics(input_tokens=9000, output_tokens=4000))

print(collector.get_session_summary()['by_agent']['Agent'])
# -> {'input_tokens': 9100, 'output_tokens': 4050, ..., 'total_tokens': 13150}
# Tenant A's isolated 150 tokens is unrecoverable: both tenants' usage/cost is now one number.

Output confirmed: {'input_tokens': 9100, 'output_tokens': 4050, 'cached_tokens': 0, 'reasoning_tokens': 0, 'audio_input_tokens': 0, 'audio_output_tokens': 0, 'total_tokens': 13150}.

Fix: Key metrics_by_agent (and the recent-interactions log) by a stable per-instance identity, not the display name — e.g. assign each Agent a UUID at construction (the codebase already does this elsewhere, see Agent._approval_scope_id = f"{self.name}:{uuid.uuid4().hex}" in agent/agent.py) and pass that as the agent key to track_tokens, keeping name only as a separate, non-keying display field:

# telemetry/token_collector.py
def add_interaction(self, model: str, agent_id: Optional[str], agent_display_name: Optional[str], metrics: TokenMetrics):
    ...
    if agent_id:
        bucket = self.metrics_by_agent.setdefault(agent_id, {"name": agent_display_name, "metrics": TokenMetrics()})
        bucket["metrics"] = bucket["metrics"] + metrics

# call site (llm.py / agent.py wherever track_tokens is invoked today)
get_token_collector().track_tokens(model, agent_id=self._approval_scope_id, agent_display_name=self.name, metrics=metrics)

This makes the per-instance identity the aggregation key, so _scoped_token_summary() in agents.py can filter by identity (unambiguous) instead of by name (collidable), and no downstream code needs to reconstruct what was already merged.

2. litellm.success_callback/_async_success_callback cleanup in _setup_event_tracking removes by type only, with no ownership check — silently deletes another agent's or the application's own logger

Where: praisonaiagents/llm/llm.py:5880-5922 (_setup_event_tracking)

# llm/llm.py:5900-5922
event_types = [type(event) for event in events]

# Remove old events of same type
for event in litellm.success_callback[:]:
    if type(event) in event_types:
        litellm.success_callback.remove(event)          # <-- removes ANY object of this type,
                                                           #     regardless of who added it
for event in litellm._async_success_callback[:]:
    if type(event) in event_types:
        litellm._async_success_callback.remove(event)    # <-- same problem

# Merge into the global list rather than replacing it. Only remove the
# callbacks this instance registered on a prior call, then append the
# current ones, preserving other instances' callbacks.
if litellm.callbacks is None:
    litellm.callbacks = []
for cb in getattr(self, "_registered_callbacks", []):
    if cb in litellm.callbacks:
        litellm.callbacks.remove(cb)                      # <-- correctly scoped to self
for event in events:
    if event not in litellm.callbacks:
        litellm.callbacks.append(event)
self._registered_callbacks = list(events)

The method's own docstring states the design intent — "only the callbacks this instance previously registered are removed... unrelated instances' callbacks are never touched" — and the litellm.callbacks block (lines 5914-5922) correctly implements that via self._registered_callbacks. But the two blocks above it, touching litellm.success_callback/_async_success_callback, never got the same ownership check: they strip any existing entry whose type matches one of events' types, whether it belongs to this instance, a different concurrent LLM/Agent instance, or was added directly by the application via litellm's own documented custom-callback API (litellm.success_callback.append(my_logger) — see litellm's observability docs). There's also no lock around this read-modify-write, so it races under threads.

Reproduction (ran against current code, litellm stubbed):

from praisonaiagents.llm.llm import LLM

class CustomLogger:               # mirrors litellm's own custom-callback pattern
    def __init__(self, owner): self.owner = owner

# Tenant A's application adds its own logger directly to litellm.success_callback
# (litellm's documented extension point), independent of PraisonAI internals.
agent_a_logger = CustomLogger('tenant-A')
litellm.success_callback.append(agent_a_logger)

# Tenant B's unrelated LLM instance sets up its own event tracking with a
# callback of the SAME class.
llm_b = LLM.__new__(LLM)
llm_b._setup_event_tracking([CustomLogger('tenant-B')])

print(agent_a_logger in litellm.success_callback)   # -> False

Confirmed: before Agent B does anything, success_callback == ['tenant-A']; after B's _setup_event_tracking([...]) call, success_callback == [] — tenant A's logger is silently gone, with no error, warning, or log line.

Fix: Apply the exact same ownership pattern already used for litellm.callbacks two lines below, tracking previously-registered success-callback objects on self instead of matching by type:

# llm/llm.py — replace the type-based removal
for cb in getattr(self, "_registered_success_callbacks", []):
    if cb in litellm.success_callback:
        litellm.success_callback.remove(cb)
for cb in getattr(self, "_registered_async_success_callbacks", []):
    if cb in litellm._async_success_callback:
        litellm._async_success_callback.remove(cb)

for event in events:
    if event not in litellm.success_callback:
        litellm.success_callback.append(event)
    ...
self._registered_success_callbacks = list(events)

(Adjust to whichever of the two lists this class is actually meant to populate — the point is: never remove an entry this instance didn't add, matching the standard already set by the litellm.callbacks block in the same method.) Guard the whole method body with a module-level threading.Lock to close the race.

3. _lazy.py swallows ImportError for required dependencies and returns None, turning a broken install into a confusing TypeError on core public classes

Where: praisonaiagents/_lazy.py:170-231 (create_lazy_getattr_with_fallback), used by praisonaiagents/__init__.py's _LAZY_IMPORTS table for AgentTeam (__init__.py:302) and its alias PraisonAIAgents (__init__.py:304) — two of the most-used top-level classes in the SDK.

# _lazy.py:222-231
except ImportError as exc:
    # Optional module not available: preserve the graceful None
    # fallback, but record why so real import failures remain
    # diagnosable instead of surfacing later as NoneType errors.
    _logger.debug(
        "Lazy import of %r (%s.%s) failed: %s",
        name, module_path, attr_name, exc,
    )
    _cache[name] = None
    return None

This treats every ImportError as "this is an optional integration the user chose not to install" and swallows it at debug level (invisible by default). But AgentTeam/PraisonAIAgents resolve through praisonaiagents.agents.agents (__init__.py:302,304), which imports from ..main import display_error, TaskOutput (agents/agents.py:10), and main.py does from pydantic import BaseModel, ConfigDict at module scope (main.py:8) — pydantic is a required, not optional, dependency (pyproject.toml). If it's missing or broken (partial install, version conflict, corrupted venv), the ImportError is caught by this same blanket handler and AgentTeam silently becomes None instead of raising — with no visible log by default.

Reproduction (ran against current code, import pydantic forced to fail):

import builtins
real_import = builtins.__import__
def fake_import(name, *a, **kw):
    if name == 'pydantic' or name.startswith('pydantic.'):
        raise ImportError('simulated: no module named pydantic')
    return real_import(name, *a, **kw)
builtins.__import__ = fake_import

import praisonaiagents
AgentTeam = praisonaiagents.AgentTeam
print(AgentTeam)                       # -> None
AgentTeam(agents=[], tasks=[])          # -> TypeError: 'NoneType' object is not callable

Confirmed output: AgentTeam resolved to: None, then Constructing raises: TypeError 'NoneType' object is not callable — a user debugging this gets zero indication their pydantic install is broken; they see an unrelated NoneType crash deep in their own code where they call AgentTeam(...).

Fix: Distinguish required from optional dependencies in the lazy-import mapping, or simplest: re-raise when the failing module is not the one actually being resolved (i.e., the ImportError came from a transitive dependency of the target module, not the target module being genuinely absent/optional):

# _lazy.py — re-raise instead of masking when the import chain fails on
# something other than the target module itself (i.e. a required transitive
# dependency is broken, not an intentionally-absent optional package)
except ImportError as exc:
    if getattr(exc, "name", None) not in (module_path, module_path.split(".")[0]):
        # Failure is in a dependency of module_path, not module_path being
        # optional — this is a broken/missing required package. Surface it.
        raise ImportError(
            f"Failed to load {name!r} because importing {module_path!r} raised: {exc}. "
            f"This usually means a required dependency is missing or broken, not that "
            f"{name!r} is an optional feature you chose not to install."
        ) from exc
    _logger.debug(...)
    _cache[name] = None
    return None

At minimum, raise the log level from debug to warning so the failure isn't invisible by default even before the required/optional distinction is implemented.

Validation notes

• Every snippet above was read directly from main at commit 43a106d and independently reproduced by running a standalone script against the installed package (not paraphrased or guessed).
• Duplicate-check performed before filing: searched issue history for "token collector"/"agent name collision", "success_callback stripped", and "lazy import ImportError swallowed" — no matching prior issue found. Confirmed #3977 (closed) covered a different callback registry (main.py's sync_display_callbacks/approval_callback dicts) and a different duplication (Mongo/Chroma knowledge adapters); #5025 (closed) covered a different bug in the same ConcurrencyRegistry file (cross-event-loop semaphore binding, already fixed) unrelated to token accounting or callback lists; #4937 covered handoff chat-history corruption, AgentTeam re-entrancy, and approval-registry leaks — none of which overlap with the three mechanisms here.
• Scope: src/praisonai-agents/praisonaiagents only, per the audit request.

Generated with Claude Code

https://claude.ai/code/session_01C8zQF2WpUVCwWfZiWBrhvB