Add automatic device detection to avoid CUDA assumption errors

Author: nouraellmCreated Jan 23, 2026Updated Jul 15, 2026

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/[email protected]/3.10.18/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/homebrew/Cellar/[email protected]/3.10.18/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/opt/homebrew/lib/python3.10/site-packages/moshi/server.py", line 480, in <module>
    main()
  File "/opt/homebrew/lib/python3.10/site-packages/moshi/server.py", line 431, in main
    mimi = loaders.get_mimi(args.mimi_weight, args.device)
  File "/opt/homebrew/lib/python3.10/site-packages/moshi/models/loaders.py", line 131, in get_mimi
    encoder_transformer = transformer.ProjectedTransformer(
  File "/opt/homebrew/lib/python3.10/site-packages/moshi/modules/transformer.py", line 745, in __init__
    self.transformer = StreamingTransformer(d_model=d_model, **kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/moshi/modules/transformer.py", line 681, in __init__
    layer_class(
  File "/opt/homebrew/lib/python3.10/site-packages/moshi/modules/transformer.py", line 506, in __init__
    self.self_attn: StreamingMultiheadAttention = StreamingMultiheadAttention(
  File "/opt/homebrew/lib/python3.10/site-packages/moshi/modules/transformer.py", line 362, in __init__
    in_proj = nn.Linear(embed_dim, mult * out_dim, bias=False, **factory_kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 99, in __init__
    self.weight = Parameter(torch.empty((out_features, in_features), **factory_kwargs))
  File "/opt/homebrew/lib/python3.10/site-packages/torch/cuda/__init__.py", line 305, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

This happens because the code (in moshi/modules/transformer.py during nn.Linear init or similar) attempts to place tensors/layers on CUDA without checking availability, and standard pip-installed PyTorch on macOS is CPU-only (no CUDA support possible on Apple hardware).

Environment details:

Chip: Apple M1
macOS: Sequoia 15.6
Python: 3.10 (Homebrew)
Command run: python -m moshi.server

Suggested Feature

Add automatic device detection at startup / model loading time in the PyTorch backend, similar to common patterns in other projects:

Check torch.cuda.is_available() → use "cuda" if true
Else check torch.backends.mps.is_available() → use "mps" (for Apple Silicon acceleration via Metal)
Else fallback to "cpu"

Ideally, make --device default to this auto-detected value if not provided, or at least gracefully fall back instead of crashing on layer creation.

Source: kyutai-labs/moshi

View original on GitHub View discussion on GitHub