[BUG] - MLX ring fails when master and workers disagree on the master's ring IP (directed-only edge lookup), and placement always selects the smallest cycle so multi-node clusters never form (pipeline stays world_size=1)
Author: tcorsi-gitCreated Sep 13, 2026Updated Sep 13, 2026
Labelsbug
While running exo v1.0.71 on a 2-node Apple Silicon M4 (16GB) cluster joined by a Thunderbolt bridge (each node also has Ethernet), we hit two independent bugs that together made distributed MLX ring inference impossible. Both are fixed by small patches to exo.master.placement_utils / exo.shared.topology. With the patches, the ring connects on the first try, and we validated a 66,773-token prompt (needle-retrieval at ~31k depth) answered correctly over a 2-node TB ring with zero ring errors.
Bug 1: Ring hostfile entries are derived from directed topology edges, so ranks disagree on IPs
_find_connection_ip(master, worker) only walks get_all_connections_between(master, worker) — the directed source→sink edges. But the
topology edges are asymmetric:
• worker → master connection carries sinkMultiaddr entries for the master (TB 10.10.0.2, Ethernet 192.168.204.30)
• master → worker connection may carry only rdma_en2 and no socket IPs
Result: each rank computes its own ring hostfile from a different view. Actual evidence (2-node ring, rank0=master over Ethernet,
rank1=worker):
• Master (rank 0) hostfile: ["192.168.204.30:62940", "192.168.204.155:62940"] (self + peer both Ethernet)
• Worker (rank 1) hostfile: ["10.10.0.2:62940", "192.168.204.155:62940"] (peer via TB prioritised, self Ethernet)
The worker then tries to connect to the master at 10.10.0.2, but the master bound its ring socket to its own hostfile entry
192.168.204.30 — the worker fails with RuntimeError: [ring] Couldn't connect (error: 60/61), and lsof confirmed the master was only
listening on the Ethernet IP.
The mismatch is structural: peers derive each other's addresses from different, one-directional edges, and the self-address
selection (first non-loopback interface) ignores interface-type priority while the peer path prioritises Thunderbolt.
Our fix: compute every rank's hostfile entry from that rank's own node_network interfaces with a single deterministic priority
(thunderbolt > ethernet > maybe_ethernet, skipping loopback/link-local/IPv6), with the topology-based _find_ip_prioritised only as a
fallback. All ranks then agree on every entry by construction, and the ring is fully TB.
Patch sketch:
─ python
def get_mlx_ring_hosts_by_node(selected_cycle, cycle_digraph, ephemeral_port, node_network):
world_size = len(selected_cycle)
if world_size == 0:
return {}
pri = {'thunderbolt': 0, 'ethernet': 1, 'maybe_ethernet': 2}
def pick_ip(node_id):
net = node_network.get(node_id)
best_ip, best_pri = None, 99
if net is not None:
for iface in net.interfaces:
ip = iface.ip_address
if not ip or ':' in ip: continue
if ip.startswith('127.') or ip.startswith('169.254.'): continue
p = pri.get(getattr(iface, 'interface_type', 'unknown'), 3)
if p < best_pri:
best_pri, best_ip = p, ip
return best_ip
hosts_by_node = {}
for rank, node_id in enumerate(selected_cycle):
left_rank = (rank - 1) % world_size
right_rank = (rank + 1) % world_size
hosts_for_node = []
for idx, other_node_id in enumerate(selected_cycle):
if idx == rank:
hosts_for_node.append(Host(ip=pick_ip(node_id) or '0.0.0.0', port=ephemeral_port))
elif idx in (left_rank, right_rank):
peer_ip = pick_ip(other_node_id)
if peer_ip is None:
peer_ip = _find_ip_prioritised(node_id, other_node_id, cycle_digraph, node_network, ring=True)
if peer_ip is None:
raise ValueError('MLX ring backend requires connectivity between neighbouring nodes')
hosts_for_node.append(Host(ip=peer_ip, port=ephemeral_port))
hosts_by_node[node_id] = hosts_for_node
for d in ((node_i, node_j), (node_j, node_i)):
for connection in cycle_digraph.get_all_connections_between(*d):
if isinstance(connection, SocketConnection):
ip = connection.sink_multiaddr.ip_address
if ip not in seen:
seen.add(ip)
yield ip
Bug 2: get_smallest_cycles makes every placement world_size=1
After fixing the ring we found instances were still single-node. In place_instance (Pipeline path):
─ python
smallest_cycles = get_smallest_cycles(cycles_with_sufficient_memory)
...
candidate_cycles = cycles_with_leaf_nodes if cycles_with_leaf_nodes != [] else smallest_cycles
selected_cycle = max(candidate_cycles, key=...)
get_smallest_cycles returns the cycles with the fewest nodes:
─ python
min_nodes = min(len(cycle) for cycle in cycles)
return [cycle for cycle in cycles if len(cycle) == min_nodes]
So when singletons are present in the cycle candidates, they always win, and the placement never distributes — every instance is
world_size=1 even when a valid multi-node cycle fits in memory. (We added singletons to get_cycles to work around ValueError: No
cycles found with sufficient memory on memory-constrained nodes; vanilla's rustworkx simple_cycles doesn't return isolated nodes as
cycles, but that makes placement fail entirely when no multi-node cycle fits.)
Fix: prefer the cycle with the most nodes that still fits memory:
─ python
def get_smallest_cycles(cycles):
max_nodes = max(len(cycle) for cycle in cycles)
return [cycle for cycle in cycles if len(cycle) == max_nodes]
Singletons remain the natural fallback when nothing larger fits, and small models still work (they just run distributed when peers
are available — which is what a cluster is for).
Also worth noting: filter_cycles_by_memory / _compute_total_memory / _allocate_and_validate_layers compare against ram_total, but on
our machines the vanilla code path checked a field that made 30B-class models fail with "No cycles found with sufficient memory"
even when pooled RAM was clearly sufficient; switching these three to ram_total fixed it.
Environment
• exo v1.0.71 (DMG bundle), macOS 26.x, 2x Mac mini M4 base 16GB
• Interconnect: Thunderbolt bridge (10.10.0.2 / 10.10.0.3) + Ethernet (192.168.204.x); exo classifies bridge0 as thunderbolt
(priority 0) correctly
• Models: 9B MLX 4bit/8bit, distributed pipeline, world_size=2
• Results with patches: ring connects first try, zero errno 54/60/61 aborts; 66,773-token prompt (needle at ~31k) answered correctly
in ~5.3 min end-to-end
Happy to share the full patched placement_utils/topology bytecode if useful, or open a PR.
Source: exo-explore/exo