[Feature Request] Sandbox logs endpoint reads a node-local shim log on the CubeMaster host — 130593 in cluster deployments with more than one compute node

Summary

CubeMaster's sandbox-logs handler opens a hardcoded node-local path on the master host:

const defaultShimLogPath = "/data/log/CubeShim/cube-shim-req.log"
...
f, err := os.Open(defaultShimLogPath)   // no sandbox -> node resolution

In a cluster deployment where CubeMaster runs separately from the compute nodes, that file never exists on the master host, so every log query fails with ErrorCode_MasterInternalError (130593) and the raw err.Error() is returned to the client. There is no configuration that fixes this when the cluster has more than one compute node.

Environment

• CubeSandbox version / commit: v0.7.0 / v0.7.1 line (code unchanged on master at the time of writing)
• Host OS and kernel version: Kylin V10 / 5.10 (KCE nodes)
• KVM info (modinfo kvm): not collected for this report
• Deployment mode: cluster — control plane (CubeMaster / CubeAPI / CubeOps / TemplateCenter) as pods, Cubelet as a DaemonSet, multiple compute nodes
• Relevant component: CubeMaster (handler) / CubeShim (log producer)

Steps to Reproduce

1. Deploy in cluster mode: CubeMaster runs on a control-plane host that does not run Cubelet, with ≥ 1 compute node (the failure appears with a single non-co-located node, and becomes unfixable with ≥ 2 nodes).
2. Create a Sandbox on a compute node.
3. GET {CUBE_API}/sandboxes/{sandboxID}/logs (CubeAPI → POST /cube/sandbox/logs → CubeMaster).
4. Observe:

HTTP 500
CubeMaster returned error code 130593:
open /data/log/CubeShim/cube-shim-req.log: no such file or directory

Expected Behavior

The endpoint returns the shim logs of the requested Sandbox regardless of which compute node hosts it. CubeMaster should resolve the Sandbox's node and read the log from that node.

Actual Behavior

GET /sandboxes/d7a94b1cfc2448aea0220e8946810c6b/logs
→ sdkext: HTTP 500: CubeMaster returned error code 130593:
  open /data/log/CubeShim/cube-shim-req.log: no such file or directory

Root Cause (code reference, master)

CubeMaster/pkg/service/httpservice/cube/sandbox_logs.go:

const defaultShimLogPath = "/data/log/CubeShim/cube-shim-req.log"

func readShimLogs(sandboxID string, cursor int64, limit int) ([]SandboxLogEntry, int64, bool, error) {
	f, err := os.Open(defaultShimLogPath)   // absolute path on the CubeMaster host
	if err != nil {
		return nil, 0, false, err
	}
	...
	for scanner.Scan() {
		var entry ShimLogLine
		if err := fastJSON.Unmarshal(line, &entry); err != nil { continue }
		if entry.InstanceID != sandboxID { continue }   // filtering itself is correct
		...
	}
}

The os.Open failure is mapped to ErrorCode_MasterInternalError and surfaced with RetMsg: err.Error(), which is what clients see as 130593.

Note the parsing/filtering logic is fine — the problem is purely where the file is read from: the master assumes the log is on its own filesystem.

Why the shared-volume workaround cannot be extended to N nodes

The usual downstream workaround is to mount a shared RWX volume (e.g. JuiceFS/NFS) at /data/log/CubeShim on both CubeMaster and the cube-node pods, so the file becomes visible to the master. This is only correct for exactly one compute node:

1. With N nodes, all N CubeShim instances append to the same filename on the shared volume.
2. Rotation is rename-based (related: #1292). Each node rotates independently, so one node renaming the file invalidates/truncates the file the other N-1 nodes are still appending to → entries are silently lost (not just missing).
3. The reader has no notion of "which node" a line came from. The InstanceID filter prevents misattribution, but it cannot recover lines that another node's rotation already destroyed.

Net effect in our production cluster (multiple compute nodes): the log view is permanently unusable, and the only option left downstream is to hide the error in the UI.

Suggested Fix

CubeMaster already knows the Sandbox → node mapping (host_id, e.g. cubemastercli ls --all; sandbox detail also exposes the host/client id). The master → Cubelet RPC surface exists and already carries every other per-Sandbox operation:

// pkgs/proto/services/cubebox/v1/cubebox.proto
service CubeboxMgr {
  rpc Create(...) ; rpc Destroy(...) ; rpc List(...) ; rpc Update(...) ; rpc Exec(...) ;
  rpc AppSnapshot(...) ; rpc CommitSandbox(...) ; rpc RollbackSandbox(...) ;
  rpc CleanupTemplate(...) ; rpc ListSandboxSnapshots(...) ; rpc ListLocalSnapshots(...) ;
  rpc GetLocalSnapshot(...) ; rpc GetStorageMetrics(...) ; rpc InspectStorageVolumes(...) ;
  rpc CleanupOrphanStorageFiles(...) ;
  // no log method
}

Proposal:

1. Add an RPC to CubeboxMgr, e.g. rpc ReadShimLogs(ReadShimLogsRequest) returns (ReadShimLogsResponse) with sandbox_id / cursor / limit, implemented by Cubelet reading its own node-local /data/log/CubeShim/cube-shim-req.log (reuse the existing parse + InstanceID filter).
2. In readShimLogs, resolve sandbox → node and dispatch to that node's Cubelet instead of os.Open on the master host. Keep the current response shape (logs / nextCursor / hasMore) so existing clients are unaffected.
3. Optional smaller alternative: make the path node-scoped in the shared-volume model (<node-id>/cube-shim-req.log). This still requires every node's file to be individually addressable by the reader, so the RPC approach is preferable.

Connectivity note: in Kubernetes deployments, Cubelet's management ports (9998/9999/9966) must be reachable from CubeMaster — this requires hostNetwork: true on the cube-node DaemonSet. With pod networking, master → node gRPC is refused, and the same failure mode is already visible today for template cleanup RPCs (CleanupTemplate → 130593 deadline exceeded). It may be worth documenting this prerequisite next to the new RPC.

Additional Context

• The error text is misleading. Clients are told to "confirm that log forwarding is enabled on the node", but there is no such configuration switch, and no switch would help in a multi-node deployment. If a node-scoped read is not implemented yet, an explicit "not supported in cluster deployments" error would be far more actionable than a bare MasterInternalError.
• Related upstream work, all node-local:
  • #1616 — shim keeps /data/cubelet/log/<sandbox-id>/stdout|stderr; cubecli logs reads the host path first.
  • #1292 — reopen CubeShim/VMM logs after rename-based rotation.
  • #1136 — log-source lifecycle proposal (source scoping, rotation, retention).
  • #1598 — v0.7.0 restored Sandbox missing stdout/stderr files.
• Because all of the above are also host-path reads, cubecli logs has the same limitation when invoked anywhere other than the Sandbox's own node — a node-aware fetch would address the CLI and the HTTP endpoint together.
• Happy to test a patch: we can reproduce on demand on a multi-node cluster (v0.7.x) and verify both the endpoint and cubecli logs per node.