Nebula Graph v3.8.0 Restore Fails with E_RESTORE_FAILURE During Meta Restore

Author: jkiran-emb-globalCreated Mar 25, 2026Updated Mar 25, 2026

Labelstype/bugseverity/noneaffects/none

br restore full fails during the meta restore phase with:

RestoreMeta failed: E_RESTORE_FAILURE

Trying to backup nebula from one cluster and restore on another cluster

OS: Kubernetes
Version: Nebula Graph v3.8.0
BR Version: 3.6.0

Steps to reproduce the behavior:

Deploy Nebula Graph v3.8.0 cluster on Kubernetes (multi-meta, multi-storage setup).
Take a full backup using br backup full to S3.
Run restore command:

bash

br restore full \
  --meta "nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559" \
  --s3.endpoint "https://s3.region.amazonaws.com" \
  --s3.access_key "<redacted>" \
  --s3.secret_key "<redacted>" \
  --s3.region "region" \
  --storage "s3://<redacted>/nebula" \
  --name "backup"

Logs:

{"level":"info","meta address":"nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2026-03-25T20:20:22.175Z"}
{"level":"info","meta address":"nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2026-03-25T20:20:22.178Z"}
{"level":"info","meta address":"nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2026-03-25T20:20:22.179Z"}
{"level":"info","meta address":"nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2026-03-25T20:20:22.181Z"}
{"level":"info","meta address":"nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2026-03-25T20:20:22.181Z"}
{"level":"info","meta address":"nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2026-03-25T20:20:22.184Z"}
{"level":"info","meta address":"nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2026-03-25T20:20:22.188Z"}
{"level":"info","meta address":"nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2026-03-25T20:20:22.190Z"}
{"level":"info","meta address":"nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2026-03-25T20:20:22.190Z"}
{"level":"info","meta address":"nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2026-03-25T20:20:22.192Z"}
{"host info":"map[nebula-graphd-0.nebula-graphd-headless.nebula.svc.cluster.local:nebula-graphd-0.nebula-graphd-headless.nebula.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebula-graphd-0.nebula-graphd-headless.nebula.svc.cluster.local:9669[GRAPH]: (data: , root: /usr/local/nebula) nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559[META]: (data: /usr/local/nebula/data/meta, root: /usr/local/nebula) nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559[META]: (data: /usr/local/nebula/data/meta, root: /usr/local/nebula) nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559[META]: (data: /usr/local/nebula/data/meta, root: /usr/local/nebula) nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:9779[STORAGE]: (data: /usr/local/nebula/data/storage, root: /usr/local/nebula) | nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:8888[AGENT]: (data: , root: ) nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:9779[STORAGE]: (data: /usr/local/nebula/data/storage, root: /usr/local/nebula) | nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:8888[AGENT]: (data: , root: ) nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:9779[STORAGE]: (data: /usr/local/nebula/data/storage, root: /usr/local/nebula) | nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:8888[AGENT]: (data: , root: )]","level":"info","msg":"Get cluster topology from the nebula.","time":"2026-03-25T20:20:22.193Z"}
{"backup":"backup","level":"info","msg":"Check backup dir successfully.","time":"2026-03-25T20:20:22.259Z","uri":"s3://bucket-backup/nebula/backup"}
{"dir":"/usr/local/nebula","host":"nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"META","time":"2026-03-25T20:20:22.325Z"}
{"dir":"/usr/local/nebula","host":"nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"META","time":"2026-03-25T20:20:32.336Z"}
{"dir":"/usr/local/nebula","host":"nebula-graphd-0.nebula-graphd-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"GRAPH","time":"2026-03-25T20:20:42.356Z"}
{"dir":"/usr/local/nebula","host":"nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"META","time":"2026-03-25T20:20:52.372Z"}
{"dir":"/usr/local/nebula","host":"nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"STORAGE","time":"2026-03-25T20:21:02.384Z"}
{"dir":"/usr/local/nebula","host":"nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"STORAGE","time":"2026-03-25T20:21:12.393Z"}
{"dir":"/usr/local/nebula","host":"nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"STORAGE","time":"2026-03-25T20:21:22.403Z"}
{"backup":"backup","level":"info","msg":"Stop cluster successfully.","time":"2026-03-25T20:21:32.415Z"}
{"addr":"nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:9779","backup path":"/usr/local/nebula/data/storage/nebula_old_1774470092","level":"info","msg":"Backup origin storage data path successfully.","origin path":"/usr/local/nebula/data/storage/nebula","time":"2026-03-25T20:21:32.417Z"}
{"addr":"nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:9779","backup path":"/usr/local/nebula/cluster.id_old_1774470092","level":"info","msg":"Backup origin cluster.id path successfully.","origin not exist":true,"origin path":"/usr/local/nebula/cluster.id","time":"2026-03-25T20:21:32.418Z"}
{"addr":"nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:9779","backup path":"/usr/local/nebula/data/storage/nebula_old_1774470092","level":"info","msg":"Backup origin storage data path successfully.","origin path":"/usr/local/nebula/data/storage/nebula","time":"2026-03-25T20:21:32.420Z"}
{"addr":"nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:9779","backup path":"/usr/local/nebula/cluster.id_old_1774470092","level":"info","msg":"Backup origin cluster.id path successfully.","origin not exist":true,"origin path":"/usr/local/nebula/cluster.id","time":"2026-03-25T20:21:32.421Z"}
{"addr":"nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:9779","backup path":"/usr/local/nebula/data/storage/nebula_old_1774470092","level":"info","msg":"Backup origin storage data path successfully.","origin path":"/usr/local/nebula/data/storage/nebula","time":"2026-03-25T20:21:32.422Z"}
{"addr":"nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:9779","backup path":"/usr/local/nebula/cluster.id_old_1774470092","level":"info","msg":"Backup origin cluster.id path successfully.","origin not exist":true,"origin path":"/usr/local/nebula/cluster.id","time":"2026-03-25T20:21:32.424Z"}
{"addr":"nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559","backup path":"/usr/local/nebula/data/meta/nebula_old_1774470092","level":"info","msg":"Backup origin meta data path successfully.","origin path":"/usr/local/nebula/data/meta/nebula","time":"2026-03-25T20:21:32.425Z"}
{"addr":"nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559","backup path":"/usr/local/nebula/data/meta/nebula_old_1774470092","level":"info","msg":"Backup origin meta data path successfully.","origin path":"/usr/local/nebula/data/meta/nebula","time":"2026-03-25T20:21:32.426Z"}
{"addr":"nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559","backup path":"/usr/local/nebula/data/meta/nebula_old_1774470092","level":"info","msg":"Backup origin meta data path successfully.","origin path":"/usr/local/nebula/data/meta/nebula","time":"2026-03-25T20:21:32.427Z"}
{"backup":"backup","level":"info","msg":"Backup origin cluster data successfully.","time":"2026-03-25T20:21:32.427Z"}
{"level":"info","msg":"Download meta data to cluster successfully.","time":"2026-03-25T20:21:33.749Z"}
{"addr":"nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:9779","external":"s3://bucket-backup/nebula/backup/data/nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:9779/data0/","level":"info","local":"/usr/local/nebula/data/storage/nebula","msg":"Download storage data successfully.","time":"2026-03-25T20:22:02.755Z"}
{"addr":"nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:9779","external":"s3://bucket-backup/nebula/backup/data/nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:9779/data0/","level":"info","local":"/usr/local/nebula/data/storage/nebula","msg":"Download storage data successfully.","time":"2026-03-25T20:22:30.930Z"}
{"addr":"nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:9779","external":"s3://bucket-backup/nebula/backup/data/nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:9779/data0/","level":"info","local":"/usr/local/nebula/data/storage/nebula","msg":"Download storage data successfully.","time":"2026-03-25T20:22:58.798Z"}
{"level":"info","msg":"Download storage data to cluster successfully.","time":"2026-03-25T20:22:58.798Z"}
{"addr":"nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559","level":"info","msg":"Start meta service successfully.","time":"2026-03-25T20:23:03.809Z"}
{"addr":"nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559","level":"info","msg":"Start meta service successfully.","time":"2026-03-25T20:23:08.818Z"}
{"addr":"nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559","level":"info","msg":"Start meta service successfully.","time":"2026-03-25T20:23:13.828Z"}
{"level":"info","msg":"Start meta service successfully.","time":"2026-03-25T20:23:23.835Z"}
{"level":"info","meta address":"nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2026-03-25T20:23:23.835Z"}
{"level":"info","meta address":"nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2026-03-25T20:23:23.838Z"}
{"dir":"/usr/local/nebula","host":"nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"META","time":"2026-03-25T20:23:23.883Z"}
{"dir":"/usr/local/nebula","host":"nebula-graphd-0.nebula-graphd-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"GRAPH","time":"2026-03-25T20:23:33.893Z"}
{"dir":"/usr/local/nebula","host":"nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"META","time":"2026-03-25T20:23:43.901Z"}
{"dir":"/usr/local/nebula","host":"nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"STORAGE","time":"2026-03-25T20:23:53.910Z"}
{"dir":"/usr/local/nebula","host":"nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"STORAGE","time":"2026-03-25T20:24:03.919Z"}
{"dir":"/usr/local/nebula","host":"nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"STORAGE","time":"2026-03-25T20:24:13.928Z"}
{"dir":"/usr/local/nebula","host":"nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"META","time":"2026-03-25T20:24:23.937Z"}
{"backup path":"/usr/local/nebula/data/storage/nebula_old_1774470092","level":"info","msg":"Moveback origin STORAGE data path successfully","name":"STORAGE[nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:9779]","origin path":"/usr/local/nebula/data/storage/nebula","time":"2026-03-25T20:24:33.983Z"}
{"backup path":"/usr/local/nebula/data/storage/nebula_old_1774470092","level":"info","msg":"Moveback origin STORAGE data path successfully","name":"STORAGE[nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:9779]","origin path":"/usr/local/nebula/data/storage/nebula","time":"2026-03-25T20:24:34.020Z"}
{"backup path":"/usr/local/nebula/data/storage/nebula_old_1774470092","level":"info","msg":"Moveback origin STORAGE data path successfully","name":"STORAGE[nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:9779]","origin path":"/usr/local/nebula/data/storage/nebula","time":"2026-03-25T20:24:34.058Z"}
{"backup path":"/usr/local/nebula/data/meta/nebula_old_1774470092","level":"info","msg":"Moveback origin META data path successfully","name":"META[nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559]","origin path":"/usr/local/nebula/data/meta/nebula","time":"2026-03-25T20:24:34.062Z"}
{"backup path":"/usr/local/nebula/data/meta/nebula_old_1774470092","level":"info","msg":"Moveback origin META data path successfully","name":"META[nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559]","origin path":"/usr/local/nebula/data/meta/nebula","time":"2026-03-25T20:24:34.064Z"}
{"backup path":"/usr/local/nebula/data/meta/nebula_old_1774470092","level":"info","msg":"Moveback origin META data path successfully","name":"META[nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559]","origin path":"/usr/local/nebula/data/meta/nebula","time":"2026-03-25T20:24:34.068Z"}
{"addr":"nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:9779","level":"info","msg":"Start STORAGE[nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:9779] by agent successfully.","time":"2026-03-25T20:24:39.126Z"}
{"addr":"nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:9779","level":"info","msg":"Start STORAGE[nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:9779] by agent successfully.","time":"2026-03-25T20:24:44.136Z"}
{"addr":"nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559","level":"info","msg":"Start META[nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559] by agent successfully.","time":"2026-03-25T20:24:49.147Z"}
{"addr":"nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559","level":"info","msg":"Start META[nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559] by agent successfully.","time":"2026-03-25T20:24:54.156Z"}
{"addr":"nebula-graphd-0.nebula-graphd-headless.nebula.svc.cluster.local:9669","level":"info","msg":"Start GRAPH[nebula-graphd-0.nebula-graphd-headless.nebula.svc.cluster.local:9669] by agent successfully.","time":"2026-03-25T20:24:59.163Z"}
{"addr":"nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559","level":"info","msg":"Start META[nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559] by agent successfully.","time":"2026-03-25T20:25:04.174Z"}
{"addr":"nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:9779","level":"info","msg":"Start STORAGE[nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:9779] by agent successfully.","time":"2026-03-25T20:25:09.183Z"}
Error: restore cluster meta failed: restore meta service nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559 failed: call nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559:RestoreMeta failed: E_RESTORE_FAILURE
Usage:
  br restore full [flags]

Flags:
  -h, --help   help for full

Global Flags:
      --concurrency int        Max concurrency for download data (default 5)
      --debug                  Output log in debug level or not
      --log string             Specify br detail log path (default "br.log")
      --meta string            Specify meta server
      --name string            Specify backup name
      --s3.access_key string   S3 Option: set access key id
      --s3.endpoint string     S3 Option: set the S3 endpoint URL, please specify the http or https scheme explicitly
      --s3.region string       S3 Option: set region or location to upload or download backup
      --s3.secret_key string   S3 Option: set secret key for access id
      --storage string         backup target url, format: <SCHEME>://<PATH>.
                                   <SCHEME>: a string indicating which backend type. optional: local, s3.
                                   now only s3-compatible is supported.
                                   example:
                                   for local - "local:///the/local/path/to/backup"
                                   for s3  - "s3://example/url/to/the/backup"

{"level":"fatal","msg":"restore cluster meta failed: restore meta service nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559 failed: call nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559:RestoreMeta failed: E_RESTORE_FAILURE","time":"2026-03-25T20:25:09.184Z"}

K8s Config to deploy cluster and trigger br from another k8s container.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nebula-metad
  namespace: nebula
  labels:
    app.kubernetes.io/cluster: nebula
    app.kubernetes.io/component: metad
    app.kubernetes.io/managed-by: nebula-operator
    app.kubernetes.io/name: nebula-graph
spec:
  podManagementPolicy: Parallel
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/cluster: nebula
      app.kubernetes.io/component: metad
      app.kubernetes.io/managed-by: nebula-operator
      app.kubernetes.io/name: nebula-graph
  serviceName: nebula-metad-headless
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/cluster: nebula
        app.kubernetes.io/component: metad
        app.kubernetes.io/managed-by: nebula-operator
        app.kubernetes.io/name: nebula-graph
    spec:
      serviceAccountName: nebula-sa
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/cluster: nebula
                    app.kubernetes.io/component: metad
                    app.kubernetes.io/name: nebula-graph
                topologyKey: kubernetes.io/hostname
      initContainers:
        - name: copy-agent
          image: vesoft/nebula-agent:latest
          imagePullPolicy: IfNotPresent
          command: ["cp", "/usr/local/bin/agent", "/shared/agent"]
          volumeMounts:
            - mountPath: /shared
              name: shared-bin
      containers:
        - name: metad
          image: vesoft/nebula-metad:v3.8.0
          imagePullPolicy: IfNotPresent
          # Start metad as a background daemon, then exec the agent as the main process.
          # The agent

Source: vesoft-inc/nebula

View original on GitHub View discussion on GitHub