Pure SSH: a download batch cannot report an object the server does not have, and `git lfs fetch` exits 0

Summary

Over the pure-SSH transfer protocol there is no way for a server to tell the client that it does not have an object, and the answer the specification tells it to give instead — noop — is read by the client as "nothing to do". A git lfs fetch for an object the remote does not hold therefore exits 0 having fetched nothing.

The same fetch over HTTP exits non-zero, because the HTTP batch API has a per-object error object. t/t-fetch.sh pins that behaviour and its comment states the intent plainly:

# should return non-zero, but should also download all the other valid files too

That test runs over HTTP only. Over pure SSH the outcome is silently the opposite.

Why the server cannot say it

docs/proposals/ssh_adapter.md is where the pure-SSH protocol is specified, and its batch response grammar has no error production:

batch-response = status-command
                 *argument
                 delim-pkt
                 (*batch-oid-line | error-message)
                 flush-pkt
batch-oid-line = PKT-LINE(oid size action *(key=[value]) LF)

error-message is the whole-batch alternative, taken only when the status is not 200; there is nothing per-object. The document then closes the two remaining doors:

If the server has no actions that are valid for an object, it should be listed once in the response with the noop action.

Unknown arguments should be ignored, as should unknown key-value pairs in the oid-line production.

So a server that invented error=404 on the oid line would be specified into being ignored. noop is the only thing it may say, and the HTTP API's per-object

{ "oid": "…", "size": 123, "error": { "code": 404, "message": "Object does not exist" } }

with the batch still 200 (docs/api/batch.md) has no counterpart here.

Why the client reads noop as success

tq/ssh.go, parsing the batch response:

if entries[2] == "noop" {
    continue
}

The transfer is left with an empty Actions map. Then tq/transfer_queue.go, dispatching it:

} else if a == nil && manifest.standaloneTransferAgent == "" {
    q.Skip(o.Size)
    q.wait.Done()
}

No error, no retry, nothing on q.errorc, nothing printed. For the upload direction that is correct — no action means the server already has the object. For the download direction it means the object is absent, and the two are indistinguishable at that point.

SSHAdapter.download does have the right error (No download action for object: %s), but it is unreachable: the queue never hands a no-action transfer to the adapter.

Reference implementation

charmbracelet/git-lfs-transfer, which this repository's own test suite uses via setup_pure_ssh, sends exactly what the specification prescribes — transfer/processor.go:

func (p *Processor) DownloadBatch() (Status, error) {
    return p.BatchData(DownloadOperation, "download", "noop")
}

missingAction is noop. So every server built on it — and that includes Gitea and Forgejo — has this behaviour.

Reproduction

git-lfs 3.7.1 (linux amd64, go 1.25.0), against a git-lfs-transfer server holding one of the two objects the client asks for and answering noop for the other, per the grammar above:

$ git lfs fetch --all origin ; echo "exit=$?"
2 objects found, done.
exit=0

One object was fetched. The other is neither present nor mentioned. .git/lfs/objects contains one of the two.

What we did about it, in case it is useful

Answering the download action for every oid in a download batch, held or not, and letting get-object answer 404 for the ones that are absent, reaches the outcome t/t-fetch.sh asks for: the fetch exits non-zero, names the missing oid, and still delivers every other object in the batch. It works because get-object does have a per-object error (status-error-response).

It costs eight round trips per fetch, because SSHAdapter.doDownload wraps every non-2xx in errors.NewRetriableError with no exception:

if status < 200 || status > 299 {
    return errors.NewRetriableError(errors.New(tr.Tr.Get("got status %d when fetching OID %s: %s", …)))
}

so a genuinely absent object walks the whole backoff ladder — 0.25, 0.5, 1, 2, 4, 8, 10, 10 seconds, 35.75s measured — before the failure is reported. Retries re-batch, so that is once for the batch rather than once per object, but it is 36 seconds spent re-asking for something the server has already said twice it does not have.

Suggestions

Either of these would close it properly; the first is a protocol change and the second is not.

1. Give batch-oid-line a way to carry a per-object error. The *(key=[value]) tail is already there, so oid size noop error=404 would fit the existing grammar, but the "unknown key-value pairs should be ignored" rule means old clients would silently do what they do today — which is the current wrong thing, so it degrades no worse than the status quo. An action name (oid size error code=404) would be rejected by strict old clients instead of ignored.

2. Make a 404 from get-object non-retriable. 404 is not a transient condition, and today it is the only channel the protocol has for "not here". This alone turns 36 seconds into one round trip and needs no wire change.

Happy to send a patch for either if there is a preference.