SAM3 mask exemplars: claimed in the paper, unusable in the release

TL;DR. The SAM3 paper/README advertise masks as a usable visual prompt for the detector ("text or visual prompts such as points, boxes, and masks"; detector "conditioned on text, geometry, and image exemplars"). In the released artifact, mask exemplars for the open-vocabulary detector are non-functional at every layer. From the code: the config never builds the mask encoder, the checkpoint has no mask-encoder weights, and the public API has no mask entry point. Box exemplars, by contrast, are fully trained and live.

The claim

README.md:

• L49: "It can detect, segment, and track objects using text or visual prompts such as points, boxes, and masks."
• L188: "The detector is a DETR-based model conditioned on text, geometry, and image exemplars."

So the paper presents masks as a first-class prompt to the concept detector.

The gap, layer by layer

1. Config never instantiates the mask encoder

model_builder.py:_create_geometry_encoder() (L235–288) is the only geometry-encoder builder (used at both L616 and L688). It constructs SequenceGeometryEncoder with points and boxes projections only:

input_geometry_encoder = SequenceGeometryEncoder(
    ...
    points_direct_project=True, points_pool=True, points_pos_enc=True,
    boxes_direct_project=True,  boxes_pool=True,  boxes_pos_enc=True,
    ...
)   # no mask_encoder=, no add_mask_label=

mask_encoder and add_mask_label therefore fall to their defaults None / False (geometry_encoders.py:507-508). In any instantiated model self.mask_encoder is None, so the mask branches at geometry_encoders.py:800 and :834 are skipped, and Prompt.append_masks (:366) feeds nothing downstream.

2. The checkpoint has no mask-encoder weights

Dumping sam3.pt (1465 params), detector.geometry_encoder.* contains only:

• points_* (direct_project, pool_project, pos_enc_project)
• boxes_* (direct_project, pool_project, pos_enc_project)
• cls_embed, label_embed, encode.{0,1,2}.*, encode_norm, final_proj, norm, img_pre_norm

There is no detector.geometry_encoder.mask_encoder.* and no mask_label_embed. The only mask_downsampler anywhere in the checkpoint is tracker.maskmem_backbone.mask_downsampler.* — the SAM2-lineage video-memory encoder, a different module on the tracker, not the detector. exemplar and visual_prompt appear zero times in the weights.

Consequence: even if you wired a FusedMaskEncoder into the config, it would be randomly initialized. The capability cannot be recovered from the released weights without training.

3. The public API has no mask entry point

Sam3Processor (sam3_image_processor.py) exposes only:

• set_text_prompt (L113)
• add_geometric_prompt (L128–150) — box-only, via append_boxes

There is no method to add a mask exemplar, despite Prompt.append_masks (geometry_encoders.py:366) and _encode_masks (:683) being fully implemented.

4. The detector's only mask-encoder reference is dead code

sam3_image.py:425 _get_best_mask calls self.geometry_encoder.mask_encoder.mask_downsampler(...), which would AttributeError on None. The method is never called anywhere in the repo, so it never fires.

Net

The mask-exemplar feature ships as inert scaffolding — FusedMaskEncoder, _encode_masks, Prompt.append_masks, Prompt.mask_embeddings, _get_best_mask — that is not instantiated by the config, not trained in the checkpoint, and not reachable from the API. Unlike box exemplars (trained and live via geometry_encoder), mask exemplars cannot be used without training a mask encoder and re-exposing it through the processor.

Reproduction

Checkpoint key inspection without torch (.pt is a zip; param names are plain strings in the pickle):

import zipfile, re
z = zipfile.ZipFile('sam3_repo/sam3.pt')
data = z.read([n for n in z.namelist() if n.endswith('data.pkl')][0])
strs = [s.decode() for s in re.findall(rb'[ -~]{4,}', data)]
print(sorted({s for s in strs if 'mask_encoder' in s or 'mask_label' in s}))
# -> only tracker.maskmem_backbone.mask_downsampler.*  (no detector.geometry_encoder.mask_encoder.*)
print(sorted({s for s in strs if 'exemplar' in s.lower() or 'visual_prompt' in s.lower()}))
# -> []