groupBy on a text[] property omits objects from groups that already exist in the result

How to reproduce this bug?

After groupBy.groups(the maximum number of groups to return) distinct values have been collected, later objects must not create another group. They should still be added to a group if they also have a value that is already in the result.

For text[] (and other array) properties that is not what happens. The implementation walks the array left to right. As soon as it sees a value that would create a new group past the limit, it skips the rest of that object, including values that already have a group.

1. Start Weaviate

docker run --rm -p 8080:8080 -p 50051:50051 \
  -e AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true \
  -e PERSISTENCE_DATA_PATH=/var/lib/weaviate \
  -e CLUSTER_HOSTNAME=node1 \
  cr.weaviate.io/semitechnologies/weaviate:1.39.5

2. Create a collection (vectorizer: none) and three objects

Vectors are chosen so nearVector: [1,0,0] returns obj1, then obj2, then obj3.

curl -sS -X POST http://localhost:8080/v1/schema \
  -H 'Content-Type: application/json' \
  -d '{
    "class": "GroupByPoc",
    "vectorizer": "none",
    "vectorIndexConfig": {"distance": "cosine"},
    "properties": [
      {"name": "title", "dataType": ["text"]},
      {"name": "tags", "dataType": ["text[]"]}
    ]
  }'

curl -sS -X POST http://localhost:8080/v1/objects \
  -H 'Content-Type: application/json' \
  -d '{
    "class": "GroupByPoc",
    "id": "00000000-0000-0000-0000-000000000001",
    "properties": {"title": "obj1", "tags": ["A"]},
    "vector": [1, 0, 0]
  }'

curl -sS -X POST http://localhost:8080/v1/objects \
  -H 'Content-Type: application/json' \
  -d '{
    "class": "GroupByPoc",
    "id": "00000000-0000-0000-0000-000000000002",
    "properties": {"title": "obj2", "tags": ["B"]},
    "vector": [0.95, 0.05, 0]
  }'

curl -sS -X POST http://localhost:8080/v1/objects \
  -H 'Content-Type: application/json' \
  -d '{
    "class": "GroupByPoc",
    "id": "00000000-0000-0000-0000-000000000003",
    "properties": {"title": "obj3", "tags": ["C", "A"]},
    "vector": [0.7, 0.3, 0]
  }'

3. Group by tags with groups: 2

curl -sS http://localhost:8080/v1/graphql \
  -H 'Content-Type: application/json' \
  -d '{"query":"{ Get { GroupByPoc(nearVector:{vector:[1,0,0]} groupBy:{path:[\"tags\"] groups:2 objectsPerGroup:5}) { _additional { group { groupedBy { value } count hits { title tags _additional { id distance } } } } } } }"}'

Search order:

After obj1 and obj2 the result already has groups A and B, so the groups: 2 limit is reached. obj3 must not create group C. obj3 still has tag A, so it should appear under group A.

The same skip happens with hybrid + groupBy on an array property. A single-value (non-array) property does not hit this: a later object whose only value is A is still added to group A.

What is the expected behavior?

Group A contains obj1 and obj3 (count: 2). Group B contains obj2. There is no group C.

groups only limits how many distinct groups are returned. An object that also has a value already present in those groups should still be counted in that group.

What is the actual behavior?

Group A contains only obj1 (count: 1). obj3 is missing, because its first tag is C.

Typical GraphQL payload:

{
  "data": {
    "Get": {
      "GroupByPoc": [
        {
          "_additional": {
            "group": {
              "groupedBy": { "value": "A" },
              "count": 1,
              "hits": [{ "title": "obj1", "tags": ["A"] }]
            }
          }
        },
        {
          "_additional": {
            "group": {
              "groupedBy": { "value": "B" },
              "count": 1,
              "hits": [{ "title": "obj2", "tags": ["B"] }]
            }
          }
        }
      ]
    }
  }
}

If obj3’s tags are ["A", "C"] instead of ["C", "A"], obj3 does appear in group A. Whether the object is included depends on array order.

Same three objects with groups: 3 (cap high enough to allow C) returns group A with obj1 and obj3. So obj3 is in the index and would belong to A; groups: 2 is what drops it.

Reproduced against cr.weaviate.io/semitechnologies/weaviate:1.39.5 (single node, anonymous access).

Supporting information

Get near* grouping (adapters/repos/db/shard_group_by.go): when len(groups) >= groupBy.Groups, a new array value does continue DOCS_LOOP, which skips the remaining values of the current object:

if !groupExists && len(groups) >= g.groupBy.Groups {
    continue DOCS_LOOP
}

Hybrid grouping (usecases/traverser/hybrid_group_by.go) does the same: skipResult = true; break on the rest of that object’s values.

Existing hybrid array tests set groups: 100 with a comment “Very high limit to avoid interference”, so this limit is not covered for arrays.

Server Version

v1.39.5 (2e3e707)

Weaviate Setup

Single Node

Nodes count

1

Code of Conduct

• I have read and agree to the Weaviate's Contributor Guide and Code of Conduct