Metadata filter "empty" / "not empty" ignores fields stored with a null value

Self Checks

• I have read the Contributing Guide and Language Policy.
• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report, otherwise it will be closed.
• 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
• Please do not modify this template :) and fill in all the required fields.

Dify version

main (1.17.1)

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

1. In a knowledge base, create a metadata field author (string).
2. Attach author to document A without setting a value, and set author = "bob" on document B. The UI stores document A's metadata as {"author": null}.
3. In a Knowledge Retrieval node (or chat app), add a manual metadata filter author empty, and run a query that matches both documents.

✔️ Expected Behavior

empty returns document A (and documents without the field); not empty returns only document B.

❌ Actual Behavior

empty does not return document A, and not empty returns both A and B.

DatasetRetrieval.process_metadata_filter_func builds doc_metadata -> 'author' IS NULL. On PostgreSQL -> returns the JSONB value null for a stored null, which is not SQL NULL:

select d->'author' is null, d->>'author' is null
from (values ('{}'::jsonb), ('{"author": null}'::jsonb)) t(d);
-- {}               | t | t
-- {"author": null} | f | t

So only documents where the key is missing count as empty. A blank string "" is also treated as not empty. #24275 reported the same symptom and was closed, but the code still uses the JSON element IS NULL check.