Date filter at the start of a query is silently ignored
A dt filter at the start of a chat query is silently ignored — the date range is never applied to the search, and the raw filter text is sent to the LLM and the embedding model as if it were part of the question.
dt>="last week" what changed in my projectThe same query with the filter at the end works correctly:
what changed in my project dt>="last week"Cause
DateFilter.defilter requires whitespace before the filter:
def defilter(self, query):
# remove date range filter from query
query = re.sub(rf"\s+{self.date_regex}", " ", query)
query = re.sub(r"\s{2,}", " ", query).strip() # remove multiple spaces
return queryA filter at position 0 has nothing before it, so \s+ cannot match and the substitution does nothing. get_filter_terms uses the bare date_regex with no such requirement, so the two disagree about whether a filter is present.
Measured on master (ae229ca):
| query | get_filter_terms |
defilter |
|---|---|---|
head dt:"1984-01-01" tail |
["dt:'1984-01-01'"] |
'head tail' |
dt:"1984-01-01" tail |
["dt:'1984-01-01'"] |
'dt:"1984-01-01" tail' |
dt:"1984-01-01" |
["dt:'1984-01-01'"] |
'dt:"1984-01-01"' |
head,dt:"1984-01-01" |
["dt:'1984-01-01'"] |
'head,dt:"1984-01-01"' |
head (dt:"1984-01-01") |
["dt:'1984-01-01'"] |
'head (dt:"1984-01-01")' |
Any character that isn't whitespace suppresses the removal, so (, , and start-of-query are all affected. Whitespace-separated filters mid-query and at the end work, which is why this hasn't been obvious.
A related case: with two adjacent filters, the first one's trailing space is consumed as the second one's required leading \s+, so only one is removed.
dt>="1984-01-01" dt<="1984-12-31" summarize my year
-> defilter: 'dt>="1984-01-01" summarize my year'Why the date range is lost
defilter_query in processor/conversation/utils.py produces the natural-language query, then routers/helpers.py:1329 recovers the filters by subtracting it from the original:
defiltered_query = defilter_query(q)
filters_in_query = q.replace(defiltered_query, "").strip()When defilter returns the query unchanged, filters_in_query is the empty string. filters_in_query is what gets appended to each inferred search query at line 1371 and is the only thing EntryAdapters.apply_filters sees, so the date range never reaches the database query.
Measured end to end on master:
query : 'dt>="last week" what changed in my project'
defiltered : 'dt>="last week" what changed in my project' <- sent to LLM + embedding model
filters_in_query: ''
date range applied by apply_filters: [] <- filter silently dropped
query : 'what changed in my project dt>="last week"'
defiltered : 'what changed in my project'
filters_in_query: 'dt>="last week"'
date range applied by apply_filters: [1784563200.0, None] <- worksSo there are two user-visible effects for the same input:
- The date filter is not applied — results come from the whole knowledge base.
dt>="last week"is left in the text handed toextract_questionsand toembed_query, degrading the search embedding and the LLM's view of the question.
Neither is reported to the user; the query just quietly behaves as though no date filter were typed.
Expected behavior
defilter should remove every filter that get_filter_terms reports, regardless of what precedes it. Documented syntax in query-filters.md doesn't say a filter has to be preceded by other words.
Note
WordFilter.defilter has no such leading-whitespace requirement and handles all these positions correctly. FileFilter.defilter has a different problem (it doesn't strip -file:"..." at all), which is already covered by #1345 — this issue is only about DateFilter.
Environment
masteratae229ca- Python 3.12,
dateparserperuv.lock
Source: khoj-ai/khoj