Hey Tom,

Thanks so much for the detailed response.

I should clarify, when Ben said consistent he meant consistent between queries, not consistent between other operators.

I recognize that replacing relevancy with text search to using vector search in a similar intended way requires using a different operator with different compatibility with other features, and we will definitely consider improving this in the future. Sort for example seems like a very natural fit to fit within the search + knnBeta stage, as users may want to view paged results based on a metadata field. You can produce this same behavior with a different syntax by using a subsequent $sort stage which can hold returned values in memory and perform a subsequent sort. If you exceed the memory limit, which could happen for large values of k, you can also spill to disk. More information on that stage is available here: https://www.mongodb.com/docs/manual/reference/operator/aggregation/sort/

In general there are a lot of features that make sense for full text search (such as analyzers) that might not make sense to carry over into knnBeta, so we are not necessarily optimizing for 1:1 parity in terms of syntax. The “should” example you provided makes a bit less sense from our perspective since knnOperator is used to rank results by vector proximity, not assess whether a vector fits within some other criteria. Where it makes sense we will try to add support, but we cannot guarantee that you can simply replace an autocorrect operator with knnBeta in the future.

Hey Benjamin,
i don't quite understand how this limitation is supposed to make the experience more consistent, because knn is thus completely different from the other operators.

Perhaps my expectations for the knn operator are also wrong, but I had thought that one could use this among other things as a replacement / improvement for the previous full text operators.

I will try to explain it briefly with a small example application.
For example this search stage:

{
$search: {
index: "fulltextsearch",
facet: {
operator: {
compound: {
filter: [
{
text: {
query: ["00017"],
path: "portfolioIds",
},
},
],
should: [
{
compound: {
must: [
{
autocomplete: {
query: "onplana",
path: "name.DE",
},
},
],
},
},
{
text: {
query: "00000000000onplana",
path: "matnr",
},
},
{
text: {
query: "onplana",
path: "toc",
},
},
],
minimumShouldMatch: 1,
},
},
facets: {
Z_P_SEGMENT: {
type: "string",
path: "attributes_localized.DE.Z_P_SEGMENT.value",
},
Z_P_BAUREIHE_NEU: {
type: "string",
path: "attributes_localized.DE.Z_P_BAUREIHE_NEU.value",
numBuckets: 500,
},
Z_P_PRODUKTKATEGORIE: {
type: "string",
path: "attributes_localized.DE.Z_P_PRODUKTKATEGORIE.value",
numBuckets: 100,
},
},
},
sort: {
"attributes_localized.DE.Z_P_SEGMENT.value_txt":
-1,
},
},
}

I have attached a picture of the associated application. A very basic search with three filter and a fulltext search (the autocomplete operator).
I have no idea how to replace the autocomplete operator with the knn operator and this is still a pretty simple example

My pains with this limitation are:

- No facets -> All the searches I have built so far include other filters in addition to a full-text search, for which I necessarily need facets

- No possibility to nest the knn operator in a compound -> For example: No possibility to make the knn-Operator optional with should

- No sort -> I love the new atlas search sort operator :(

- My search logic (and I guess other people's too) would have to be refactored, because all previous filters now have to go into the knn operator

Thank you for taking the time
Tom

Hey Tom, we will not be removing the requirement that knnBeta be a top level operator, but the functionality will evolve. Requiring knnBeta be a top level operator was done to make the feature more usable and provide a more consistent experience. Is there a specific outcome you're unable to achieve with this requirement?

Can you share more about the filter parameter being "omitted", it is an optional parameter, you don't have to have a filter.