Allow aggregate pipeline on input documents before Atlas Search indexing
If Altas Search allowed us to apply an aggregate pipeline to input documents before they are sent to Atlas Search for indexing, it would open up a TON of new possibilities not currently supported. We could do data transformations, type conversions, synthetic fields, etc.. I've opened up a number of other suggestions for these things, but an input aggregate pipeline could solve all of them in one "easy" solution. Since aggregate pipelines already exist and are deeply embedded in MongoDB, I'm hoping it would be trivial to implement as well. Simply pick up one or more new documents to be indexed, run them through the pipeline and then send them to the Atlas Search engine for indexing.
Here's what that might look like:
{
"mappings": {
"dynamic": false,
"aggregate": [
{ <stage> },
...,
{
"$addFields": {
"my-synthetic-field": <expression>
}
}
],
"fields": {
"my-synthetic-field": {
"type": "string"
}
}
}
}
Here are some of the other suggestions I (or others) have made that could be solved with this approach:
https://feedback.mongodb.com/forums/924868-atlas-search/suggestions/47049898-allow-indexing-of-synthetic-fields
https://feedback.mongodb.com/forums/924868-atlas-search/suggestions/47049757-data-type-coercion-when-document-field-type-doesn
https://feedback.mongodb.com/forums/924868-atlas-search/suggestions/47049775-allow-sort-by-boolean-fields
https://feedback.mongodb.com/forums/924868-atlas-search/suggestions/47049775-allow-sort-by-boolean-fields
https://feedback.mongodb.com/forums/924868-atlas-search/suggestions/40734700-support-decimal128
David
shared this idea
-
Tom
commented
I like this idea, because it would also allow us not to have to add the entire collection to the search index, but only documents that meet a certain condition.
Typical scenarios would be, for example, archive products that are still in the collection but do not necessarily have to be in the search index. This would reduce the search index.