Internal-node lookup for Sharded Collections
We have a system with "Write First" strategy for multiple collections which we need to Shard.
We also need to run lookups command on those collections, sometimes from a collection to itself and sometimes from one Sharded collections to another Sharded collection.
For those collections we have the same Shard-Keys and the lookup operation will always contain the shard key as the lookup field (or part of it).
We need the ability to allow Internal-node lookup operations for those Sharded collection - a new command or a flag to the existing lookup command for allowing lookup on a Sharded collection but only in the same node where the "from" document resides.
We also need the ability to control the Sharding distribution of two or more collections with the same Shard-keys, to force those collections to be Sharded together so that documents from those different collections with the same Shard-key will be saved / managed in the exact same nodes.
By forcing this Sharding strategy we will be able to use the new "Internal-node" lookup command also for multiple collections.
Support for sharded collections in $lookup and $graphLookup was released in MongoDB 5.1.
Please check it out.
https://docs.mongodb.com/upcoming/release-notes/5.1/#aggregation
-
Eyal commented
super good idea! this would enable so many use cases and make the lookup capability much more useful