Per-Shard Triggers to improve scalable/performance/resilience
In our geo-sharded cluster, we use Change Streams and Atlas Triggers.
Most of our pipelines include an exact match for the shard key - so we can process the data in the same region where the data resides.
However, due to the design of change streams with sharded clusters, the change streams are collected from all shards even if the pipeline matches only a specific match. In addition, if a shard has no changes to report, mongos waits up to 10 seconds (by default) before responding to the driver. This makes the data processing too slow.
I am suggesting to be able to limit what shard(s) are used as the data source of the Trigger.
This suggestion would also greatly improve scalable, performance and resilience of a MongoBD Global Cluster by not waisting resources on unrelated shards.
