Capture pre-images for ONLY specified change events
Currently, pre-images are captured for all updates, replacements and deletes on a given collection.
Ask: Enable capture of pre-images for ['DELETED', ...] documents only.
Context: A customer uses a collection for ingestion, where 16-20M documents are ingested every 20 minutes. Documents are indexed with Elastic Search upon insertion, and are removed from Elastic Search upon deletion. However, the ingestion process requires subsequent operations: an average of 30 updates are made on each document, and the customer doesn't want to pay the penalty of generating pre- and post-images on updates.
Impact: Updating a collection of 80GB (20M docs of 4kb each) 30 times per document unnecessarily creates a Change Stream storage requirement of 2.4TB. This prevents the customer's adoption of full document change streams.
Matthew
shared this idea
-
Matthew
commented
Added improvement ticket https://jira.mongodb.org/browse/SERVER-94469
-
Matthew
commented
A large customer needs this.