Introduce a new field BucketLifeSpan Optional along with Granularity
An enhancement to MongoDB's management of time series collections could involve the introduction of a BucketLifeSpan attribute, in addition to the existing Granularity setting. This new, optional attribute would automate the duration a bucket can remain open, with the condition that Granularity should be less than or equal to BucketLifeSpan.
Consider a use case involving a time series collection for tracking data from 70,000 socket devices daily, with DeviceId as the metafield. Assuming data is organized into daily collections and granularity is set to minutes to optimally fill the buckets unless they reach their size limit.
For a collection named FirstDayCollection, by the end of the day—before data ingestion stops—we could have a minimum of 70,000 open buckets in memory, awaiting data. However, as we transition to SecondDayCollection and create a new set of at least 70,000 buckets for the next day's data ingestion, the buckets from the first day may still not be filled to their size capacity, leaving them in memory awaiting more data.
To address this issue without necessitating a server restart or finding other ways to clear the cache, we could utilize the proposed BucketLifeSpan field. This would allow for the automatic closure and data writing of buckets once they reach their lifespan, facilitating the setup of daily collections without forcing device restarts or manual cache management.
Implementing a BucketLifeSpan feature could significantly enhance the management of time series data in MongoDB, especially for use cases with high-volume, time-sensitive data ingestion.
I guess this kind of behavior would really helpful and should not that hard to be to implement and maybe making it optional.
Cheers.