Collection size metrics
Hi,
From time to time we have Atlas auto-scale up our clusters' disks. We then need to start analyzing why. In some cases it is organic growth of the data we store, but in some cases we are missing TTLs or they are misconfigured and we accumulate data we do not need.
In both cases, trying to realize what causes the disk increase is a very tedious process as some clusters have thousands of collections.
To overcome this, we started running a small utility that gathers some data over all our collections. It iterates on all the organizations, all the projects, all the clusters, all the databases and all the collections to collect total size, total index size, document count, etc, and then stores all this info once a day into BigQuery. We then created a simple looker dashboard over it and now it is very easy to find the collections that grow and figure out if that's OK or not.
We think this is something that will be beneficial for more than just us. We believe this should become some service + charts dashboard Atlas should keep for each project (or better - each organization).
We'd be happy to share more info with your product team if this seems a good idea to you.
Thanks,
Oren
