Allow disabling auto removal for capped collections
Capped collections work as a circular buffer - when adding new documents after the size limit is reached, old documents are removed.
I would like to have an option for the collection, where this behavior can be changed - instead of removing old documents, the insert should fail with a specific error (eg: collection full). This would allow implementing log-rotation, by creating a new collection whenever the old one is full.
I have two services, one creating documents (events), and another one processing them. I'm looking for a reliable way to:
- process all documents from start, without skipping or double-processing any of them
- after catching up, process all newly inserted documents
- avoid updating the documents by the processor (eg: avoid adding a "processed" field to the processed documents)
In order to be able to iterate all documents, and be able to continue from where I left off during an error/re-deployment of the processor service, the documents need to have some kind or order, where newly inserted documents can't get added before an already processed one. My options right now:
- use a normal collection, with a global auto-increment id
- this is what i'm using, but it's a bit complicated with mongodb
- use a capped collection, where the insertion order is kept
- there is a risk of "losing" documents if the processor gets too behind, that's not acceptable
- using a "large enough" storage is not scalable