Combine data lake snapshots into a single federated collection
A common use case for data analytics is to analyse how your data evolve over time.
For example, imagine you have an e-commerce database and your products have their price change every day. You may only store the price in your database but you'd like to make a chart that shows the evolution of your product prices over time (price y axis and time for x axis).
It is possible today to make this happen with the combination of Data Lake
and Data Federation
, but the Storage Configuration JSON need to be manually updated like this:
{
"databases": [
{
"collections": [
{
"name": "collectionName",
"dataSources": [
{
"datasetName": "v1$atlas$snapshot$Cluster0$env$collectionName$20230814T050424Z",
"provenanceFieldName": "provenance",
"storeName": "..."
},
{
"datasetName": "v1$atlas$snapshot$Cluster0$env$collectionName$20230813T050415Z",
"provenanceFieldName": "provenance",
"storeName": "..."
},
{
"datasetName": "v1$atlas$snapshot$Cluster0$env$collectionName$20230812T050424Z",
"provenanceFieldName": "provenance",
"storeName": "..."
},
...,
...,
...,
...,
...,
...,
]
}
],
}
],
...
}
The Data Federation configuration json is going to make thousands of lines and need to be maintained daily or maybe using a script + API. (3 lines of json per collection * 365 snapshots a year * 20 collections = 22'000 lines of json a year)
One idea could be to use a simple wildcard instead of the timestamp like this:
{
"databases": [
{
"collections": [
{
"name": "collectionName",
"dataSources": [
{
"datasetName": "v1$atlas$snapshot$Cluster0$env$collectionName$*",
"provenanceFieldName": "provenance",
"storeName": "..."
}
]
}
]
}
]
}
P.S: I know that time series collection could be useful in the specific example I just gave. But, sometime, you may want to analyse various properties over time, that's where a Data Lake solutions make sense.
![](https://secure.gravatar.com/avatar/758465a8faf4f0b376b36c10742f2666?size=40&default=https%3A%2F%2Fassets.uvcdn.com%2Fpkg%2Fadmin%2Ficons%2Fuser_70-6bcf9e08938533adb9bac95c3e487cb2a6d4a32f890ca6fdc82e3072e0ea0368.png)