56 results found

1. Add eu-west-3 as a option for AWS private endpoint

   Currently Data Lake doesn't support France/Paris eu-west-3 to set up a private endpoint. It would be great to support eu-west-3 as well.

   17 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   2 comments  ·  Infrastructure Options  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

2. Simplified JSON support for $out to S3

   The ability to $out to S3 from a federated database instance is a game-changer for those working with their own data warehouses and data lakes.

   One improvement that would make it better would be to support simplified JSON for json exports. Currently, $out uses extended json v2, which may not be compatible for systems reading from the destination S3 bucket, which require simplified JSON (which aligns with other tools like kafka source connector). Technically, it is possible to make this conversion yourself with clever use of the $toString aggregation pipeline operator in stages preceding $out. However there are several challenges:
   + Increased computation time
   + The more general a solution is needed (ie--in cases where you don't know/cannot make assumptions about the schema), the more complex the aggregation stages become. One such solution would be to $objectToArray the document, $map over the resulting array, converting the v field conditionally, then $arrayToObject back and $replaceRoot to recompose the document. This is already complex enough for most MongoDB users; handling nested arrays and objects makes it vastly more complex.

   The ability to $out to S3 from a federated database instance is a game-changer for those working with their own data warehouses and data lakes.

   One improvement that would make it better would be to support simplified JSON for json exports. Currently, $out uses extended json v2, which may not be compatible for systems reading from the destination S3 bucket, which require simplified JSON (which aligns with other tools like kafka source connector). Technically, it is possible to make this conversion yourself with clever use of the $toString aggregation pipeline operator in stages preceding $out. However there are several challenges:… more

   6 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   1 comment  ·  File Formats  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

3. a GUI for setting SQL Query sampling size (like in bi connector Atlas console)

   Provide the ability to set SQL Query sampling size (like in bi connector Atlas console). This would allow our business customers that use the Power BI/ Tableu to easily set and manage sampling without having to use cli command (i.e., sqlGenerateSchame) whenever a new document is added to the database.

   5 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   1 comment  ·  Query Functionality  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

4. Write "_SUCCESS" File when finish data exporting

   We use MongoDB to store time-series data, and export the data via Data Federation incrementally on daily basis onto s3 as Parquet. The data is relative big, and duration to export data varies from day to day. It’s hard for downstream services to know when data exporting completes. Sometimes, the downstream service start reading the parquets while MongoDB is writing, which causes partial extraction. Normally, a big data job would create a flag file, such as _SUCCESS, to indicate that the job has finished writing the dataset. This file serves as a marker, indicating that all tasks associated with the job were finished successfully, and the data files in the directory are complete and consistent. Could you consider adding such feature?

   const outStage = {
   "$out": {
   "s3": {
   "bucket": ${aws-s3-bucket},
   "filename": ${fileName},
   "format": {
   "name": "parquet",
   "maxFileSize": "1GB",
   "maxRowGroupSize": "128MB",
   }
   }
   }
   }
   const coll = db.collection(collName);
   await coll.aggregate([
   matchStage,
   outStage
   ], { background: true }).toArray();
   console.log(Job Submitted);

   We use MongoDB to store time-series data, and export the data via Data Federation incrementally on daily basis onto s3 as Parquet. The data is relative big, and duration to export data varies from day to day. It’s hard for downstream services to know when data exporting completes. Sometimes, the downstream service start reading the parquets while MongoDB is writing, which causes partial extraction. Normally, a big data job would create a flag file, such as _SUCCESS, to indicate that the job has finished writing the dataset. This file serves as a marker, indicating that all tasks associated with the… more

   1 vote

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   0 comments  ·  Automation  ·  Edit…  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

5. Allow a single timestamp field to be split by Year Month Day and Hour for folders instead of just one field like Year in filepath for Azure

   I checked internally, and it has been confirmed that an attribute can only appear once in a template. If Atlas Data Federation (ADF) has a template like the one you are using, it wouldn't know what value to assign to StatusDatetime because it's being assigned multiple values. Unfortunately, ADF doesn't support defining a single field value across multiple segments of the path. Instead, each of those segments should be different attributes.

   {
   "path": "/HistoryCollection/{StatusDatetime isodate:Year}/StatusDatetime isodate:Month}/StatusDatetime isodate:Day}/StatusDatetime isodate:Hour}/{RecordSource string}/{Status string}/*",
   "storeName": "sampledatabase"
   }

   We would like to have the store we are creating as an archive be queried by StatusDatetime RecordSource and Status so it matches the queries we use against the live collections under Federation instead of extracting the Year Month Day and Hour fields which don't exist in the live collection.

   I checked internally, and it has been confirmed that an attribute can only appear once in a template. If Atlas Data Federation (ADF) has a template like the one you are using, it wouldn't know what value to assign to StatusDatetime because it's being assigned multiple values. Unfortunately, ADF doesn't support defining a single field value across multiple segments of the path. Instead, each of those segments should be different attributes.

   {
   "path": "/HistoryCollection/{StatusDatetime isodate:Year}/StatusDatetime isodate:Month}/StatusDatetime isodate:Day}/StatusDatetime isodate:Hour}/{RecordSource string}/{Status string}/*",
   "storeName": "sampledatabase"
   }

   We would like to have the store we are creating as an archive be queried by StatusDatetime… more

   1 vote

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   0 comments  ·  Query Functionality  ·  Edit…  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

6. Background aggregation queries return a query ID or correlation ID to be able to quickly poll for completion
   1 vote

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   0 comments  ·  Query Functionality  ·  Edit…  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

7. Support Azure Data Federation private endpoint

   Now you have supported Azure blobs for data federation it will be great to have a private endpoint connection to the storage account

   1 vote

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   0 comments  ·  Connectors  ·  Edit…  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

8. Include support for Federated Database Instance with Data API services.

   •  

   1 vote

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   0 comments  ·  Edit…  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

9. sqlGetSchema Sampling

   There is currently no way to know what a current sampling size is on a collection. I would recommend adding this to the sqlGetSchema output.

   3 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   0 comments  ·  Query Functionality  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

10. Support for Superset and other Python DB-API / SQLAlchemy connections to SQL Atlas

    Superset uses SQL Alchemy and/or Python DB-API drivers, not JDBC or ODBC drivers. Superset is the most popular, open-source Apache visualization tool.

    Others have made it work like this: https://preset.io/blog/building-database-connector/

    7 votes

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    0 comments  ·  Connectors  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

11. Modify SQL Query Schema

    Add a gui front end to sqlSetSchema. I would be good for customers to return a schema, then have the ability to 'surgically' add fields. This would reduce reliance on large sample sizes. With BI Connector in Atlas, sometimes we have to wait hours for the sqlDaemon to restart, resulting in long downtime.

    2 votes

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    0 comments  ·  Query Functionality  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

12. Support to export backups to Azure Blob Storage in Atlas

    I would like the capability to export my cloud snapshots to Azure blob storage.

    7 votes

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    0 comments  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

13. AWS IAM AuthN for Atlas SQL

    Support AWS IAM Authentication mechanism in JDBC and ODBC drivers (Atlas SQL)

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    0 comments  ·  Connectors  ·  Edit…  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

14. Create a read/write Data Federation connection string

    Some customers need a connection string both to the cluster and to Online Archive with the ability to write to the cluster only.

    So far, the only option is to use more than a connection string in the application.

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    0 comments  ·  Query Functionality  ·  Edit…  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

15. Implement a feature to track data download volume per DB user

    In order to enhance data security and prevent unauthorized data exfiltration, our team proposes the implementation of a metric within MongoDB Atlas that allows administrators to monitor and measure the amount of data downloaded by each database user over a specified period. This feature would provide critical insights into user behavior, helping to identify unusual data access patterns or potential data breaches. By tracking network data usage at the user level, we can more effectively audit data access and transfer, ensuring that data is used appropriately and in compliance with organizational data governance policies. This granularity in monitoring would be a significant step forward in data management and security within MongoDB Atlas, offering a proactive tool for administrators in safeguarding sensitive data.

    In order to enhance data security and prevent unauthorized data exfiltration, our team proposes the implementation of a metric within MongoDB Atlas that allows administrators to monitor and measure the amount of data downloaded by each database user over a specified period. This feature would provide critical insights into user behavior, helping to identify unusual data access patterns or potential data breaches. By tracking network data usage at the user level, we can more effectively audit data access and transfer, ensuring that data is used appropriately and in compliance with organizational data governance policies. This granularity in monitoring would be… more

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    0 comments  ·  Reporting  ·  Edit…  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

16. Support AWS IAM for Data Federation Authentication

    We would like to be able to connect to the Federated Database Instance using AWS IAM for Authentication just like you can for Atlas Clusters.

    5 votes

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    2 comments  ·  Infrastructure Options  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

17. Add support region at south korea for federated database
    6 votes

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    0 comments  ·  Infrastructure Options  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

18. Combine data lake snapshots into a single federated collection

    A common use case for data analytics is to analyse how your data evolve over time.
    For example, imagine you have an e-commerce database and your products have their price change every day. You may only store the price in your database but you'd like to make a chart that shows the evolution of your product prices over time (price y axis and time for x axis).

    It is possible today to make this happen with the combination of Data Lake and Data Federation, but the Storage Configuration JSON need to be manually updated like this:

    {
      "databases": [
        {
          "collections": [
            {
              "name": "collectionName",
              "dataSources": [
                {
                  "datasetName": "v1$atlas$snapshot$Cluster0$env$collectionName$20230814T050424Z",
                  "provenanceFieldName": "provenance",
                  "storeName": "..."
                },
                {
                  "datasetName": "v1$atlas$snapshot$Cluster0$env$collectionName$20230813T050415Z",
                  "provenanceFieldName": "provenance",
                  "storeName": "..."
                },
                {
                  "datasetName": "v1$atlas$snapshot$Cluster0$env$collectionName$20230812T050424Z",
                  "provenanceFieldName": "provenance",
                  "storeName": "..."
                },
                ...,
                ...,
                ...,
                ...,
                ...,
                ...,
              ]
            }
          ],
    
        }
      ],
      ...
    }
    

    The Data Federation configuration json is going to make thousands of lines and need to be maintained daily or maybe using a script + API. (3 lines of json per collection * 365 snapshots a year * 20 collections = 22'000 lines of json a year)

    One idea could be to use a simple wildcard instead of the timestamp like this:

    {
      "databases": [
        {
          "collections": [
            {
              "name": "collectionName",
              "dataSources": [
                {
                  "datasetName": "v1$atlas$snapshot$Cluster0$env$collectionName$*",
                  "provenanceFieldName": "provenance",
                  "storeName": "..."
                }
              ]
            }
          ]
        }
      ]
    }
    

    P.S: I know that time series collection could be useful in the specific example I just gave. But, sometime, you may want to analyse various properties over time, that's where a Data Lake solutions make sense.

    A common use case for data analytics is to analyse how your data evolve over time.
    For example, imagine you have an e-commerce database and your products have their price change every day. You may only store the price in your database but you'd like to make a chart that shows the evolution of your product prices over time (price y axis and time for x axis).

    It is possible today to make this happen with the combination of Data Lake and Data Federation, but the Storage Configuration JSON need to be manually updated like this:

    {
      "databases": [

    … more
    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    0 comments  ·  Storage Configuration  ·  Edit…  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

19. readPreference=Secondary for Federated Data Store

    We're using Online Archives to ensure our cluster data size stays manageable, while at the same time enabling our data extraction process to have access to older date on an exception basis.

    Not being able to set the read preference on our mongoexport connection string for our federated data source ( https://www.mongodb.com/docs/atlas/app-services/mongodb/read-preference/) is a significant issue for our use case.

    12 votes

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    0 comments  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

20. Add last modified timestamp to Data Federation Provenance for S3

    It would be great to have the last modified timestamp of a file in S3 returned with the provenance functionality in Atlas Data Federation.

    2 votes

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    0 comments  ·  Storage Configuration  ·  Delete…  ·  Admin →
    How important is this to you?

    Not at all You must login first! Important You must login first! Critical You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close