In this old thread from 2016 (https://groups.google.com/forum/#!topic/mongodb-user/LCeFZZRz5EY) it was asked whether there was a way to increase the 100mb in memory limit of each stage of an aggregation pipeline. The responses centered around two points:

1. If too much memory is used per aggregation pipeline stage then it will reduce performance for the overall MongoDB database, impacting other queries negatively.
2. You can set allowDiskUse: true and revert to performing these pipeline stages on disk when they exceed 100mb.

I believe this subject needs to be revisited for the following reasons:

1. “Too much memory” is very subjective, and the 100mb number is arbitrary. The hardware for a typical MongoDB database setup in 2016 is not the same as in 2020. And what is the MongoDB use case for one company isn’t the use case for another company. Company A might have top of the line hardware they are running MongoDB on and only have a few concurrent users, and need large in memory aggregation queries to run fast so want to set a limit of 200mb or even higher. Company B might have lesser hardware and many concurrent users, and want to set the limit to 50mb or less.
2. The allowDiskUse: true option is nice to have, but it obviously comes at a huge performance hit once your aggregation pipeline stages exceed the 100mb limit, even on high performance SSD drives.

This really comes down to who knows best what the limit should be, the company’s developers and database / server ops teams or the MongoDB developers? The MongoDB developers obviously have much more knowledge on the inner workings of Mongo, but that advice can still be taken into account by having a default limit of 100mb. I believe very strongly that today in 2020 this limit should be configurable so that we can customize the behavior for our own specific situation. The documentation can of course provide a warning that changing the setting can impact performance (although everyone already knows that.)

If you want to get into the details of why my company needs to increase this limit, and why it has been holding us back from switching from SQL Server to MongoDB for the last year (despite spending 6 months developing a working Mongo solution) then we can get into specifics. But really that will distract away from the central issue, which is that different customers have different scenarios, and a “one size fits all” solution isn’t appropriate and needs to be customizable.

We frequently denormalise either full documents or subsets to different documents in order to speed up reading, create indexes or paginate/sort on fields.

Consider a user collection and a task collection, if a task can be assigned to the user, it makes sense to just put the user document on the task they are assigned. But an update to a user now requires you to update the user both in the user collection and all tasks with that user in the tasks collection.

This can be achieved but does introduce some complexity, however the introduction of updates using aggregation pipelines could eliminate this problem if they supported the $lookup operator.

So we could achieve the desired result using two simple operations

db.collection('users').updateOne({id: '1234'}, {$set: ...})
db.collection('tasks').updateMany({"user.id": '1234'}, [
{
$lookup: {
from: 'users',
localField: 'user.id',
foreignField: 'id',
as: 'user'
}
}
]);

This would make it really simple to synchronise data between documents in different collections

I have observed the different profile result with regard to $comment query operator in association with find and update operation respectively as follows;

• find operation shows a comment on the command.filter.$comment and the command.comment field in the system.profile collection.

  op: 'query',
  ns: 'db101.Bets',
  command: {
  find: 'Bets',
  filter: {
  _id: ObjectId("61a9db4b3bd34e4f68fb9abc"),
  '$comment': 'test-dba'
  },
  comment: 'test-dba',
  lsid: { id: UUID("43ebee67-3184-4ede-9cee-ecca7457861a") },
  '$db': 'db101'
  },

• update operation shows a comment only on the command.q.$comment field in the system.profile collection.

  op: 'update',
  ns: 'db101.Bets',
  command: {
  q: {
  _id: ObjectId("61a9db4b3bd34e4f68fb9abc"),
  '$comment': 'test-dba'
  },
  u: { '$set': { odds: 0.5 } },
  multi: false,
  upsert: false
  },

The syntax for joins that bring back multiple documents from foreign collections is very straightforward and yields exactly what one would expect, but simple joins that are bread and butter in SQL require very convoluted and expensive to run syntax.

Consider a product database that has products, categories and reviews collections. Each product has a unique category and may have multiple reviews. Getting all reviews in an aggregation is very straightforward (top stage), but getting categories, similar to SQL, is as convoluted as it gets (bottom stage).

db.products.aggregate(
[
    //
    // Document aggregates naturally aggregate foreign documents
    // into the primary document.
    //
    {$lookup: {
      from: "reviews",
      localField: "_id",
      foreignField: "product_id",
      as: "reviews"
    }},

    //
    // A simple join that is supposed to bring a couple of fields
    // into the primary document translates into this convoluted
    // syntax just to get those two fields.
    //
    {$lookup: {
      from: "categories",
      let: {category_id: "$category"},
      pipeline: [
        {$match: {
            $expr: {$eq: ["$_id", "$$category_id"]}
        }},
        {$project: {
            _id: 0,
            cat_name: "$category",
            cat_description: "$description"
        }}
      ],
      as: "category_details"
    }},
    {$replaceRoot: {
      newRoot: {
        $mergeObjects: [
          "$$ROOT", {$arrayElemAt: ["$category_details", 0]}
        ]
      }
    }},
    {$project: {
        category_details: 0
    }}
])

Notice that keeping category.category and category.description in products, the noSQL-way, isn't an option because updating 10000's of products when category or description change is just not a good idea any way one looks at it. Denormalization is useful in many noSQL scenarios, but this isn't one of them.

Having some simple join syntax that would identify the joined collection as a 1-to-1 join and would bring field names and values as if they existed in the primary document, possibly aliased via a projection-like syntax, would be extremely helpful in keeping aggregations readable, keeping data reasonably normalized, and would run much faster because it would be just multiple parallel BSON append calls when constructing aggregation results documents.

Data for the query:

products:

{"_id":{"$oid":"6140e6c6dfe7b9d6e597d07e"},"name":"Product A","category":1,"price":111}
{"_id":{"$oid":"6140e6c6dfe7b9d6e597d07f"},"name":"Product B","category":2,"price":222}
{"_id":{"$oid":"6140e6c6dfe7b9d6e597d080"},"name":"Product C","category":1,"price":333}

categories:

{"_id":1,"category":"Clothing","description":"All kinds of clothing"}
{"_id":2,"category":"Electronics","description":"Cool gadgets"}

reviews:

{"_id":{"$oid":"6140e7cadfe7b9d6e597d08a"},"product_id":{"$oid":"6140e6c6dfe7b9d6e597d07e"},"review":"Product A is great","rating":4}
{"_id":{"$oid":"6140e813dfe7b9d6e597d090"},"product_id":{"$oid":"6140e6c6dfe7b9d6e597d07f"},"review":"This product is horrible","rating":1}
{"_id":{"$oid":"6140e8eff4ff76778b48ba88"},"product_id":{"$oid":"6140e6c6dfe7b9d6e597d07f"},"review":"This product is not the best, but usable","rating":2}