What problem are you trying to solve?

Focus on the what and why of the need you have, not the how you'd like it solved.

When an Atlas Organization Service Account secret expires (in a standard 2-secret
rotation pattern), Atlas Search's mongot process retains the expired secret in its
internal auth cache. From that moment on, all $vectorSearch aggregations on the
cluster return:

pymongo.errors.OperationFailure:
  PlanExecutor error during aggregation ::
  caused by :: Authentication failed.
  {'code': 8, 'codeName': 'UnknownError'}

This error message strongly suggests the user has a credentials or permissions
problem on their query — when in reality their connection succeeds,
find/count/listSearchIndexes all work with the same credentials, and the user's role
(atlasAdmin, readWrite, etc.) is correct. The actual root cause is server-side
mongot state.

Engineers hitting this spend hours debugging their own users, roles, indexes,
drivers, and network — none of which is the issue. The Atlas auto-triggered
HOST_MONGOT_RESTARTED events that fire 5 minutes after the secret expiration do not
clear the stale state either.

What would you like to see happen?

Describe the desired outcome or enhancement.

Two improvements:

1. ERROR MESSAGE — make it actionable. Replace "PlanExecutor :: Authentication
   failed" with something that points at the actual root cause, e.g.:

   "Atlas Search internal authentication state is stale (likely an expired
   Service Account secret that was not explicitly deleted). Verify with
   GET /orgs/{ORG_ID}/serviceAccounts/{SA_ID} and DELETE any secret whose
   expiresAt is in the past. See [docs URL]."

   This would reduce diagnosis time from hours to minutes.

2. DOCUMENTATION — the SA rotation guide currently describes "create new secret,
   update applications, wait for old to expire". Add an explicit final step:

   "After the old secret expires, DELETE it via Admin API
   (DELETE /orgs/{ORG_ID}/serviceAccounts/{SA_ID}/secrets/{SECRET_ID}).
   Atlas Search caches the secret internally; without explicit deletion,
   $vectorSearch queries will fail with an opaque auth error."

3. (Optional, server-side fix) Make the HOST_MONGOT_RESTARTED events triggered
   automatically by Atlas in response to secret expiration actually refresh mongot's
   auth cache. Currently they restart the process but don't clear the bad state.

Why is this important to you or your team?

Explain how the request adds value or solves a business need.

I run a course infrastructure with ~55 students using Atlas Search for a
RAG/vector-search exercise. The cluster's SA secret expired between sessions; if it
had happened during class, the entire session would have been blocked with no
obvious recovery path.

More broadly: anyone with an SA rotation policy will eventually hit this bug. The
misleading error sends them to debug the wrong layer. Several similar reports exist
in the wild (different language drivers, different error wrappers) suggesting the
underlying issue isn't isolated.

Reducing diagnosis time from ~8 hours (my case) to ~5 minutes (with an actionable
error) is a major DX improvement for a customer-impacting failure mode.

What steps, if any, are you taking today to manage this problem?

1. Built a runbook (SA_SECRET_ROTATION.md) documenting the symptoms, verification,
   fix, and prevention steps.

2. Wrote helper scripts:

   • sa_secret_status.py — flags expired SA secrets still present

   • delete_expired_sa_secret.py — safe DELETE (refuses if not yet expired)

   • quick_vectorsearch_test.py — sanity check post-fix

3. Calendar reminders to verify SA secret state before each known expiration date.

4. Public write-ups for the community:

   • https://stackoverflow.com/questions/79945813

   • https://www.reddit.com/r/mongodb/comments/1tm61jt/atlas_search_vectorsearch_fai
     ls_with/

These help individual operators recover but don't fix the underlying UX issue or
prevent the next person from rediscovering it the hard way.