External Secrets Operator (ESO)

Manage every CrewAI Platform secret through your existing secret store — AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, or HashiCorp Vault.

Overview

The CrewAI Platform chart can source every runtime secret from an external secret store via the External Secrets Operator. When enabled fully, your backend (AWS Secrets Manager, Vault, etc.) becomes the canonical source of truth — the chart neither generates random values nor reads any secret material from values.yaml.

This page covers:

• How CrewAI’s chart consumes secrets — three modes, what each looks like
• Fresh install with ESO — clean deployment from day one
• Migration paths — moving an existing deployment onto ESO
• Backend key reference — what to populate, where
• The pre-upgrade wait Job — how the chart sequences sync
• Troubleshooting — what to do when things go wrong

This page applies to direct Helm and ArgoCD deployments. KOTS / Replicated deployments auto-populate the secrets stored in crewai-secrets from license fields and do not need ESO. If you’re on KOTS and want to layer ESO on top, the same flags apply but you’ll need to coordinate with Replicated’s secret hydration.

How the chart consumes secrets

The chart supports three modes, controlled by two values: externalSecret.enabled and externalSecret.manageAutoGenerated.

Mode	externalSecret.enabled	externalSecret.manageAutoGenerated	Where secrets come from
A. Inline	false	(ignored)	Chart generates auto-gen values via lookup + randAlphaNum. Pre-set values in secrets: block override.
B. Partial ESO	true	false	ESO pulls DB, GitHub, SSL, AWS/Azure keys. Auto-gen keys are missing from the materialized Secret — pods CrashLoop.
C. Full ESO	true	true	ESO pulls every key the chart’s pods need from your backend. WHARF_POSTGRES_URL and CREWAI_OAUTH_API_KEY are derived inside ESO; everything else is fetched.

Mode A is the chart’s default. Mode B existed before the manageAutoGenerated flag was introduced and is not a viable production state — it’s the historical default that ships an incomplete Secret. Mode C is what you want for any deployment that treats the external store as the source of truth.

For the rest of this page, “ESO” means Mode C unless explicitly noted.

Fresh install with ESO

If you are deploying CrewAI Platform for the first time and want ESO from day one, follow these steps in order.

Prerequisites

• External Secrets Operator installed and healthy in the cluster (typically in the external-secrets namespace).
• A SecretStore resource that can authenticate against your backend. The chart can render one via secretStore.enabled: true (see Configuration Reference), or you can bring your own.
• An empty path reserved in your backend for CrewAI secrets (e.g. AWS SM secret crewai/platform/prod). For OAuth, optionally a second path (e.g. crewai/platform/prod/oauth).

1. Generate the auto-gen secret values

The chart generates these values for you on a non-ESO install via lookup + randAlphaNum. Even for a fresh ESO install, the easiest way to get reasonable defaults is to render them once locally:

helm template crewai-platform <chart> --values your-values.yaml \
  | yq 'select(.kind == "Secret" and (.metadata.name | contains("-secrets"))) .data
        | to_entries | .[]
        | .key + ": " + (.value | @base64d)' \
  | grep -E '^(SECRET_KEY_BASE|ENCRYPTION_KEY|ACTIVE_RECORD_ENCRYPTION_|REGISTRY_HTTP_SECRET|CREWAI_PLUS_INTERNAL_API_KEY|WHARF_JWT_SECRET|DEPLOYMENT_INSTANCE_JWT_SECRET|CUBE_JWT_SECRET|FACTORY_DEBUG_TOKEN|OIDC_PRIVATE_KEY|OIDC_KEY_ID|OAUTH_COOKIE_SECRET|OAUTH_DB_ENCRYPTION_KEY|OAUTH_INTERNAL_API_KEY)'

Alternative: generate with openssl

If you prefer to generate independently of the chart, use these commands. The | tr -dc 'A-Za-z0-9' | head -c N filter mirrors the chart’s randAlphaNum output exactly — pure alphanumeric, no +/= from base64 or special characters that would otherwise drift from the values a non-ESO chart install produces.

# 64-char alphanumeric: SECRET_KEY_BASE, ACTIVE_RECORD_ENCRYPTION_* (×3),
# DEPLOYMENT_INSTANCE_JWT_SECRET, FACTORY_DEBUG_TOKEN, WHARF_JWT_SECRET,
# CUBE_JWT_SECRET, CREWAI_PLUS_INTERNAL_API_KEY, OAUTH_COOKIE_SECRET,
# OAUTH_INTERNAL_API_KEY
openssl rand -base64 64 | tr -dc 'A-Za-z0-9' | head -c 64

# 32-char alphanumeric: REGISTRY_HTTP_SECRET
openssl rand -base64 32 | tr -dc 'A-Za-z0-9' | head -c 32

# 64-char hex: ENCRYPTION_KEY, OAUTH_DB_ENCRYPTION_KEY
openssl rand -hex 32

# OIDC signing key (multi-line RSA PEM)
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048

# OIDC key id
echo "crewai-oidc-$(openssl rand -hex 8)"

2. Upload to your backend

Store the values as a JSON document at externalSecret.secretPath. Property names must match the lowercased Secret key (SECRET_KEY_BASE → secret_key_base) unless you override via autoGeneratedPropertyMap.

aws secretsmanager create-secret \
  --name crewai/platform/prod \
  --region us-west-2 \
  --secret-string "$(jq -n \
    --arg skb '<SECRET_KEY_BASE value>' \
    --arg ek '<ENCRYPTION_KEY value>' \
    --rawfile opk /path/to/oidc-private-key.pem \
    '{secret_key_base: $skb, encryption_key: $ek, oidc_private_key: $opk}')"

See Backend key reference below for the complete list.

OIDC_PRIVATE_KEY is a multi-line PEM. Use jq --rawfile (as shown) to preserve newlines through JSON round-tripping. Hand-edited JSON or --secret-string "$(cat file.pem)" easily mangles the line breaks.

3. Configure values

# your-values.yaml
externalSecret:
  enabled: true
  manageAutoGenerated: true
  secretStore: my-secret-store
  secretPath: crewai/platform/prod
  databaseSecretPath: crewai/platform/prod/db
  # Optional — only if oauth.enabled and OAuth keys live at a different path
  oauthSecretPath: crewai/platform/prod/oauth
  # Optional — add extras like UV_DEFAULT_INDEX that aren't in the default set
  autoGeneratedPropertyMap:
    UV_DEFAULT_INDEX: uv_default_index

Do not populate the secrets: block. With externalSecret.enabled=true the chart skips crewai-secrets.yaml entirely; values there are silently ignored.

4. Install

helm install crewai-platform <chart> --version <version> --values your-values.yaml -n <namespace>

The chart’s pre-upgrade flow (which also runs on first install) sequences ESO sync before pod startup. You’ll see:

crewai-eso-wait-sa, Role, RoleBinding created           (hook weight -50)
crewai-external-secret created                          (hook weight -30)
crewai-wait-for-eso-<rev> Job runs                      (hook weight -20)
  → polls until materialized Secret has every required key
crewai-pre-upgrade-migration Job runs                   (hook weight -5)
  → runs Rails db:migrate against the now-complete Secret
Main manifest applies, Deployments roll

If any backend property is missing, the wait Job times out with an explicit message naming the missing key. Add it to your backend, re-run helm install.

Migration paths

If you have an existing CrewAI deployment and are moving onto ESO, follow the path that matches your current state.

Path 1 — From inline secrets (Mode A) to full ESO (Mode C)

You’re currently running with externalSecret.enabled: false. The chart generates the auto-gen secrets and stores them in the in-cluster Secret <release>-secrets. You want to move to ESO.

This is the safest migration path. The existing Secret is annotated with helm.sh/resource-policy: keep — it survives the cutover, and ESO adopts it on first reconcile.

Step 1 — Extract the current values. They’re what your pods are currently using; reusing them avoids rotating session-encryption keys (which would invalidate all existing user sessions) or Active Record encryption keys (which would render encrypted database columns unreadable).

NS=<your-namespace>
RELEASE=<release-name>

kubectl get secret -n "$NS" "$RELEASE-secrets" -o json \
  | jq -r '.data | to_entries[] | "\(.key)\t\(.value | @base64d)"' \
  > /tmp/existing-secrets.tsv

If oauth.enabled: true, also extract <release>-oauth-secrets.

Step 2 — Upload to your backend. Map each Secret key to its lowercased property name (or whatever your autoGeneratedPropertyMap overrides specify). For OIDC_PRIVATE_KEY, use jq --rawfile to preserve newlines.

# Build the AWS SM JSON, then:
aws secretsmanager create-secret --name crewai/platform/prod --region us-west-2 \
  --secret-string file:///tmp/platform.json

Step 3 — Update values and upgrade.

externalSecret:
  enabled: true
  manageAutoGenerated: true
  secretStore: my-secret-store
  secretPath: crewai/platform/prod
  databaseSecretPath: crewai/platform/prod/db
  # oauthSecretPath: crewai/platform/prod/oauth   # if oauth.enabled

Delete the secrets: block from your values file — it has no effect with ESO enabled, and keeping it gives future readers a false sense that those values are wired in.

helm upgrade <release> <chart> --version <new-version> --values your-values.yaml -n "$NS"

The inline Secret persists (via keep). The new ExternalSecret is created during the pre-upgrade hook, ESO adopts the existing Secret (sets its OwnerReference), the wait Job verifies completeness, and migrations run. Pods do not need to restart unless any backend value differs from the current in-cluster value.

Path 2 — From partial ESO (Mode B) to full ESO (Mode C)

You’re currently running with externalSecret.enabled: true and manageAutoGenerated unset (or false). Your <release>-secrets materialized Secret is partial — it has the keys the old ExternalSecret requested (DB, GitHub, SSL, AWS/Azure) but is missing the auto-gen keys. Pods are likely already failing on missing env vars, or surviving only because the Secret was created by an even older non-ESO install and the old keys have been frozen in place via helm.sh/resource-policy: keep.

This path requires a one-time orphan-delete because Helm refuses to convert a regular resource into a hook resource in place. The new chart annotates the ExternalSecret as a pre-install,pre-upgrade hook; on first sync, Helm sees a same-name non-hook resource already exists and aborts with:

externalsecrets.external-secrets.io "crewai-external-secret" already exists

This step is only required on the first upgrade from a chart version that didn’t use hooks for the ExternalSecret. Subsequent upgrades are fully idempotent — Helm handles the hook lifecycle automatically via before-hook-creation,hook-failed.

Step 1 — Confirm AWS SM has every key the new ExternalSecret will request. This is where most migrations stall. Take inventory of what’s in your backend vs. what the chart’s full mode needs:

expected="secret_key_base encryption_key
  active_record_encryption_primary_key
  active_record_encryption_deterministic_key
  active_record_encryption_key_derivation_salt
  registry_http_secret factory_debug_token
  deployment_instance_jwt_secret
  oidc_private_key oidc_key_id wharf_jwt_secret
  github_token github_crew_studio_token github_client_secret
  github_webhook_secret github_app_private_key
  internal_api_key
  platform_ssl_key platform_ssl_cert
  crew_ssl_key crew_ssl_cert
  entra_client_secret"   # if includes_azure_credentials

actual=$(aws secretsmanager get-secret-value --secret-id crewai/platform/prod \
         --region us-west-2 --query SecretString --output text | jq -r 'keys[]')

for k in $expected; do
  echo "$actual" | grep -qx "$k" && printf "  ok  %s\n" "$k" || printf "MISS  %s\n" "$k"
done

For any MISS, extract the value from the live cluster Secret (still alive thanks to keep):

NS=<your-namespace>
kubectl get secret -n "$NS" <release>-secrets -o json \
  | jq -r ".data.\"<MISSING_KEY>\" | @base64d"

Upload it to the backend.

Step 2 — Orphan-delete the existing ExternalSecret(s). This removes the CR but preserves the materialized Secret (which is what your running pods depend on). --cascade=orphan strips the OwnerReference from the Secret instead of garbage-collecting it.

kubectl delete externalsecret -n "$NS" <release>-external-secret --cascade=orphan
kubectl delete externalsecret -n "$NS" <release>-oauth-external-secret --cascade=orphan 2>/dev/null || true

Running pods continue to function — they cached their env vars at startup and don’t re-read the Secret. The Secret resource itself stays in the cluster with no owner; ESO will adopt it on first reconcile after the upgrade.

Step 3 — Update values:

externalSecret:
  enabled: true
  manageAutoGenerated: true            # ← the new flag
  secretStore: my-secret-store         # unchanged
  secretPath: crewai/platform/prod     # unchanged
  databaseSecretPath: crewai/platform/prod/db   # unchanged
  oauthSecretPath: crewai/platform/prod/oauth   # if oauth.enabled

Step 4 — Upgrade.

helm upgrade <release> <chart> --version <new-version> --values your-values.yaml -n "$NS"

Helm now applies the hook-annotated ExternalSecret cleanly during the pre-upgrade phase, the wait Job confirms the Secret has every key, and migrations run.

Path 3 — From values-file pre-set secrets to ESO

You may have been on a hybrid where you pre-set every secret in your secrets: block in values.yaml (the Pre-Set All Secrets pattern). Migrating to ESO is the same as Path 1 — extract the values from the cluster Secret (or directly from your values file), upload to the backend, then flip the flags. Delete the secrets: block once values are confirmed in the backend.

Backend key reference

Every property the chart will request from your secret store, with defaults and conditions.

Main secret (at externalSecret.secretPath)

Pulled when manageAutoGenerated: true. Property names below are the defaults; override per-key via autoGeneratedPropertyMap.

Property name	Maps to env var	Always required?
password	DB_PASSWORD (at databaseSecretPath, not secretPath)	yes
secret_key_base	SECRET_KEY_BASE	yes
encryption_key	ENCRYPTION_KEY	yes
active_record_encryption_primary_key	ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY	yes
active_record_encryption_deterministic_key	ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY	yes
active_record_encryption_key_derivation_salt	ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT	yes
registry_http_secret	REGISTRY_HTTP_SECRET	yes
factory_debug_token	FACTORY_DEBUG_TOKEN	yes
deployment_instance_jwt_secret	DEPLOYMENT_INSTANCE_JWT_SECRET	yes
oidc_private_key	OIDC_PRIVATE_KEY (multi-line PEM)	yes
oidc_key_id	OIDC_KEY_ID (must rotate with oidc_private_key)	yes
wharf_jwt_secret	WHARF_JWT_SECRET	only if wharf.enabled: true
cube_jwt_secret	CUBE_JWT_SECRET	only if cube.enabled: true
internal_api_key	CREWAI_PLUS_INTERNAL_API_KEY	yes
github_token	GITHUB_TOKEN	yes (chart-managed)
github_crew_studio_token	GITHUB_CREW_STUDIO_TOKEN	yes
github_client_secret	GITHUB_CLIENT_SECRET	yes
github_webhook_secret	GITHUB_WEBHOOK_SECRET_TOKEN	yes
github_app_private_key	GITHUB_APP_PRIVATE_KEY (multi-line PEM)	yes
platform_ssl_key	SSL_PRIVATE_KEY	yes
platform_ssl_cert	SSL_CERTIFICATE	yes
crew_ssl_key	CREW_SSL_KEY	yes
crew_ssl_cert	CREW_SSL_CERT	yes
aws_access_key_id / aws_secret_access_key	AWS_*	only if includes_aws_credentials: true
azure_storage_access_key / azure_client_secret / entra_client_secret	AZURE_* / ENTRA_ID_CLIENT_SECRET	only if includes_azure_credentials: true

Synthesized / auto-linked (do NOT put in the backend)

These appear in the materialized Secret but you do not populate them yourself:

• WHARF_POSTGRES_URL — when wharf.enabled: true. The chart synthesizes this at materialization time as postgres://<DB_USER>:<DB_PASSWORD>@<host>:<port>/<wharf_db_name>. DB_PASSWORD is pulled from your backend; everything else comes from chart values. Rotating DB_PASSWORD in your backend propagates here automatically on the next ESO refresh — no second key to keep in sync.
• CREWAI_OAUTH_API_KEY — when oauth.enabled: true. Sourced from the same backend property as OAUTH_INTERNAL_API_KEY (default oauth_internal_api_key at oauthSecretPath). The chart guarantees the two values are byte-identical, eliminating a silent OAuth-to-Rails 401 failure mode.

The chart’s values.schema.json rejects attempts to override autoGeneratedPropertyMap.CREWAI_OAUTH_API_KEY or autoGeneratedPropertyMap.WHARF_POSTGRES_URL to enforce this invariant.

OAuth secret (at externalSecret.oauthSecretPath)

Rendered as a second ExternalSecret targeting <release>-oauth-secrets when both oauth.enabled and manageAutoGenerated are true. Falls back to secretPath if oauthSecretPath is empty.

Property name	Maps to env var	Required
oauth_cookie_secret	OAUTH_COOKIE_SECRET	yes
oauth_db_encryption_key	OAUTH_DB_ENCRYPTION_KEY	yes
oauth_internal_api_key	OAUTH_INTERNAL_API_KEY (also feeds CREWAI_OAUTH_API_KEY)	yes

Provider client IDs/secrets (GOOGLE_*, MICROSOFT_*, HUBSPOT_*, NOTION_*, SALESFORCE_*) are not requested by default. Add them to oauthPropertyMap if you use those OAuth providers for end-user integrations:

externalSecret:
  oauthPropertyMap:
    GOOGLE_CLIENT_ID: google_client_id
    GOOGLE_CLIENT_SECRET: google_client_secret
    MICROSOFT_CLIENT_ID: microsoft_client_id
    MICROSOFT_CLIENT_SECRET: microsoft_client_secret
    MICROSOFT_TENANT_ID: microsoft_tenant_id

UV_DEFAULT_INDEX (the private PyPI registry URL) deserves a special callout. In KOTS deployments it auto-derives from the Replicated license; in direct-Helm/ArgoCD deployments it must be set explicitly. With ESO enabled, only the autoGeneratedPropertyMap route works — values placed under secrets.UV_DEFAULT_INDEX are ignored because the inline Secret template does not render. Put the URL in your backend at externalSecret.secretPath (property uv_default_index) and list it in the map.

The pre-upgrade wait Job

When manageAutoGenerated: true and waitForSync: true (the default), the chart renders a pre-upgrade Job (<release>-wait-for-eso-<revision>) that blocks Helm until ESO has populated the materialized Secret with every required key.

This sequences ESO synchronization ahead of every pre-upgrade hook that depends on the auto-generated keys — Rails db:migrate, wharf migrate, oauth migrate. Without it, those migrations would race against the ExternalSecret being created during the same phase, hitting couldn't find key X in Secret.

Helm hook weight ordering when manageAutoGenerated: true:

-50   SA / Role / RoleBinding (read-only on Secrets + ExternalSecrets)
-30   ExternalSecret + OAuth ExternalSecret           ← ESO begins reconciling
-20   wait-for-eso-sync Job                            ← blocks until Secret has all keys
 -5   pre-upgrade-migration (Rails db:migrate)         ← reads complete Secret
  0   wharf-migrate-job, oauth-migrate-job

The wait Job uses BusyBox + raw HTTPS calls to the in-cluster Kubernetes API. Default timeout: 3 minutes. Successful syncs converge in under 30 seconds; the tight timeout surfaces broken syncs quickly without making operators wait.

Troubleshooting

Wait Job times out

Secret/<release>-secrets still missing keys after 3 minutes: <KEY_LIST>

Followed by a dump of the live ExternalSecret YAML. The <KEY_LIST> is exactly the set of property names missing from your backend. Add them and re-run helm upgrade.

If the dump shows ESO errors (e.g. secretsmanager:GetSecretValue Forbidden), the issue is in your SecretStore configuration — IAM role trust policy, IRSA annotation on the ServiceAccount, or the role’s permission boundary. Fix at the platform level; ESO reconciles automatically.

Wait Job hard-fails on TLS / connectivity

FATAL: cannot reach Kubernetes API.
wget stderr was:
wget: SSL not available

Your busybox.image.tag was pulled as a minimal variant that ships without OpenSSL. Pin to a known-good version:

busybox:
  image:
    tag: "1.36"

Wait Job itself is broken (unrecoverable bug)

If the wait Job is producing unexpected failures and you need to unblock an upgrade immediately, two escape valves:

Chart-level disable (preferred)

Set in your values file:

externalSecret:
  waitForSync: false

The wait Job stops rendering. Migrations still run as pre-upgrade hooks; they may CrashLoop briefly until ESO catches up. After the upgrade settles, kubectl rollout restart deploy -n <ns> to refresh pod env caches.

Per-upgrade bypass

For a one-off upgrade:

helm upgrade <release> <chart> --values <file> -n <ns> --no-hooks

Skips ALL hooks (including the wait Job) for this upgrade only. Subsequent upgrades run the wait Job again.

Disabling the wait Job reverts to the pre-hook behavior where there is no ordering guarantee between ESO synchronization and pre-upgrade migrations. Use only as a temporary unblock while the wait Job is being patched.

”ExternalSecret already exists” on first upgrade to a chart with hook-based ESO

pre-upgrade hooks failed: warning: Hook pre-upgrade ...external-secret.yaml failed:
externalsecrets.external-secrets.io "<release>-external-secret" already exists

You’re upgrading from a chart version where the ExternalSecret was a regular manifest to one where it’s a hook. This is the one-time migration documented in Path 2. Orphan-delete the existing ExternalSecret(s), then re-run the upgrade. Running pods are unaffected because the materialized Secret is preserved by --cascade=orphan.

Materialized Secret has only WHARF_POSTGRES_URL, everything else is missing

This was a real bug fixed in chart version that introduced mergePolicy: Merge on the ExternalSecret template. If you’re seeing it on a current version, your custom ExternalSecret manifest may have been hand-modified — check that spec.target.template.mergePolicy is Merge, not Replace (the default).

OIDC_PRIVATE_KEY mangled in the Secret

The most common cause is uploading the PEM to your backend with \n literals instead of real newlines. Round-trip verify:

aws secretsmanager get-secret-value --secret-id crewai/platform/prod --region us-west-2 \
  --query SecretString --output text \
  | jq -r .oidc_private_key \
  | openssl rsa -check -noout
# Expected: RSA key ok

If openssl rejects the key, re-upload using jq --rawfile (which preserves newlines through JSON encoding) instead of any approach that involves shell string interpolation.

Secret has the key but it’s empty (0 bytes)

GITHUB_CREW_STUDIO_TOKEN:   0 bytes

ESO fetched an empty string from your backend. Either the property exists with an empty value, or your backend treats absent properties as empty strings. Pods consuming such a value will fail when the value is actually needed. Populate the property — even if your deployment doesn’t use the feature, an empty string is safer than failing at runtime.

Related

• ArgoCD Deployment — using ESO with GitOps
• Configuration Reference: External Secret — every value with type and default
• Cloud Provider Identity — setting up IRSA for the ESO SecretStore