Prod.keys Switch — __exclusive__

For ten minutes, everything worked. Then the on-call phone buzzed.

{ "environment": "dev", "api_keys": { "ai_provider": "dev_sk_test_123", "payment_gateway": "dev_pk_test_456" } } She knew the routine. Never, ever commit prod keys to code. Instead, the system used a —an environment variable called PROD_KEYS_ENABLED . When set to false , the app used dev keys. When set to true , it reached into a locked, encrypted vault and loaded the real production keys. prod.keys switch

deploy --service=wishlist --prod-keys-switch=false The switch flipped to OFF . The app instantly fell back to using dev_sk_test_123 —the fake key. The AI calls failed gracefully, and Wishlist displayed a polite message: "Gift ideas temporarily unavailable. Shop our curated collections!" For ten minutes, everything worked

Maya updated the vault with the new toy brand’s key. Then she ran the deployment script: Never, ever commit prod keys to code

She needed to roll back, but there was a problem: if she simply reverted the code, the prod.keys switch would still be true . The old AI provider’s prod key was already deleted from the vault. Without it, the app would crash entirely.

Three days before Black Friday, the CEO announced a last-minute partnership with a major toy brand. The integration required swapping the AI provider’s key for a new one—immediately.

This was exactly why the existed separately from the code rollback.