When Both Postgres Replicas Vanished: Inside OpenAI's November 21 Authentication Outage

Stackademic

A maintenance window that killed both read replicas triggered a 75% ChatGPT failure rate and a platform-wide authentication cascade.

A Maintenance Window That Became a Platform Event

On November 21, 2026, OpenAI suffered one of its most visible infrastructure failures in the ChatGPT era: a planned PostgreSQL maintenance operation removed both read replicas from service for roughly three hours and thirty-three minutes, from approximately 1:48 a.m. to 5:21 a.m. Pacific Time. During that interval, OpenAI estimated that roughly seventy-five percent of ChatGPT requests failed or degraded severely. What began as a familiar story—database ops touching the wrong lever—became a case study in auth cascade: as replica health dropped, systems far from the storage layer began denying legitimate users, retry storms amplified load on the surviving primary, and feature flags meant for gradual traffic shifts turned into emergency brakes applied too late.

The incident is instructive because OpenAI’s stack is not unique. Thousands of companies run “boring” Postgres as the system of record for identities, entitlements, billing state, and audit logs, while flashy GPU fleets steal the architectural glamour. When Postgres blinks, the API gateway may still accept TLS handshakes, the model router may still schedule tokens, and yet nothing trustworthy can be said about who is calling. That is the auth cascade: authentication and authorization dependencies fail open, fail closed, or fail weirdly at the same time, and operators discover which path their code chose under stress only when customers cannot log in.

Timeline: Replicas Down, Platform Shaking

According to OpenAI’s preliminary incident report and corroborating third-party uptime monitors, engineers initiated maintenance on a Postgres cluster that backs several control-plane services. The work was scoped to read replicas—routine vacuuming, storage resharding, or version alignment, depending on which internal runbook fragment leaked to status-page readers. The critical mistake was sequencing: both replicas were taken offline in overlapping windows without a validated hot stand-in for read traffic that authentication services require even when “writes” theoretically continue on the primary.

Between 1:48 and 5:21 PT:

  • ChatGPT web and mobile clients reported elevated error rates, spinner loops on conversation load, and inconsistent session continuity.
  • Third-party API consumers saw elevated 5xx responses, with a disproportionate share tagged as auth or session errors rather than model capacity errors.
  • Internal dashboards later shown to enterprise customers indicated connection pool saturation on the primary, not merely “replica lag.”

Seventy-five percent failure rate is a platform-level number. It does not mean three quarters of all HTTP packets dropped; it means three quarters of user-visible actions—starting a chat, listing history, switching workspaces—could not complete. For a product positioned as daily workflow infrastructure, that is catastrophic even in off-peak hours. Global user bases do not sleep in Pacific Time.

Primary API and the Auth Cascade Mechanism

OpenAI’s Primary API—the developer-facing surface distinct from consumer ChatGPT—exhibited a related but not identical failure signature. Many requests failed at layers that validate API keys, organization membership, usage tier, and safety policy flags stored relationally. When read replicas vanished, services that normally served read-heavy auth checks from replicas fell back to the primary or to stale caches with inconsistent TTL behavior.

Cascade patterns observed in similar outages elsewhere, and described in OpenAI’s technical retrospectives shared under NDA with large customers, include:

  1. Retry amplification: API clients and internal microservices retried auth lookups on timeout, multiplying QPS on the already wounded primary.
  2. Connection churn: Pool sizing tuned for steady state did not absorb a sudden shift of all read traffic plus retries; connections were opened and discarded aggressively, worsening Postgres max_connections pressure.
  3. Negative caching gaps: Some services cached “user not found” on transient DB errors, briefly locking out valid keys until cache expiry—a classic auth cascade footgun.
  4. Dependent service fan-out: A single login flow might touch org service, billing service, compliance flags, and feature store; any one relational dependency timing out failed the whole chain.

The auth cascade explains why model inference capacity could remain partially available while the product felt “down.” Tokens are expensive; denying them without payment and policy checks is riskier than denying chat UI loads. Enterprise customers with dedicated capacity still reported failures when key validation could not complete—proof that GPU headroom does not matter if the control plane cannot attest entitlement.

Connection Churn Tuning: The Hidden Fire

Database operators often focus on query plans and disk I/O. Incidents like November 21 redirect attention to connection lifecycle: how application servers acquire, hold, and release Postgres connections under spike conditions. OpenAI’s post-incident notes emphasized tuning performed in the days after the outage—lowering idle timeouts cautiously, adjusting pool maximums per service, separating pools for auth reads versus billing writes, and introducing circuit breakers on microservices that had previously treated DB timeouts as infinitely retryable.

Connection churn is particularly vicious with ORMs and serverless-style burst scaling. A replica outage does not just add read load to the primary; it changes latency distributions, causing pools to think connections are bad, discard them, and open new ones—each new connection a expensive handshake on Postgres. When dozens of services do this together, the primary spends CPU accepting TLS and auth instead of executing queries.

For readers building their own stacks, the lesson is not “buy a bigger primary.” It is isolate blast radius with role-specific pools, cap retry budgets on auth paths, and load-test failover with both replicas gone—not only with one replica lagging. Many disaster recovery drills stop at “promote replica”; fewer ask whether the application tier can survive zero read replicas for an hour.

Traffic Shift Feature Flag: Late but Necessary

OpenAI disclosed that a traffic shift feature flag—infrastructure intended to drain user-facing requests from unhealthy cells or regions—was engaged during the incident, but not early enough to prevent the worst customer impact. Feature flags for traffic management are seductive: they promise graceful degradation without redeploys. In practice they require pre-declared dependency graphs and automated health signals that Postgres maintenance tools do not emit by default.

During November 21, engineers manually verified replica state, debated whether to abort maintenance, and only then pulled traffic shifts that reduced load on relational-dependent paths. In hindsight, OpenAI stated that automated triggers tied to replica quorum loss should have initiated a partial shift within minutes, prioritizing read-only consumer experiences that could degrade to cached responses while protecting the primary for writes and auth.

For platform teams, this validates a design pattern: decouple “read your last conversation” from “prove who you are.” The former can come from edge caches with explicit staleness banners; the latter should not guess. OpenAI’s consumer product complexity makes that decoupling hard—but not impossible.

Moratorium on Similar Maintenance

OpenAI announced a moratorium on similar maintenance until procedural and technical guardrails ship. Translation: no more overlapping replica maintenance without proven read-path redundancy; mandatory game-day exercises; executive sign-off for changes touching auth clusters during global peak windows (with peak redefined to follow UTC curves, not San Francisco nightlife).

Moratoriums calm customers but accumulate debt. Index bloat, replication version skew, and certificate rotations still need doing. The sustainable outcome is not perpetual freeze but maintenance windows with mathematically enforced quorum: at least one replica must accept read traffic verified by synthetic auth transactions every N seconds while work proceeds on another node.

Customers should expect written maintenance policies from vendors after this incident. If your vendor cannot explain replica quorum rules in one page, assume you will be the status-page audience next time.

ChatGPT at Seventy-Five Percent Failure: What Users Saw

From a product perspective, the outage bruised trust differently than model quality glitches. Users forgive occasional hallucinations; they do not forgive unable to load conversation during a launch week. Support forums documented:

  • Mobile apps logging users out repeatedly
  • Team workspaces showing empty history until recovery
  • Plugins and enterprise SSO flows failing while personal accounts partially worked—symptoms of sharded auth paths not failing uniformly

Seventy-five percent failure also means twenty-five percent success, which creates painful split-brain perception: one engineer’s demo works in a meeting while another’s fails—exactly the pattern that makes incident communication harder.

Lessons for Builders Outside OpenAI

You do not need OpenAI scale to replicate this failure mode. If Postgres holds API keys, OAuth states, or organization roles, you are vulnerable to auth cascade.

Actionable takeaways:

  • Never take all read replicas offline without a documented, tested fallback—including serving stale reads from a read-through cache with explicit expiry semantics.
  • Put retry budgets on auth middleware; exponential backoff must not multiply QPS during DB incidents.
  • Instrument auth latency percentiles separately from inference latency; alert on them independently.
  • Run game days that kill replicas, not just primaries.
  • Treat traffic-shift flags as production-critical: test them weekly, not only during incidents.
  • Write customer-facing status messages that distinguish “models unavailable” from “cannot verify account”—users troubleshoot differently.

Closing: Control Plane Humility

November 21 was a reminder that AI products are still internet services. The magical model is a subroutine; the control plane decides who may call it, how often, and with what data handling. Postgres maintenance is unglamorous until it isn't. OpenAI’s moratorium, connection churn tuning, and improved traffic-shift automation are the right vocabulary for recovery. The deeper obligation—for OpenAI and for every team shipping agents—is to design auth and data paths that fail narrowly even when databases misbehave.

Until then, read replica maintenance windows deserve the same fear and respect as kernel upgrades on the inference fleet. The auth cascade is silent until it isn't—and when it roars, three hours is an eternity.