Most system design content treats multi-tenancy as a database question. Shared schema or separate schema, pick one, move on.

That question is maybe 10% of the problem.

The other 90% is this: a salesperson clicks a button, and forty seconds later a customer needs a working URL, a valid TLS certificate, an isolated database with reference data in it, an admin account that can log in, a permission model, and a branded login page. Any one of those steps can fail. Several of them touch systems you don't control. None of them can be rolled back with a ROLLBACK.

This post is the architecture of that button. I've written it as the thing I wish I'd read before my first system design interview on multi-tenancy — concrete mechanics, the tradeoffs behind each choice, and the bugs that only show up in production.

Service names here are generic stand-ins. The patterns are the point.

The shape of the system

Two planes. Getting this separation right determines how much pain everything downstream causes.

┌──────────────────────── CONTROL PLANE ────────────────────────┐
│                                                               │
│   Operator Console ──▶ Tenant Control Service                 │
│                              │                                │
│                              ├──▶ control database            │
│                              │    (tenants, drafts, runs)     │
│                              │                                │
│                              └──▶ Provisioning Orchestrator   │
│                                          │                    │
└──────────────────────────────────────────┼────────────────────┘
                                           │
┌──────────────────────── DATA PLANE ──────┼────────────────────┐
│                                          ▼                    │
│   Schema Migrator ──▶ tenant schemas                          │
│   Directory Service ──▶ users, roles, permissions             │
│   Catalog Service ──▶ offerings, plans, features              │
│   Asset Service ──▶ logos, documents, media                   │
│   Edge Router ──▶ subdomains, TLS, request routing            │
│                                                               │
└───────────────────────────────────────────────────────────────┘

Control plane manages tenants. It's low-traffic, high-consequence, and internal-only. A handful of operators use it.

Data plane serves tenants. It's high-traffic, and every request is scoped to exactly one tenant.

The rule worth internalising: the control plane knows about all tenants; the data plane knows about one at a time. Every request in the data plane carries a tenant identity, resolved once at the edge, and services below that never think about "which tenant" again — they're handed the answer.

When you blur this line, you get the classic multi-tenant bug: a query that forgets its tenant filter and returns another customer's data. The architecture should make that hard to write, not rely on remembering.

Choosing an isolation model

This is the question interviews open with. Three options, and the honest answer is that it depends on your buyer.

                          Shared schema      Schema per tenant    Database per tenant
  ──────────────────────────────────────────────────────────────────────────────────
  Isolation               row-level          namespace-level      physical
  Cost per tenant         lowest             low                  high
  Noisy neighbour         real risk          partly contained     contained
  Per-tenant restore      very hard          straightforward      trivial
  Schema migrations       one                N, orchestrated      N, orchestrated
  Scale ceiling           millions           thousands            hundreds
  Passes security review  sometimes          usually              always

We chose schema per tenant, in a shared database instance.

The reasoning is worth stating plainly because it's the kind of answer that lands well: our customers are enterprises. Their procurement teams ask "is our data physically separated from your other customers?" — and "separated by a WHERE clause" is an answer that loses deals. Schema per tenant is defensible in a security questionnaire. It also gives you per-tenant restore almost free, which matters the first time a customer deletes something important and asks for it back.

The cost is real and you should name it: migrations become an orchestration problem. One schema change means N schema changes, applied with partial failure handling, version tracking per tenant, and the ability to have tenants temporarily on different versions. You need tooling for this on day one, not day three hundred.

The ceiling is real too. Postgres gets unhappy somewhere in the low thousands of schemas — catalog bloat, connection pooling pressure, pg_dump times. If your model is self-serve with a hundred thousand tenants, this is the wrong choice and shared-schema-with-RLS is right. Know which business you're in.

Stage 1: The wizard is server-side state

The console is a multi-step wizard — company details, subscription plan, branding, admin user, review.

The instinct is to hold it in client state and POST once at the end. Don't.

Three reasons, and they generalise well beyond onboarding:

1. Onboarding a real enterprise takes days, not minutes. Someone starts it, needs a tax ID they don't have, closes the laptop. That work must survive.

2. Uniqueness must be reserved, not just checked. A subdomain is a global resource. Checking availability at step 2 and claiming it at step 5 is a race — two operators can both see "available".

3. Validation belongs where it's enforced. If validation lives only in the client, the API is unprotected, and every rule now exists in two places that will drift.

So: creating a draft is the first call, not the last. It allocates a tenant identifier and creates a DRAFT row immediately. Each wizard step is a partial update against that draft. The GET returns both the saved state and the highest step reached, so resume lands the user where they left off rather than at step one.

The draft is genuinely partial — a half-filled draft is a valid state. Completeness gets checked once, at submit. This is a different validation posture than a normal API and it's worth saying out loud in an interview: "the draft validates field shape; the submit validates completeness." Two different rulesets over the same data.

Stage 2: Provisioning is a durable step machine

Submit does not provision. Submit enqueues.

This is the single most important design decision in the whole system, so let's be precise about why.

Provisioning takes 30–90 seconds on a good day. It calls five services and two cloud APIs. Certificate issuance alone can take minutes to hours and is entirely outside your control. If you do that work inside the HTTP request:

• the request times out at whatever your gateway's limit is

• the operator sees a spinner, then an error, for work that actually succeeded

• a retry re-runs everything from the top

• a deploy mid-provision kills the run with no record of where it stopped

Instead, submit writes a provisioning run row with an ordered list of steps, publishes a message, and returns 202 immediately. The console polls a status endpoint.

  run_id: 4f2a...
  tenant: acme-corp
  status: RUNNING
 
  seq  step                  status      attempts
  ─────────────────────────────────────────────────
   1   CREATE_SCHEMA         SUCCEEDED   1
   2   SEED_REFERENCE_DATA   SUCCEEDED   1
   3   CREATE_ADMIN          SUCCEEDED   2
   4   INIT_DEFAULTS         RUNNING     1
   5   CONFIGURE_ROUTING     PENDING     0
   6   SEND_WELCOME          PENDING     0

The run table is the state machine. It's not a log of what happened — it's the authoritative record of what has and hasn't been done. That distinction matters: a worker picking up the run reads this table to decide what to do next. Restarting a failed run means resuming at the first non-succeeded step, not starting over.

This is the saga pattern, and naming it in an interview is worth doing. You're replacing a transaction you cannot have — no distributed transaction spans your database, a DNS provider, and a certificate authority — with an explicit sequence of durably-recorded steps.

Ordering is a real constraint

Steps aren't arbitrary. The dependency graph is load-bearing:

• Storage buckets before anything writes files to them

• Schema before seeding data into it

• Reference data before creating the admin (the admin needs a role; roles come from the seed)

• Everything before the welcome email — because that email contains a login link, and sending it before login works is worse than sending it late

That last one is a good interview point. Notification is always the final step, and it should be, because a notification is the one action you cannot compensate. You can delete a schema. You cannot un-send an email to a customer's CEO telling them their platform is ready when it isn't.

Idempotency, or: the run that executed twice

Here's a bug that will find you.

Two workers pick up the same message. Both begin the run. Both reach CREATE_ADMIN. One creates the admin; the other fails with "user already exists". The run is marked FAILED. The tenant is, in fact, perfectly fine.

Every layer you'd hope prevents this doesn't:

• Message queues are at-least-once. Exactly-once delivery is marketing. Duplicates are normal operation, not an error.

• Long steps outlive visibility timeouts. A step that takes longer than the ack deadline gets redelivered while still running.

• Retries stack. The queue retries; your code retries; now you have four.

The fixes, in increasing order of strength:

1. Make every step idempotent. CREATE SCHEMA IF NOT EXISTS. Look up the admin before creating. Upsert instead of insert. This is the baseline and it is not optional.

2. Guard the transition, not the operation. Claim the step with a conditional update — set it to RUNNING only where it is still PENDING — and proceed only if you actually changed a row. The database becomes the lock.

3. Deterministic task identity. If you schedule continuation work, derive its ID from the run, the step, and the attempt. Schedulers that reject duplicate IDs will then reject your duplicate for you.

There's a subtle trap in (3) worth knowing, because it bit us. Deterministic task names are great — but many schedulers reserve a completed task's name for a while afterwards. If your polling loop schedules the same name every time it checks, the second poll is silently deduplicated against the first, and the run stalls forever with no error anywhere. The fix is to include the attempt counter in the name so each poll is genuinely distinct.

That class of bug — the silent no-op — is the hardest to find in distributed systems, because nothing fails. Design for it: if a scheduled continuation doesn't fire, something must eventually notice. A watchdog that fails runs stuck in RUNNING past a deadline is worth its weight.

Seeding reference data: two bugs you will hit

A new tenant needs baseline data before it's usable — roles, permission definitions, default configuration, category taxonomies.

The naive approach is dumping this from a reference tenant and replaying it. Two things break.

Foreign keys demand ordering. A dump ordered alphabetically will insert a child row before its parent and fail. Self-referencing tables — a role hierarchy where a role points at a parent role — will fail even when the table order is right. You need a topological sort of the dependency graph, and within self-referencing tables, an ordering by depth.

Sequences don't come along for free. You insert 500 rows with explicit IDs 1–500. The table's sequence still sits at 1. The tenant's first real insert collides on the primary key. Every seed must reset its sequences to just past the highest inserted value.

Both bugs share a shape: they don't surface during seeding. They surface later, in the tenant's first real use, as a confusing constraint violation with no obvious connection to onboarding. When you're asked "how do you test provisioning?" — this is the answer. Provision a tenant, then use it. Create a record. The seed isn't verified until something writes on top of it.

The better long-term design is declarative seeds under version control, applied by the same migration tooling that handles schema changes — not a dump. Same ordering guarantees, same versioning, reviewable in a pull request.

Identity bootstrap and the credential problem

The tenant needs a first administrator, and this is where a lot of systems quietly do something bad.

The chain is: create the user record → assign a role → grant that role its permissions → get the human logged in.

The interesting part is the last step. You need to give someone access to an account that doesn't have credentials yet.

Don't generate a temporary password and store it. A password sitting in a database column, a run record, an audit log, and an email is four places it can leak, and it will outlive its usefulness in all four.

Use a single-use, time-bounded activation token:

• Store only a hash of it, never the token

• Bind it to one user and one purpose

• Expire it in hours

• Invalidate it on first use

• Make the user choose the password, so it's never transmitted or stored by you at all

This generalises to every "send someone a link" flow — password reset, email verification, invitations. Same primitive. If an interviewer asks how you'd handle first-login for a provisioned account and you describe a token with those five properties, that's a strong answer.

While we're here: role and permission definitions are reference data, not code. They're seeded per tenant, which means a tenant can be provisioned with a role that has no permissions attached if the seed is incomplete. That fails at the tenant's first login, not at provisioning. Verify the grants as part of the run, not just the inserts.

Networking: subdomain, certificate, route

Each tenant gets customer.yourplatform.com. Three distinct things must happen, and they're often conflated.

1. DNS. A record pointing the subdomain at your edge. Fast — seconds to write, though propagation and caching mean it isn't instantly visible everywhere.

2. TLS certificate. The certificate authority must verify you control the domain before issuing. This is the slow, unpredictable step: usually minutes, occasionally much longer, and it depends on step 1 having propagated. You cannot make this fast. You can only design around it.

3. Routing. Your edge needs a rule mapping that hostname to the right backend. Only after this does the URL actually serve anything.

Two design consequences worth carrying into an interview.

Model certificate issuance as a polled asynchronous step, not a call. The run parks on it and checks periodically with a backoff, with a generous deadline. Anything that blocks a worker for an unbounded external process is a design error.

Tell the customer the truth about timing. If the welcome email says "your platform is ready" and the certificate hasn't issued, the customer clicks, gets a browser security warning, and their first experience of your product is a scary red page. Either gate the email on the certificate being live, or say "provisioning completes within ~15 minutes" in the email itself. This is a rare case where the right fix is copywriting, not code.

The shared-mutable-config problem

Edge routing configuration is usually one object shared by every tenant — a single routing table you append a rule to.

Two provisioning runs finishing at the same moment both read it, both append their rule, both write. The second write silently erases the first. One tenant is live; the other has DNS, a certificate, and no route, which is the worst kind of failure — everything looks provisioned and nothing works.

The fix is optimistic concurrency: read the config with its version identifier, send that version with your write, and let the server reject the write if it's stale. On rejection, re-read and re-apply. Retry only on the version conflict — retrying a genuine validation error just burns your quota.

This pattern — read-modify-write against shared state with a version check — comes up constantly. Recognise it whenever provisioning has to mutate something global rather than create something new. It's the same shape as an ETag, a compare-and-swap, or an optimistic lock on a row.

Assets: the two-phase storage problem

Branding assets get uploaded during the wizard — logo, favicon.

The ordering problem: the operator uploads a logo at step 3, but the tenant's storage doesn't exist until provisioning runs. You cannot write to a bucket that isn't there.

Two-phase placement:

  during the wizard          after provisioning
  ─────────────────          ──────────────────
  staging area,       ──▶    tenant's own storage,
  keyed by draft             keyed by tenant

Uploads land in a staging area owned by the control plane, under a deterministic path derived from the draft. Provisioning creates the tenant's storage, then copies the assets across as an explicit step.

Three things fall out of this that are worth knowing:

The path changes. The URL the wizard previews is not the URL the tenant will use. Anything that persisted the staging URL now points at the wrong place. Store the identifier, resolve the URL at read time — never persist a rendered URL.

Deterministic paths make re-upload safe. If the object name is derived from the draft and the asset kind, uploading twice overwrites, which is exactly what you want — last write wins, no orphans, no cleanup job. The exception is a re-upload with a different extension, which leaves the old object behind. Delete the previous path when the extension changes.

Private storage needs signed URLs. Staged assets belong to a customer who hasn't signed a contract yet. That bucket should be private, and previews served through short-lived pre-signed URLs — minutes, not hours.

That last point has a consequence the frontend team must know about: a pre-signed URL is not a stable link. Cache one in client state, come back to the draft an hour later, and it's expired. Either fetch fresh URLs when the page loads, or expose an endpoint that re-signs. Deciding this before the UI is built saves an unpleasant conversation later.

When steps fail

Not all failures are equal, and treating them identically is a design smell.

Transient — timeout, connection reset, rate limit, a dependency mid-deploy. Retry with exponential backoff and jitter. The jitter matters: without it, everything that failed during an outage retries in perfect lockstep the moment recovery starts, and you take the dependency down again.

Permanent — invalid input, a missing plan, a name collision. Retrying is pure waste. Fail the step, record the reason, stop.

Classifying failures explicitly at the point where you raise them — a flag on the error saying "this is worth retrying" — is a small thing that pays for itself immediately.

Partial failure is the normal state, and this is the part that separates a good answer from a great one. Step 4 of 6 fails. You now have a schema, seeded data, and an admin user, and no routing. What do you do?

• Roll everything back? Compensating actions are themselves fallible. Your rollback can fail halfway, leaving a state neither forward nor back. And rollback destroys the evidence you need to debug the original failure.

• Leave it and retry forward? The tenant sits in a broken intermediate state, possibly for hours, but every completed step is real and re-runnable.

We chose retry forward, and I'd defend that in any interview: because every step is idempotent, resuming is safe, and because the alternative — automated rollback of partially-completed distributed work — is substantially harder to get right than the thing it's protecting you from. The tenant stays in a PROVISIONING_FAILED state, visible to operators, resumable with one click.

The exception is anything with external side effects that cost money or reach a human. Those get compensated deliberately, not retried blindly.

Making it observable

A run that fails at 3am must be diagnosable at 9am without a debugger.

• A correlation ID generated at submit and propagated through every downstream call, into every log line. One ID reconstructs the whole distributed trace.

• Per-step attempt counts and error messages on the run record itself. First question is always "which step, and what did it say" — that should be one query, not a log search.

• Duration per step, so you can see the certificate step drifting from 4 minutes to 40 before customers tell you.

• An alert on stuck runs, not just failed ones. A run RUNNING for an hour is a worse signal than a run that failed, because nothing is going to page you about it. Silence is not success.

That last one again: the failure modes that hurt are the silent ones. Build something that notices absence.

The interview version

If you get "design tenant onboarding for a B2B SaaS", here's the skeleton that covers it.

Open by separating control plane from data plane. It frames everything else and signals you've thought about this before.

State the isolation model as a tradeoff, not a preference. "Schema per tenant, because our buyers are enterprises with security reviews, and per-tenant restore is nearly free. The cost is that migrations become an orchestrated N-schema operation, and it doesn't scale past a few thousand tenants — if this were self-serve with a hundred thousand tenants I'd use shared schema with row-level security instead."

Make onboarding asynchronous immediately. Submit returns 202. Explain that certificate issuance alone rules out doing this in a request.

Name the saga. An ordered, durably-recorded step machine replacing a distributed transaction you can't have.

Bring up idempotency before you're asked. At-least-once delivery is the default. Every step must be safe to run twice.

Have an opinion on partial failure. Retry forward, because steps are idempotent and automated rollback is harder to get right than the problem it solves.

Mention the shared-config race. Optimistic concurrency on the routing table. It shows you've thought past the happy path of a single tenant.

Close on observability. Correlation IDs, per-step state, and alerting on stuck runs — not just failed ones.

The one idea worth keeping

Provisioning a tenant is a distributed transaction that you are not allowed to have.

Every technique here — the durable run record, idempotent steps, optimistic concurrency, retry-forward, deterministic paths, signed URLs, the welcome email going last — is a workaround for that single fact. You cannot make eight systems commit atomically, so you make each step individually safe to repeat, record precisely how far you got, and design every failure to be resumable rather than reversible.

That reframe is worth more than any individual pattern. Once you see a workflow as "a transaction I can't have", the right architecture tends to follow.