A few weeks into rebuilding part of our learning platform's backend, I got asked for a feature that sounded almost boring: let an admin upload a CSV and create a few hundred learner accounts in one go, instead of adding them one at a time through a form.

It sounded like a half-day task. It took a lot longer than that — not because the CSV parsing was hard, but because of two bugs that only showed up once I started creating learners in bulk and in a scope I hadn't tested before. Both taught me more about distributed systems than anything else I'd built that quarter.

The feature:

Admins go through a short wizard: upload a CSV, map its columns to our fields (name, email, phone, temporary password), see a validation summary (how many rows are valid, how many already exist, how many are broken), pick where these learners should land, and hit import. A progress bar ticks up as learners get created.

Behind that wizard sit two API calls — one to validate the rows, one to actually create them — and three services talking to each other: an orchestrator that owns the request, a service that actually creates the user record, and a separate service that grants that user permission to act inside a given workspace. Nothing exotic. Which is exactly why the bug was so easy to miss.

The "easy" part:

The CSV parsing itself lives entirely on the frontend — it reads the file, lets the admin map columns, and sends me clean JSON rows. My side never touches a .csv file directly, which kept the backend simple: no upload handling, no file storage, just an array of objects to validate and process.

Validation checks the obvious things — is the email actually an email, is the password present, is a row a duplicate of another row in the same file — plus one cross-service check: does this email already exist for this tenant. Rows come back labeled valid, invalid, or already-existing, and only the valid ones move on to actual creation.

This part worked the first time I tested it. Confidence was high. That confidence did not survive contact with bulk creation.

Bug #1: the user that almost existed

Creating one learner isn't a single database write — it's a small sequence: create the user record, hash their password, set up their default access, add them to any groups, and then ask the permission service to actually grant them a role in the workspace.

The first version of this had no transaction wrapped around any of it. The user row got saved to the database immediately, and only afterward did it ask the permission service to grant access. Which is fine, right up until that permission call fails for any reason — and across a few hundred rows, "any reason" happens more than you'd think.

When it failed, I'd end up with a user that existed in the database but had no actual access to anything. Functionally invisible, but not actually gone. Worse: if I tried to create that same learner again, the system saw an existing email and rejected it as a duplicate — which meant there was no way back in through the normal flow. The user was stuck half-built, forever, unless someone went and fixed it by hand.

I found this purely by testing — running a batch of imports, then noticing some learners just... didn't work, with no error pointing at why.

The fix:

Two changes, working together:

• Wrap the local writes in one transaction. The user record, their settings, and their group memberships now commit together or not at all. No more half-saved user.

• Make the permission grant idempotent and retryable. It happens after that transaction commits, and asking for the same role twice is harmless — it just confirms it's already there. If the grant fails, the row stays in a safe "pending" state instead of being deleted, and the same row can simply be retried later. Re-submitting a failed row either does nothing (if it's already fixed) or finishes the job (if it isn't). Nothing stays broken in between.

This is the kind of bug that's invisible at small scale and inevitable at large scale — a single test learner almost never hits a transient failure mid-creation. Three hundred of them, created back to back, will.

Bug #2: the scope nobody had configured

With the orphan-user issue fixed, I moved on to testing across different workspaces — and ran into something that looked, at first, like the exact same bug coming back. Every learner in one particular scope failed at the exact same step: the permission grant.

Except this time retrying didn't help. It failed the same way, every time, for every learner, in that one scope only. That consistency was the actual clue — a flaky network call fails sometimes. This was failing always, which meant it wasn't transient at all.

It turned out that scope simply hadn't had the right permission policies assigned to it. Every other workspace I'd tested against had been set up correctly from the start, so the gap had never surfaced. The moment I pointed bulk import at a scope that was missing that setup, every single learner in the batch failed at the same step — not because the import logic was wrong, but because the workspace itself wasn't ready to receive learners at all.

It's a good reminder that "it works in every environment I've tested" and "it works" are not the same sentence.

Why chunking matters more than it sounds like it should

Once creation was reliable, the next question was speed. Creating one learner is genuinely slow — password hashing is deliberately expensive, plus there are a couple of network calls per user. Do that one at a time for a thousand rows and you're looking at minutes.

The fix isn't bigger batches — it's concurrency. Each import call processes a bounded number of rows at the same time (in our case, twenty in flight at once) instead of one after another or all-at-once. Twenty overlapping requests that each take a fraction of a second finish in roughly the time of one, and capping it at twenty keeps from overwhelming the downstream services or the database connection pool.

This led to a distinction that wasn't obvious to me at first: how many rows the frontend sends per API call and how fast the backend processes them are two completely separate levers. Chunk size controls how smoothly the progress bar moves — smaller chunks mean more, smaller jumps; bigger chunks mean fewer, bigger jumps. Speed comes entirely from how much concurrency the backend allows inside each call. You could send the whole file in one request and it would still create everything at the same speed; you'd just get one big jump on the progress bar at the end instead of a smooth crawl.

What this actually taught me

Single-record features hide assumptions that bulk features expose. Creating one learner by hand, I'd never have noticed the missing transaction — a single failure is rare enough to feel like bad luck. I'd never have noticed the missing scope policy either, because I kept testing in the same well-configured workspace.

Bulk operations are a stress test for assumptions you didn't know you were making. The fix wasn't really "handle CSVs" — it was "make user creation safe to retry, and make failures specific enough that you can tell which assumption broke."

If I were starting this feature again, I'd write the idempotency and transaction handling first, before the happy path — and I'd deliberately test against a half-configured environment on day one, instead of discovering by accident that "works" had been quietly meaning "works in the one place I keep testing."