Handling Bulk API Calls Without Hitting Limits

The first Bulk API job I ran that genuinely mattered fell over at around 2am. We were migrating a bit over 8.3 million Contacts into an APRA-regulated insurer, I'd handed the load to Data Loader with a batch size of 200, and by the time I opened the results file it was a wall of UNABLE_TO_LOCK_ROW. Somewhere north of four per cent of the records had failed, and nearly every one of them pointed back at the same handful of parent Accounts.

That night taught me more about Bulk API than any documentation ever did. The limits that bite in production are almost never the ones in the "Bulk API Limits" table. They're row locks, parallel batches fighting each other over the same parent record, and automation you forgot was still switched on.

Bulk API 2.0 Is the Default Now, Not the Exception

I reach for Bulk API 2.0 on almost every new integration, and I only fall back to 1.0 when I have a specific reason. The 2.0 model is simpler in the ways that matter: you create one job, upload your CSV in a single request, tell Salesforce you're done, and it handles the batching for you. No manually splitting a file into 10,000-record chunks and tracking each batch ID yourself.

The job lifecycle is three calls. Create the job, upload the data, close it.

# 1. Create the ingest job
curl -s -X POST "$INSTANCE/services/data/v58.0/jobs/ingest" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "object": "Contact",
        "operation": "insert",
        "contentType": "CSV",
        "lineEnding": "LF"
      }'
# response includes an id and a contentUrl

# 2. Upload the CSV to the job
curl -s -X PUT "$INSTANCE/services/data/v58.0/jobs/ingest/$JOB_ID/batches" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: text/csv" \
  --data-binary @contacts.csv

# 3. Tell Salesforce the upload is complete
curl -s -X PATCH "$INSTANCE/services/data/v58.0/jobs/ingest/$JOB_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"state": "UploadComplete"}'

The one thing worth calling out here is lineEnding. If your extract came off a Windows box it's probably CRLF, and if you tell Salesforce LF when it's actually CRLF, every row fails to parse with an error that looks nothing like a line-ending problem. I've wasted an hour on that more than once.

Where I still choose 1.0 is bulk queries against very large objects, where PK chunking lets you pull tens of millions of rows without timing out. For ingest, though, 2.0 is where I start.

The Limit That Actually Bites Is the Row Lock

Here's the thing nobody warns you about when you first read the docs. Bulk API 2.0 processes its internal batches in parallel by default, and there's no serial mode switch the way there was in 1.0. Parallelism is great for throughput and terrible for related records.

When you insert a Contact, Salesforce briefly locks the parent Account to maintain the relationship. If two batches running in parallel both contain Contacts that roll up to the same Account, one of them wins the lock and the other throws UNABLE_TO_LOCK_ROW. Multiply that across millions of records where a few large Accounts own hundreds of thousands of Contacts each, and you get exactly the 2am results file I described above.

This is not a bug and it's not a quota you can raise. It's a contention problem, and you fix it by controlling how the data is organised before it ever reaches the API — the same principle that governs how you structure queries so you don't collide with governor limits, which I've written about separately in my SOQL query patterns piece.

Sort by Parent Before You Load Anything

The fix for the insurer migration was almost anticlimactic. I sorted the entire Contact extract by AccountId before loading.

When related records are grouped together in the file, Salesforce's automatic chunking is far more likely to keep all the Contacts for a given Account inside the same internal batch. Same batch means same thread, which means the parent lock is acquired once and held cleanly rather than being fought over across parallel workers.

-- The extract query, ordered so related children stay together
SELECT Id, FirstName, LastName, AccountId, Email
FROM Contact_Staging__c
ORDER BY AccountId NULLS LAST

That single ORDER BY took the failure rate from somewhere north of four per cent down to a few dozen records across the whole 8.3 million — genuine data problems, not lock contention. The load window went from an ugly eleven hours of retries to just under three clean ones. Ravi, the data lead on that project, still brings it up as the cheapest fix he's ever seen for a problem that looked catastrophic at midnight.

If you can't sort at the source, sort the CSV before upload. Order of records in the file is the single biggest lever you have over lock contention, and it costs nothing.

Turn Off the Automation You Forgot Was On

A Bulk API load doesn't happen in a vacuum. Every record you insert fires triggers, runs record-triggered Flows, evaluates assignment rules, and kicks off sharing recalculation. During a bulk migration, most of that automation is either irrelevant or actively harmful — you're loading historical data that doesn't need a welcome email or a round-robin owner assignment.

I disable everything I safely can for the duration of the load and turn it back on afterwards. For Apex, this is exactly why my trigger handlers read their on/off state from custom metadata rather than hardcoding behaviour — I covered the full pattern in the trigger framework I use on every engagement. The bypass check lives in the base class:

public abstract class TriggerHandler {
    protected Boolean isDisabled() {
        Trigger_Setting__mdt setting =
            Trigger_Setting__mdt.getInstance(handlerName);
        return setting != null && setting.Disable_Trigger__c;
    }
}

An admin flips the Disable_Trigger__c boolean, deploys the metadata record, the migration runs with the trigger dormant, and then they flip it back — all version-controlled and fully auditable, with no Apex deployment in the middle.

For the rest, the checklist I run before a large load is short but non-negotiable. Deactivate the record-triggered Flows that don't apply to historical data. Uncheck "Send email" on Lead and Case assignment rules, or deactivate the rule entirely. And for orgs with heavy sharing, enable Defer Sharing Calculation in Setup so the org isn't recalculating the sharing table on every single insert — you run one recalculation at the end instead of millions along the way.

I'll be honest about the risk here: turning off automation during a load is one of the more dangerous things you can do to a production org, because the org that comes back up is only as good as your memory of what you switched off. I keep a written toggle list every time, tick each item back on, and have someone else verify it. The one occasion a Flow stayed dormant for three days after a load, nobody noticed until the sales team asked why their new Leads had stopped routing.

Poll the Job, Don't Babysit It

Once the job is closed, resist the urge to sit and refresh. Bulk API is asynchronous by design, and the right pattern is to poll the job resource on an interval until it reaches a terminal state.

JOB_ID=750XXXXXXXXXXXXXXX
while true; do
  STATE=$(curl -s -H "Authorization: Bearer $TOKEN" \
    "$INSTANCE/services/data/v58.0/jobs/ingest/$JOB_ID" \
    | jq -r '.state')
  echo "$(date '+%H:%M:%S') state: $STATE"
  case "$STATE" in
    JobComplete|Failed|Aborted) break ;;
  esac
  sleep 30
done

# Pull the records that failed, with the error against each row
curl -s -H "Authorization: Bearer $TOKEN" \
  "$INSTANCE/services/data/v58.0/jobs/ingest/$JOB_ID/failedResults" \
  -o failed.csv

JobComplete does not mean every record succeeded — it means Salesforce finished processing the job. The failed results endpoint returns each rejected row with a sf__Error column, and that file is where the real work is. On a clean, sorted load it's a handful of malformed emails and duplicate rule hits. On an unsorted one it's a lock-contention horror show. Either way, you reconcile against it before you call the migration done.

This asynchronous, poll-and-reconcile shape is the same discipline that makes asynchronous Apex reliable — you hand the work off, you don't block on it, and you assume you'll need to inspect the results rather than trust that everything went through.

When Bulk API Isn't the Answer

Bulk API earns its overhead somewhere in the low thousands of records and up. Below that, it's the wrong tool. If I'm syncing a couple of hundred records from an external system, the job-create-upload-close-poll dance is pure ceremony, and the composite REST API or a straightforward set of SObject calls will finish before a Bulk job has even left the queued state.

The other case where Bulk API is wrong is anything that needs to be real-time. It's batch infrastructure. If a user is waiting on the other end of the request, the asynchronous model is working against you, and you want a synchronous REST call or a platform event instead.

The heuristic I use: high volume and can-wait means Bulk API; low volume or must-be-now means REST. Getting that split right is most of what separates an integration that scales from one that quietly melts under its first real load. If you want a second pair of eyes on where your integration sits on that line, that's the kind of review I do.

A well-run Bulk API load is genuinely boring — you sort the file, switch off the noise, kick off the job, and go make a coffee while it polls. A badly-run one is a 2am results file full of lock errors and a migration you have to run three times. I know which side of that I'd rather be on.