Bulk processing

The _bulk endpoint allows for efficient processing of multiple requests in a single operation. It supports streaming, parallel or sequential processing, and atomic execution.

Request formats

Non-streaming

For standard bulk requests, send operations as a JSON array with Content-Type: application/json:

[
  {"action": "CREATE_TRANSACTION", "data": {"postings": [{"source": "world", "amount": 100, "asset": "USD", "destination": "alice"}]}},
  {"action": "CREATE_TRANSACTION", "data": {"postings": [{"source": "world", "amount": 200, "asset": "USD", "destination": "bob"}]}}
]

By default, the ledger has a maximum bulk transaction size of 100 items. This limitation helps prevent timeouts when processing large bulk requests.

Streaming

Bulk requests can be streamed without requiring the entire request to be loaded into memory.

Results are kept in memory until the full stream is complete, which may result in large responses for big datasets. Consider breaking very large operations into smaller batches.

For processing more than 100 items, we recommend using streaming mode instead of increasing the bulk size limit.

To enable streaming, include one of the following content type headers in your HTTP request:

• For a script stream, include content type application/vnd.formance.ledger.api.v2.bulk+script-stream
• For a JSON stream, include content type application/vnd.formance.ledger.api.v2.bulk+json-stream

Script stream format

For a script stream, each Numscript transaction must be wrapped with //script and //end delimiters:

//script
send [USD 100] (
  source = @world
  destination = @alice
)
//end
//script
send [USD 100] (
  source = @world
  destination = @bob
)
//end

JSON stream format

For a JSON stream, send an array of elements in the request body. Example:

[
  {"action": "CREATE_TRANSACTION", "data": {"postings": [{"source": "world", "amount": 100, "asset": "USD", "destination": "bank"}]}},
  {"action": "CREATE_TRANSACTION", "data": {"postings": [{"source": "world", "amount": 200, "asset": "USD", "destination": "bank"}]}}
]

Available actions for JSON bulk operations (script streams support CREATE_TRANSACTION only):

• CREATE_TRANSACTION - Create a new transaction
• ADD_METADATA - Add metadata to an account or transaction
• REVERT_TRANSACTION - Revert a transaction
• DELETE_METADATA - Delete metadata from an account or transaction

Processing options

Parallel v. Sequential Processing

Bulk elements can be processed either in parallel or sequentially, controlled by the parallel query parameter:

• parallel=true: Elements are processed in parallel, improving throughput.
• parallel=false: Elements are processed sequentially, maintaining order.

Atomic Bulk Execution

Bulk elements can be committed atomically or independently using the atomic query parameter:

• atomic=true: The entire batch is committed as a single transaction. If any element fails, the entire batch is rolled back.
• atomic=false: Each element is processed independently. If one element fails, it does not affect the others.

Since requests can be processed either in parallel or atomically, you cannot set parallel=true and atomic=true at the same time.

Idempotency

Idempotency keys prevent duplicate transactions when replaying bulk requests after failures. Each bulk element can specify its own key.

For script streams, add ik=<key> to the script header:

//script ik=transaction-001
send [USD 100] (
  source = @world
  destination = @alice
)
//end

For JSON streams, add the ik field to each element:

{"action": "CREATE_TRANSACTION", "ik": "transaction-001", "data": {"postings": [{"source": "world", "amount": 100, "asset": "USD", "destination": "bank"}]}}

Configuration

Bulk size limits

The 100 item limit applies only to non-streaming requests. If your use case requires a larger limit, you can configure it using the --bulk-max-size flag or BULK_MAX_SIZE environment variable:

Command Line

ledger serve --bulk-max-size 1000

Increasing the bulk size does not necessarily improve write performance. Test different values to find the optimal setting for your use case.

Examples

Script streaming

To send a bulk request with script streaming:

curl -H "Authorization: Bearer $(fctl cloud generate-personal-token)" \
     -X POST <stack-url>/api/ledger/v2/<ledger-name>/_bulk?parallel=true&atomic=false \
     -H 'Content-Type: application/vnd.formance.ledger.api.v2.bulk+script-stream' \
     --data-binary @<stream file location>

JSON streaming

To send a bulk request with JSON streaming:

curl -H "Authorization: Bearer $(fctl cloud generate-personal-token)" \
     -X POST <stack-url>/api/ledger/v2/<ledger-name>/_bulk?parallel=true&atomic=false \
     -H 'Content-Type: application/vnd.formance.ledger.api.v2.bulk+json-stream' \
     --data-binary @<json stream file location>

For more details on the bulk API parameters and response format, see the API Reference.