Home
Documentation

Data Submission Flows

Submitting Datasets to the Dataset Exchange API

There are two options for submitting datasets through the Dataset Exchange API:

  • One for those with dataset files at existing pre-signed URLS

  • One for users who want to request a pre-signed URL where they can upload a dataset file

These flows are available after Authentication to the Dataset Exchange API is completed.

📌

Option 1

A dataset in JSON or CSV format is already available at a pre-signed URL to load into the Dataset Exchange API.

If datasets (formatted as JSON, CSV, or TSV) are already available via a pre-signed URL available via AWS S3 or GCP Cloud Storage, it can be loaded directly to the Dataset Exchange API:

  1. Hit POST /datasets to create the new dataset in DDx API's database and define the schema.

  2. Once an id is returned for the newly created dataset in the response body of Step 1, hit POST /datasets/:id/records:load with the pre-signed URL in the body of the request. This responds 202 Accepted with a jobId, meaning the load has been queued — the records are not loaded yet.

  3. Poll GET /datasets/jobs/:jobId/logs:current until the job reaches a terminal status. See Knowing when your records have actually loaded below.

  4. Done!

📌

Option 2:

If not already available, request a pre-signed URL to load a dataset into the Dataset Exchange API.

To load a dataset to a pre-signed UR, request an upload URL from the Dataset Exchange API.

In order to generate a pre-signed URL and load a dataset:

  1. Hit POST /datasets to create the new dataset in our database and define the schema.

  2. Once an id is returned for the newly created dataset in the response body of Step 1, hit POST /datasets/:id/uploadUrl with a string specifying the content type of the dataset. Content type options are CSV or JSON. The /uploadUrl endpoint will return a pre-signed URL to a GCS bucket where the dataset can be loaded, along with a jobId. Please note pre-signed URLs are only valid for 1 hour. This endpoint responds 200 OK, but that only means the URL was generated — no records are loaded until you complete Step 3.

  3. Load the dataset to the pre-signed URL received in Step 2.

  4. Poll GET /datasets/jobs/:jobId/logs:current until the job reaches a terminal status. See Knowing when your records have actually loaded below.

  5. Done!

Knowing when your records have actually loaded

Loading records is asynchronous in both flows above. A success response tells you the work was accepted, not that the records are queryable. Neither status code means the load has finished:

EndpointResponseWhat it means
POST /datasets/:id/records:load202 AcceptedThe load job was queued. Records are not loaded yet.
POST /datasets/:id/uploadUrl200 OKThe pre-signed URL was generated. Nothing loads until you upload your file to that URL, and the load then runs asynchronously after that.

Both responses include a jobId. That job is the only source of truth for whether a load finished. To check it, call:

GET /datasets/jobs/:jobId/logs:current

The datasetJobStatusName field in the response will be one of:

StatusMeaning
PendingThe job has been created but has not started processing. Still in flight.
ProcessingThe job is actively loading. Still in flight.
CompletedTerminal. The records are loaded.
UnknownErrorTerminal. The load failed — see the errorData field in the response for details.
⚠️

Wait for Completed before loading again

Poll until the job reaches Completed or UnknownError before sending another load request for the same dataset — including ?mode=append.

Sending a second load request for a dataset whose previous job is still Pending or Processing returns 409 Conflict. The response names the job ID you need to wait on:

A records load job is already in progress for this dataset (job ID 6f1c…).
Record loads are asynchronous. Poll GET /datasets/jobs/6f1c…/logs:current
until the job status is Completed or UnknownError, then retry this request.

A 409 is safe to retry once the job you were told about has finished. Poll on an interval with backoff rather than in a tight loop — load duration scales with file size.

ℹ️

About recordCount

The recordCount on a dataset (for example from GET /datasets/:id) is a pre-deduplication count. It counts every row written to the dataset, including superseded versions of a record and rows marked as deleted. As a result:

  • Loading records with a primary key that already exists adds to recordCount rather than replacing the earlier version.
  • Deleting records does not decrease recordCount.
  • recordCount can be larger than the number of unique, live records in the dataset.

The deduplicated set (latest version per primary key, with deletes removed) is what is delivered downstream and on retrieval, not recordCount.

🤔

Questions? We're here to help! Reach out to us at [email protected].


Updated 16 days ago

What’s Next

Did this page help you?