Deleting documents in bulk in Master Data

In this guide, you will learn how to delete every document that matches a filter in a Master Data data entity, using a single asynchronous deletion job. The job is available for both Master Data v1 and Master Data v2 data entities. When you need to remove everything matching one filter, use this job instead of the scroll and per-document DELETE loop described in Deleting documents in Master Data v1.

This feature is in open beta.

This guide focuses on the create, poll, and confirm flow. For the full operation set, parameters, schemas, and errors, see the API reference in the following table:

This action cannot be undone. Deleted documents are permanently lost. Confirm how many documents your filter matches before you create the job, so you can compare that number with the DocumentsDeleted value at the end.

To erase data for one specific customer for privacy reasons, follow Erasing customer data, which uses search and per-document deletion. This guide does not replace that flow.

• Both the create and status requests require a valid user token in the VtexIdclientAutCookie header.
• The filter field is required and uses the same syntax as the data entity search filter. Wildcards aren't allowed.
• Every field in the filter must be indexed. Internal fields such as createdIn are already indexed. Custom fields follow a different rule in each Master Data version:
  • In Master Data v1, a custom field is indexed when it exists in the data entity and has isSearchable enabled. You don't need to send the schema field.
  • In Master Data v2, a custom field is indexed when it is listed in the v-indexed array of a schema, and the request must declare that schema in the schema field.
• There is no limit to how many documents a single job can delete, but deleting hundreds of thousands of documents or more at once raises the chance of failure. Whenever possible, narrow the filter down to batches of a few tens of thousands of documents. For example, in a data entity with 10 million documents, run several jobs partitioned by a date field instead of a single filter that matches everything.
• Only one deletion job can be active per account and data entity at a time. While a job is InProgress, creating another job for the same data entity returns 409. Follow the existing job as described in step 2 and send the new request once that job reaches Success or Failed. If a job appears stuck, Master Data automatically allows a new job for that data entity after 12 hours.
• Before you create the job, search the data entity with the same filter criteria, and the same schema in the case of Master Data v2, record the number of matching documents as your baseline count, and save a few of the returned document IDs. You'll use these in step 3. To learn the query patterns and count how many documents a filter matches, see Extracting data from Master Data with search and scroll.

Deletion is asynchronous and runs in three steps. The create request deletes nothing by itself. It only creates the job and returns a JobId.

1. Create the job with Create bulk document deletion job.
2. Poll Get bulk document deletion job status until the job reaches Success or Failed.
3. Confirm the result against your baseline count and a follow-up search.

Send a POST request to Create bulk document deletion job, or to the Master Data v1 equivalent.

The following request body filters on an internal field:

In Master Data v2, filtering on a custom indexed field also requires the schema that declares the field as indexed:

In Master Data v1, filtering on a custom field doesn't require schema. The field only needs isSearchable enabled:

A successful request returns HTTP status 202 Accepted:

Save the JobId. It's the only way to track the operation.

Send a GET request to Get bulk document deletion job status, or to the Master Data v1 equivalent, using the JobId from step 1.

Poll until Status is Success or Failed. Example response for a finished job:

Each status requires a different action:

• InProgress: The job is running. Keep polling. Don't submit another job for this data entity.
• Success: The job deleted all matching documents. Compare DocumentsDeleted with your baseline count, as described in step 3.
• Failed: Processing stopped after the job exhausted its internal retries. Deletion is partial. Search the data entity with the same filter to see which documents remain, then create a new job with that filter to finish the deletion. For errors that reject a new job, see Create bulk document deletion job or the Master Data v1 equivalent.

Failed doesn't mean that nothing was deleted. Documents from batches the job already completed are permanently gone.

1. Read DocumentsDeleted from the job status response and compare it with the baseline count you recorded before creating the job.
2. Search the data entity again with the same filter criteria, and the same schema in the case of Master Data v2, using Search documents or the Master Data v1 equivalent. It should return no documents.
3. Send a GET request for one of the document IDs you saved, using Get document or the Master Data v1 equivalent. It should return an empty response.

After the documents are deleted, they are no longer counted in stored volume.

Master Data v1 and Master Data v2 validate the request on separate code paths, so some 400 errors are specific to one version, as indicated in the Version column.

• Deleting documents in Master Data v1: The difference between data entities and documents, per-document deletion for selective cleanups, and how to recover access to a data entity that was deleted from the Master Data v1 interface
• Extracting data from Master Data with search and scroll: How to query the documents a filter matches before deleting them
• Master Data API v1 reference and Master Data API v2 reference: Parameters, schemas, and errors for both versions