Learn how to extract data from Master Data using the search and scroll endpoints and best practices for optimizing queries and handling large datasets.
In this guide, you will learn how to extract data from Master Data using the search and scroll endpoints, including when to use each route, how to optimize queries, and best practices.
In Master Data v1, you can export data directly from the interface. See Exporting data from Master Data v1 for more information.
Search
The search route is ideal when you need to find a specific set of documents within your store. It is particularly useful for paginated queries, where you want to retrieve up to 10,000 documents in small chunks over multiple requests and each page is limited to 100 documents.
When paginating, the
_sortparameter is recommended. The API does not guarantee a specific order by default. Therefore, omitting the_sortparameter may lead to duplicate documents or return unexpected pages.
Check the search endpoint reference for your Master Data version:
Best practices
When using the search endpoint, these best practices will help optimize your data retrieval process:
- Apply filters to narrow your search: Improve performance by reducing the number of documents returned. This speeds up the query and ensures that your requests are more efficient.
- Use exact values for queries instead of wildcards (
*): Heavy usage of wildcards may be subject to temporary blocks. - Avoid large datasets: If you are querying many documents, break your query into smaller intervals.
Learn more about Search pagination in the Master Data API.
Scroll
The scroll route is designed for extensive data retrieval, especially when integrating Master Data with external systems. It is the best choice if you need to query the entire database or deal with over 10,000 documents.
Check the scroll endpoint reference for your Master Data version:
Your first scroll request will return a token in the X-VTEX-MD-TOKEN response header. Submit this value in the _token query parameter for your next requests until you receive an empty list, which indicates that all documents have been retrieved.
Learn more about Scroll pagination in the Master Data API.
Scroll best practices
To ensure efficient and reliable data retrieval, follow these strategies when using the scroll endpoint:
- Implement filters to divide the request into smaller batches: This reduces the likelihood of timeouts. For example, you might filter by created date and process data month by month. Smaller batches are also easier to reprocess if a timeout occurs, making your operation more resilient.
- Run up to 10 scrolls simultaneously per account: Limiting the number of parallel scrolls helps prevent errors and timeouts. By using filters to create smaller batches and parallelizing these batches in a controlled manner, you can speed up data retrieval while reducing the risk of overloading the account.
Scroll behavior and limitations
- Each scroll operation allows only one query for the duration of the token. This means that you cannot change a scroll’s query by changing parameters after the first request. You can navigate pages of the original first request until the token expires, or initiate other scrolls (up to 10 simultaneously).
- If Master Data stops receiving requests with the scroll
X-VTEX-MD-TOKENtoken, it will expire in 20 minutes. After that, you can make new scroll requests, limited to 10 simultaneous scrolls.- The maximum number of documents per scroll request is 1,000.