This guide explains how to develop and release a Robots app to manage ´robots.txt´ files for cross-border stores using Store Framework.

A robots.txt file controls which folders or files search engines can crawl on a website. Given its impact on store SEO, the following steps explain how to create robots.txt files for cross-border stores.

This guide only applies to cross-border stores. If your store isn't cross-border, see this guide.

Below is the basic structure of a robots.txt file:

• User-agent: Specifies the search engine robot to which the following rules apply. To apply the same rules to all robots, use *.
• Disallow: Defines the paths (relative to the root directory) of files or folders that the search engine specified in User-agent must not index.
• Allow: Specifies the paths of files within a Disallowed folder that can be crawled and indexed.

You must adjust these directives based on your scenario.

This feature is available for stores using the vtex.edition-store@3.x Edition app or newer. To check which Edition app is installed on your account, run vtex edition get. If your store is using an older Edition, open a ticket with VTEX Support requesting the installation of vtex.edition-store@5.x.

Follow the steps below to develop your Robots app:

1. Use the command below to clone the store-theme-robots app boilerplate repository.

   
   

   _10

   git clone https://github.com/vtex-apps/store-theme-robots

   
   

2. Once successfully cloned, open the local app directory in your code editor.

3. Open the manifest.json file and edit the vendor field with the name of the developing account.

   
   

   _10

   {

   _10

   "vendor": "myaccount",

   _10

   "name": "robots",

   _10

   "version": "0.0.1",

   _10

   "builders": {

   _10

   "sitemap": "0.x"

   _10

   },

   _10

   ...

   _10

   }

   
   

4. Inside the sitemap/robots folder, create a .txt robots file for each supported locale binding. The name of each file must be the id value of its respective binding.

   Follow this tutorial to check the store binding ids.

   Your app folder should have a structure similar to the following:

   
   

   _10

   store-theme-robots

   _10

   ├── manifest.json

   _10

   ├── README.md

   _10

   └─┬ sitemap

   _10

   └─┬ robots

   _10

   ├── 706e9126-d0fc-47de-9o2d-5f9649e61877.txt

   _10

   └── 748aafcf-1674-456d-9ffc-7ddb3f26e43f.txt

   
   

5. Edit each file in the sitemap/robots folder with the content you want for your robots.txt files.

6. Once everything is set up, use the terminal and VTEX IO CLI to log in to the VTEX account you're currently working in.

7. Run vtex use {workspace} to select a developer environment.

   Replace the values between curly brackets based on your scenario.

8. Run cd store-theme-robots to go to the local app directory.

9. Run vtex link to link your new app to your development workspace.

10. Check the robots file generated for each store by accessing https://{workspace}--{account}.myvtex.com/{locale}/robots.txt on your browser.

11. Follow the guide Deploying a new app version to test your Robots app in a production workspace and, once you're satisfied with the result, promote it to master.

    The Robots app must be tested in a production workspace.

Now you're ready to check your store's robots.txt files by accessing https://{account}.myvtex.com/{locale}/robots.txt in your browser, replacing the values between the curly brackets based on your scenario.