robots.txt and Preventing your Site Being Indexed by Search Engines

StoreConnect provides a robots.txt1 file automatically via an automatically generated XML file which is regenerated daily.

For stores not in test mode, your store’s robots.txt is available at all times via your-store.com/robots.txt. If you are running multiple stores within StoreConnect, each store will have it’s own, unique robots.txt file.

When downloading, use your browser’s ‘Private Mode’ to ensure you get the most current version otherwise you could be downloading an older version from your cache.

StoreConnect auto-disables the robots.txt via a Disallow (instead of Allow) when in test mode or if the site is accessed via the temporary domain that comes with your app.

 

Learn more about robots.txt

 

Note: each store will have it’s own separate robots.txt file.

 

The robot.txt file generated looks like this:

Example Store robots.txt file per http://www.robotstxt.org/ 

User-agent: *
Allow: /

Sitemap: https://store.example.com/sitemap.xml

 

How to Disable robots.txt in a Live Store

  1. Go to your store and add or edit the store's Head Content
  2. Add this <meta> tag: <meta name="robots" content="noindex">
  3. Save the Content Block

 

 

  1. robots.txt: A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of a search engine. To keep a web page out of a search engine, block indexing with noindex or password-protect the page. ↩

 
 
Back to Documentation