If you’re on WP Engine and you’re trying to modify robots.txt
on a subdomain.wpengine.com
URL (i.e. before you launch the site for real), then be aware that robots.txt
will always be set to ‘disallow all’ by WP Engine until you associate a custom domain with the site.
As per their website:
NOTE: By default, WP Engine restricts the traffic of search engines to any site using the install.wpengine.com domain. This means search engines will not be able to visit sites which are not currently in production using a custom domain.
Robots.txt and WP Engine, 8 May 2020