If you’re on WP Engine and you’re trying to modify
robots.txt on a
subdomain.wpengine.com URL (i.e. before you launch the site for real), then be aware that
robots.txt will always be set to ‘disallow all’ by WP Engine until you associate a custom domain with the site.
As per their website:
NOTE: By default, WP Engine restricts the traffic of search engines to any site using the install.wpengine.com domain. This means search engines will not be able to visit sites which are not currently in production using a custom domain.Robots.txt and WP Engine, 8 May 2020