[wp-trac] [WordPress Trac] #33156: Allow admin-ajax crawling

[WordPress Trac]

#33156: Allow admin-ajax crawling
-----------------------------------+-----------------------------
 Reporter:  joostdevalk            |       Owner:  SergeyBiryukov
     Type:  enhancement            |      Status:  closed
 Priority:  normal                 |   Milestone:  4.4
Component:  General                |     Version:
 Severity:  normal                 |  Resolution:  fixed
 Keywords:  2nd-opinion has-patch  |     Focuses:
-----------------------------------+-----------------------------

Comment (by KnowingArt_com):

 Replying to [comment:24 dmchale]:
 > 1. Create a physical robots.txt file on your server. If WordPress
 detects a physical file at the web root, it will not add to / remove from
 / modify that file in any way (this includes any plugins that dynamically
 modify the robots.txt file as well)
 >
 > 2. Use the `robots_txt` filter to modify the contents of the WordPress
 defaults https://developer.wordpress.org/reference/hooks/robots_txt/

 1. Nice, but without comments in robots.txt, how will the casual user
 know? My first impression was to 'touch robots.txt' but...

 1a. How will anyone know this won't break wp-sitemap.xml? That's how I
 ended up here. Upon further investigation, it seems *Google* contributed
 the code that adds wp-sitemap.xml to robots.txt If a Google employee adds
 some code to the WordPress robots.txt, that tells me Google wants that
 code to be there, and I am going to think twice about removing it.

 1b. Also casual user: If I remove this weird ajax thing, am I going to
 break something? Maybe I should investigate further. And down the rabbit
 hole we go into ajax themes with broken Googlebot renderings :-(

 2. That was my first attempt, but a) I crashed my site by trying some
 random Stack Exchange solution, b) I have many blogs to manage on several
 servers, and what if the blog changes themes? Is there a "functions.php"
 that affects all themes?, c) I don't really want a filter, I want to
 completely disable the creation of robots.txt, which is probably harder
 than it sounds.

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/33156#comment:26>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform