[wp-trac] [WordPress Trac] #18465: robots.txt should tell Google to not index wp-admin and wp-includes

[WordPress Trac]

#18465: robots.txt should tell Google to not index wp-admin and wp-includes
--------------------------+--------------------
 Reporter:  Viper007Bond  |       Owner:
     Type:  enhancement   |      Status:  new
 Priority:  lowest        |   Milestone:  3.3
Component:  General       |     Version:  3.2.1
 Severity:  trivial       |  Resolution:
 Keywords:  has-patch     |
--------------------------+--------------------

Comment (by SergeyBiryukov):

 The main concern in #17601 were the errors in server logs due to indexing
 `wp-admin` and `wp-includes`. Indexing `readme.html` doesn't create such
 errors, but probably doesn't make much sense too, so we could reduce
 unnecessary crawling even more.

 That said, I'm not sure `readme.html` ends up in search results, since we
 have `index.php` in the root directory. Perhaps there's a link to
 `readme.html` somewhere on those sites, or they're missing `index.php` in
 `DirectoryIndex`, or their hosts don't support PHP (so they can't run
 !WordPress anyway).

 For `wp-includes`, there are
 [http://www.google.com/search?q=intitle%3A%22Index+Of%22+inurl%3Awp-
 includes much more results] (currently about 5,540,000).

-- 
Ticket URL: <http://core.trac.wordpress.org/ticket/18465#comment:20>
WordPress Trac <http://core.trac.wordpress.org/>
WordPress blogging software