[wp-trac] [WordPress Trac] #36390: Disallow: /wp-json/ crawling

[WordPress Trac]

#36390: Disallow: /wp-json/ crawling
----------------------------+-----------------------------
 Reporter:  SergeyBiryukov  |      Owner:
     Type:  defect (bug)    |     Status:  new
 Priority:  normal          |  Milestone:  Awaiting Review
Component:  REST API        |    Version:
 Severity:  normal          |   Keywords:
  Focuses:                  |
----------------------------+-----------------------------
 Since 4.4, I'm seeing support topics about `/wp-json/` being crawled by
 search engines, apparently leading to penalties for duplicate content in
 some cases, caused by `wp_oembed_add_discovery_links()`.

 Some blog posts suggest disallowing `/wp-json/` in `robots.txt` or even
 disabling REST API altogether.

 Should we add `Disallow: /wp-json/` to `do_robots()`?

--
Ticket URL: <https://core.trac.wordpress.org/ticket/36390>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform