[wp-trac] [WordPress Trac] #36390: Disallow: /wp-json/ crawling

[WordPress Trac]

#36390: Disallow: /wp-json/ crawling
----------------------------+------------------------------
 Reporter:  SergeyBiryukov  |       Owner:
     Type:  defect (bug)    |      Status:  new
 Priority:  normal          |   Milestone:  Awaiting Review
Component:  REST API        |     Version:
 Severity:  normal          |  Resolution:
 Keywords:                  |     Focuses:
----------------------------+------------------------------
Description changed by SergeyBiryukov:

Old description:

> Since 4.4, I'm seeing support topics about `/wp-json/` being crawled by
> search engines, apparently leading to penalties for duplicate content in
> some cases, caused by `wp_oembed_add_discovery_links()`.
>
> Some blog posts suggest disallowing `/wp-json/` in `robots.txt` or even
> disabling REST API altogether.
>
> Should we add `Disallow: /wp-json/` to `do_robots()`?

New description:

 Since 4.4, I'm seeing support topics about `/wp-json/` being crawled by
 search engines, apparently leading to penalties for duplicate content in
 some cases, caused by `wp_oembed_add_discovery_links()`, or higher server
 load in other.

 Some blog posts suggest disallowing `/wp-json/` in `robots.txt` or even
 disabling REST API altogether.

 Should we add `Disallow: /wp-json/` to `do_robots()`?

--

--
Ticket URL: <https://core.trac.wordpress.org/ticket/36390#comment:1>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform