[wp-trac] [WordPress Trac] #52900: Instantly index WordPress web sites content in Search Engines

[WordPress Trac]

#52900: Instantly index WordPress web sites content in Search Engines
--------------------------------------+-----------------------------
 Reporter:  fabricecanel              |       Owner:  (none)
     Type:  feature request           |      Status:  reopened
 Priority:  normal                    |   Milestone:  Future Release
Component:  General                   |     Version:
 Severity:  normal                    |  Resolution:
 Keywords:  has-patch has-unit-tests  |     Focuses:
--------------------------------------+-----------------------------

Comment (by desrosj):

 Finally coming back to this one after having it open in a browser tab for
 a few weeks.

 Initially I planned on adding the `close` keyword to again suggest closing
 as `maybelater`. Before doing that, I looked a bit deeper at how the
 plugin works while setting up IndexNow on two sites that I mange: one that
 has been verified in Webmaster Tools and one that has not. That
 designation ended up not mattering in the end, I was just unable to view
 IndexNow outside of the WordPress dashboard for the unverified site to see
 if anything has changed.

 While I am still in the `maybelater` camp, I'll admit that I'm a bit more
 open to this request knowing that the key is more of a shared secret
 instead of a true key that can be changed at anytime. The key itself does
 not matter. What matters is that the key can be verified in some way.
 Usually this verification happens is by confirming that
 `https://siteurl.com/KEYVALUE.txt` exists and contains the key). The
 plugin currently uses `wp_generate_uuid4()` to generate a string, replaces
 hyphens, and then stores the key in an option base64 encoded.

 This addresses one of my previous concerns about the user possibly needing
 to be aware of the key at all. Requiring an API key to make use of a
 service or feature in WordPress Core is always a blocker. But this could
 reasonably be managed quietly behind the scenes without requiring an
 external request or any user intervention at all, so I'm willing to give
 it some consideration.

 That leaves my other concern about the overall adoption of IndexNow. So
 far, most participants (myself included) have considered this under the
 lens that Google is by far the dominant search engine by market share.
 This is still true today, but I'm wondering if we should re-frame this
 thinking because it focuses on usage, not the actual amount of traffic
 related to crawling a site. Each search engine will always crawl a site
 independently, which means a site known by 10 search engines with 1,000
 URLs the sitemap could regularly be crawled around 10,000 times over a
 given time period just for crawling purposes.

 @fabricecanel above you mentioned "improving crawl efficiency, reducing
 costs, and decreasing CO2 emissions," but could you possibly provide more
 specific data? This makes sense in theory, but I'm interested in learning
 more about the impact of this feature at an individual site level.

 I'd like to see some data about:
 - what is the actual difference for a site when using IndexNow vs. the
 traditional methods such as relying on a Sitemap? As an example, "when
 site A adds support for IndexNow, Bing decreases the overall number of
 requests to the site by Y% over time period Z." Averages are probably
 fine, but having several tiers of site size would be helpful to identify
 how differently blogs without much content vs. large websites with 1,000s
 of posts and pages.
 - How has checking `lastmod` dates changed the number of requests made for
 crawling purposes to each of the same site categories when using only the
 "daily crawl to catch up" technique? How does this new baseline compare to
 similar sites using IndexNow?
 - Any other data that could show the positive impact on a per site basis.

 Updating your site in search engine indexes near instantly is compelling
 but not necessarily enough on it's own. If the data shows that search
 engines using IndexNow substantially decreases the overall amount of
 crawling-related traffic to a site, then that could strengthen the
 rationale here and make a better case.

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/52900#comment:55>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform