[buddypress-trac] [BuddyPress Trac] #6841: Framework for bulk data handling after updates

[buddypress-trac]

#6841: Framework for bulk data handling after updates
----------------------------------+-----------------------------
 Reporter:  boonebgorges          |      Owner:
     Type:  enhancement           |     Status:  new
 Priority:  normal                |  Milestone:  Awaiting Review
Component:  API - Update/Install  |    Version:
 Severity:  normal                |   Keywords:
----------------------------------+-----------------------------
 We have in the past, and will have in the future, cases where large
 amounts of data needs to be processed after a major BP update. The case
 I'm currently working on is #6413 - profile field visibility - but it's
 come up before: moving user last_activity to the activity table, migrating
 signups to wp_signups, etc. On large (or even medium) installations,
 handling huge amounts of data in a single pageload can easily exhaust
 system resources. We need a better way.

 On #6413, I suggested one possibility: a background batch processor, based
 on wp-cron. This is good because it requires no UI, and uses existing
 tools. It's bad for a bunch of reasons. wp-cron is buggy and prone to race
 conditions. wp-cron jobs can't run frequently enough for our purposes,
 leaving us in a state of half-migration for long periods of time. wp-cron
 is not reliably usable on all installations. Etc.

 Another possibility is a loopback batch processor. It could also work in
 the background, but would directly fire off an asynchronous request
 (`wp_remote_post()` or whatever) instead of waiting for the next cron run.
 So: BP update is pageload A. A fires off a `wp_remote_post()`, trigger
 pageload B. B runs a batch process, and when it's finished, fires
 `wp_remote_post()` to C. And so forth, until the batches are completed.
 One problem here is that self-request can sometimes be blocked by
 webserver configuration. Another is that an out-of-control loop, due to
 faulty data or a bug in the implementation, could set off an infinite
 series of requests that would be difficult for the average BP user to
 debug or stop.

 The last option is to use an AJAX-powered interface. This avoids a lot of
 the complications described above. It'll work on all servers, and a bad
 migration can be stopped by closing the browser window. But (a) it
 requires JS, and (b) it requires user action. Item (a) means that we have
 to build no-JS compatibility  of at least a rudimentary kind (I think,
 though maybe I could be convinced otherwise). Item (b) means that the
 admin may decide not to run the migrator right away - or at all. And this
 has ramifications for the way we build data migrations: the post-migration
 schema must always be able to fall back on the pre-migration schema, since
 the migration may not be finished for days or weeks or months. (In fact,
 this is probably true no matter what route we take.)

 Does anyone have strong opinions on this? If we're going to go with an
 AJAX interface, let's have a discussion about the UX. How do we nag the
 administrator to run the updater after BP update? Do we need a new admin
 panel? Etc.

--
Ticket URL: <https://buddypress.trac.wordpress.org/ticket/6841>
BuddyPress Trac <http://buddypress.org/>
BuddyPress Trac