[wp-hackers] State of WP

[Kevin McCann]

Brian Layman wrote:
> OK, Kevin, I'll get back to your topic here.  The requirements you've
> mentioned are significant.  100,000 sites, running out of any central
> system, will require significant resources and optimizations.  AFAIK, any
> existing blog/CMS system that you choose will require customization to scale
> to that level of deployment.  
>
> MySQL, to be brutally honest, is not a very mature database system. (Does
> anyone else here have a transactions-check mark shirt from the 4.1(?)
> deployment?)  MySQL IS rapidly improving.  So, I've got to ask you if your
> 100,000 site quote is an eventually goal, or a realistic appraisal of your
> current planned deployment.  (BTW, either way, I believe WordPress will give
> you a good foundation.)  The size of the initial deployment, may be the
> biggest factor in deciding what you want to do next.
>   

Hi Brian,

Thanks for the detailed response (and thanks to others who responded).

We'd probably have at least 10,000 right off the bat (with blog data 
ported from another system). It would likely grow to 100,000 within 6-12 
months. Could eventually get to half-a-million or more. We currently 
have a half-decent infrastructure with multiple web servers, mail 
servers, Master and Slave MySQL servers, load balancers, etc. which we 
use for other services. We already use MySQL, which is why we'd want to 
use it if we can.

> DrMike has already told you that MySQL cannot handle that capacity.  So, if
> you are deploying at that size, you need to invest design and implementation
> into converting the database over to something that can handle that scale.
> For example, Interbase or Firebird should handle that with out a problem and
> doesn't limit your platform, but you'll have to over come issues like how to
> handle the missing MySQL InsertID feature.  MSSQL is another option, of
> course, and there are others beyond that.
>   

DrMike mentioned a limitation of 32k files, but isn't it actually 32k 
sub-directories within a directory? So if it's one big database, isn't 
it just a whole lot of files within one directory, not 32k+ 
sub-directories within a directory? Is the other limitation in the 
number of directories required for per-site media file storage (like 
images, etc.)? Also, someone mentioned to me that MySQL uses 3 files per 
table only when using ISAM. He said if you use InnoDB it's not a 
problem. True?

> If, in the near term, your actual needs are significantly smaller,
> implementing your site in phases, maybe the key to success. Take advantage
> of improved compatibility with the WordPress core by deploying under MySQL.
> You could then use the first phase to determine if what your scaling
> problems will be and THEN decide your best avenue of attack. WordPress and
> MySQL will have matured further, you will get a number of improvements by a
> quick upgrade to the current system, you will be more proficient with the
> actual WordPress code, and then you can switch over to the actual end system
> that is capable of handling the workload you've mentioned.
>   

My fear is that we would reach 32k quickly. If that is indeed the danger 
number, I'd rather be prepared for it. Maybe we have outgrown MySQL? 
Will investigate further.

> That's my take on the whole thing without knowing more about the why's and
> wherefore's of your actual plan.  And please don't take the grumblings and
> gripes on a news list as large red flags.  The core folks who live and
> breath WordPress each day are a good bunch of people who give every change
> request due consideration.  
>   

No worries, good to know you're a nice bunch of people!

> As for how many developers there are, that varies.  There are a large number
> of people that contribute.  Each has their own special concern, point of
> view, and desired goal.  That's the nature of open source.  The number of
> people who can commit the changes that are submitted is smaller number, oh,
> around half a dozen, I guess.  They take a look at how WordPress is being
> used today, and just as importantly, how it isn't being used.  The then the
> development to the benefit of the community as a whole.
>
> WordPress is being adopted for ever larger deployments and is the core
> technology of an increasingly larger number of media outlets.  You would not
> be alone in your choice of WordPress for your foundation and I expect the
> choice would serve you well.
>
> I hope that answers some of your questions.
>   

It sounds like there would be benefits to starting with WP as a foundation.

I did read some threads in the WPMU forum and found them rather 
interesting (thanks for the tip DrMike). I'm a bit confused about some 
of what I read (especially where there is talk of using multiple 
databases (wp.com uses 4096?), as I thought the whole idea was to use 
many tables within one database. We currently use Drupal, with one DB 
per site, and this is what we wanted to get away from. I have also been 
wondering why the approach for multi-sites would not be to have one 
database and a fixed number of tables, and have blog-specific data 
referenced by blog_id (like "post_blog_id" in a "posts" table). 
Therefore "select post_id, post_title ... from posts where post_blog_id 
= '$blogid' and ..." Simply too many records and slow queries? Is the 
WPMU forum the best place to follow up with these questions?

Thanks for your help.

- Kevin