[wp-edu] How do you manage a site with 100+ sub sites with WordPress?

Mon Mar 16 20:39:08 UTC 2015

We are about to hit 25K sites on a multisite install; we also use domain mapping. We found that so far we haven’t needed to shard the database. As long as your database server can handle that volume/load then you will be fine. A database this large does require doing some things differently. Our production sysadmin switched to using the percona tools for backup quickly and we had also looked at doing LVM snapshots which would have also been a good solution. All tables are innodb be default and we use hyperdb to manage the load between a single master and several slave/read databases. 

We also use memcache, probably soon redis, to handle the WP Object Cache. This helps greatly with logged in users, as a single page load can be taken from 100+ sql queries to build down to less than 10 on each load. For anonymous user caching we use two varnish servers on the frontend to manage that load, balanced between four web servers.

We are working an management tools to start pruning old sites, as usually this seems to be hindsight, that will look for discontinued student/staff accounts and start to disable/delete/archive them from the system.

Jason

p: (814) 865-1840, c: (814) 777-7665
Systems Administrator
Teaching and Learning with Technology, Information Technology Services
The Pennsylvania State University

> On Mar 13, 2015, at 10:23 AM, Michael Barnard <mbarnard at mtholyoke.edu> wrote:
> 
> I'm curious about the need for HyperDB and its ilk. I'm at Mount Holyoke College now, but I used to maintain a large Wordpress installation for UMass Amherst. It was backed by MySQL using MyISAM tables and, when it was handed to me, it had been running for several years and never had any blogs deleted from it. I seem to recall there being something on the order of 10,000+ blogs, with 9 database tables per blog and three files per table, so a couple hundred thousand files in total. Many (most?) of these were abandoned, but they still saw some traffic from spammers and crawlers. Surprisingly, we had no significant performance problems with this installation, and we never considered doing any kind of database sharding. The one problem we did run into was with mysqldump - there were just too many tables for it to cope with, so it couldn't dump the Wordpress database at all. I didn't really ever need SQL dumps of Wordpress, though - I dealt with backups using LVM snapshots, for example. In any case, I never ran up on any kind of limit on the total number of tables per database. Is that an InnoDB thing?
> 
> 
> On Fri, Mar 13, 2015 at 9:49 AM, Joseph Ugoretz <joseph.ugoretz at mhc.cuny.edu <mailto:joseph.ugoretz at mhc.cuny.edu>> wrote:
> D'Arcy, what numbers are you at now? We aren't seeing performance problems yet at 4,000 sites but I expect to go to hyperdb soon. 
> 
> I think my colleague Luke at Baruch College (10,000 or so sites) is using SharDB. 
> 
> -- 
> Joseph Ugoretz, PhD
> Associate Dean
> Teaching, Learning and Technology
> Macaulay Honors College, CUNY
> 35 West 67th St. <>
> New York, New York 10023 <>
> 212-729-2920 <tel://212-729-2920>
> 
> On Mar 13, 2015, at 9:26 AM, D'Arcy L Norman <dlnorman at ucalgary.ca <mailto:dlnorman at ucalgary.ca>> wrote:
> 
>> The biggest tweak you can do is add database scaling, so you don't hit the max-tables limit in a single database. HyperDB or MultiDB would do that - I use MultiDB on UCalgaryBlogs.ca <http://ucalgaryblogs.ca/> and it works great. Spreads the tables across several databases (16, 256 or 4096), which can hold up to "millions" of blog sites. Haven't had enough to test that claim, though. The 16 database config scales to 50,000 blogs. That should hold us for a few years.
>> 
>> Multi-DB: https://premium.wpmudev.org/project/multi-db/ <https://premium.wpmudev.org/project/multi-db/>
>> 
>> HyperDB: https://wordpress.org/plugins/hyperdb/ <https://wordpress.org/plugins/hyperdb/>
>> 
>> I started with MultiDB, back before HyperDB was a thing. If I was starting fresh, I'd go with HyperDB because it's free, maintained by Automattic, and basically what's used on WordPress.com <http://wordpress.com/>. MultiDB woks find, but requires a subscription fee. I haven't had the energy to migrate UCalgaryBlogs to it, because it works.
>> 
>> - D
>> 
>> 
>> From: wp-edu [wp-edu-bounces at lists.automattic.com <mailto:wp-edu-bounces at lists.automattic.com>] on behalf of Joseph Ugoretz [joseph.ugoretz at mhc.cuny.edu <mailto:joseph.ugoretz at mhc.cuny.edu>]
>> Sent: March 13, 2015 06:48 AM
>> To: Low-traffic list discussing WordPress in education.
>> Subject: Re: [wp-edu] How do you manage a site with 100+ sub sites with WordPress?
>> 
>> These are class sites and student eportfolios, so we do not ever retire or purge sites. 
>> 
>> Students do return to old sites and take them up again even if they have been inactive for a long time, even years. 
>> 
>> Part of our policy and what we guarantee to students is that these sites will remain available until and unless the students themselves delete them (except in the case of serious violations of our community guidelines...has not happened yet). 
>> 
>> Joe
>> -- 
>> Joseph Ugoretz, PhD
>> Associate Dean
>> Teaching, Learning and Technology
>> Macaulay Honors College, CUNY
>> 35 West 67th St. <http://urlblockederror.aspx/>
>> New York, New York 10023 <http://urlblockederror.aspx/>
>> 212-729-2920 <tel://212-729-2920>
>> 
>> On Mar 13, 2015, at 1:47 AM, Hugh Paterson III <hugh at thejourneyler.org <mailto:hugh at thejourneyler.org>> wrote:
>> 
>>> What sort of site retirement policy do you maintain? How do admin know when to purge a site from a set of 4,000?
>>> 
>>> - Hugh Paterson III
>>> 
>>> 
>>> 
>>> On Mar 12, 2015, at 4:37 PM, Joseph Ugoretz <joseph.ugoretz at mhc.cuny.edu <mailto:joseph.ugoretz at mhc.cuny.edu>> wrote:
>>> 
>>>> WordPress multisite works very well for this. We currently have, on our big multisite install, over 4,000 sites, with many different themes. We aren't currently using domain mapping but may soon. 
>>>> 
>>>> My CUNY colleagues at Baruch College have a similar set up with far more sites. 
>>>> 
>>>> -- 
>>>> Joseph Ugoretz, PhD
>>>> Associate Dean
>>>> Teaching, Learning and Technology
>>>> Macaulay Honors College, CUNY
>>>> 35 West 67th St. <http://urlblockederror.aspx/>
>>>> New York, New York 10023 <http://urlblockederror.aspx/>
>>>> 212-729-2920 <tel://212-729-2920>
>>>> 
>>>> On Mar 12, 2015, at 7:26 PM, Mcmillan, Scott <scott.mcmillan at ubc.ca <mailto:scott.mcmillan at ubc.ca>> wrote:
>>>> 
>>>>> Hi Ryan,
>>>>> 
>>>>> On the UBC CMS we have just over 900 domain mapped sites.
>>>>> We run one Network within multi site and have one central theme and allow users to map their domains using the Domain Mapping Plugin. 
>>>>> 
>>>>> It's worked well for users so far. 
>>>>> 
>>>>> -Scott
>>>>> From: wp-edu [wp-edu-bounces at lists.automattic.com <mailto:wp-edu-bounces at lists.automattic.com>] on behalf of Ryan Kite [rkite at yvcc.edu <mailto:rkite at yvcc.edu>]
>>>>> Sent: Thursday, March 12, 2015 3:46 PM
>>>>> To: wp-edu at lists.automattic.com <mailto:wp-edu at lists.automattic.com>
>>>>> Subject: [wp-edu] How do you manage a site with 100+ sub sites with WordPress?
>>>>> 
>>>>> We are looking at going with WordPress with at our college, currently have about 100+ sub sites under one master, each with unique navigation and content.
>>>>>  
>>>>> How do you manage this in WordPress?
>>>>>  
>>>>> Ryan Kite
>>>>> Web Designer
>>>>> 509.574.4788 <tel:509.574.4788>  | rkite at yvcc.edu <mailto:rkite at yvcc.edu>
>>>>> Yakima Valley Community College
>>>>> http://www.linkedin.com/in/ryankite/ <http://www.linkedin.com/in/ryankite/>
>>>>>  
>>>>> _______________________________________________
>>>>> wp-edu mailing list
>>>>> wp-edu at lists.automattic.com <mailto:wp-edu at lists.automattic.com>
>>>>> http://lists.automattic.com/mailman/listinfo/wp-edu <http://lists.automattic.com/mailman/listinfo/wp-edu>
>>>> _______________________________________________
>>>> wp-edu mailing list
>>>> wp-edu at lists.automattic.com <mailto:wp-edu at lists.automattic.com>
>>>> http://lists.automattic.com/mailman/listinfo/wp-edu <http://lists.automattic.com/mailman/listinfo/wp-edu>
>>> _______________________________________________
>>> wp-edu mailing list
>>> wp-edu at lists.automattic.com <mailto:wp-edu at lists.automattic.com>
>>> http://lists.automattic.com/mailman/listinfo/wp-edu <http://lists.automattic.com/mailman/listinfo/wp-edu>
>> _______________________________________________
>> wp-edu mailing list
>> wp-edu at lists.automattic.com <mailto:wp-edu at lists.automattic.com>
>> http://lists.automattic.com/mailman/listinfo/wp-edu <http://lists.automattic.com/mailman/listinfo/wp-edu>
> 
> _______________________________________________
> wp-edu mailing list
> wp-edu at lists.automattic.com <mailto:wp-edu at lists.automattic.com>
> http://lists.automattic.com/mailman/listinfo/wp-edu <http://lists.automattic.com/mailman/listinfo/wp-edu>
> 
> 
> _______________________________________________
> wp-edu mailing list
> wp-edu at lists.automattic.com
> http://lists.automattic.com/mailman/listinfo/wp-edu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.automattic.com/pipermail/wp-edu/attachments/20150316/b4cd1de3/attachment-0001.html>