[HyperDB] Any advantage to partitioning on a single database server?

[Jim McQuillan]

The number of files in a directory is exactly the reason I was thinking that
partitioning would be needed.  We're using MyISAM so that would be 3 files
per table, and if there are 50,000 blogs with 9 tables per blog that would
be 1,350,000 files in a single directory.

I'm no database or filesystem expert, but wouldn't that large number of
files be an issue?  Unfortunately it isn't easy way to "test" different
configurations before actually growing to that size, but we're looking to
set things up ahead of time to be prepared.

Thanks!
-Jim


On Thu, Aug 26, 2010 at 12:53 PM, Andy Skelton <skeltoac at gmail.com> wrote:

> Jim McQuillan wrote:
> > I've been doing more research into scaling a WordPress multi-site
> > installation, and I've been under the assumption that partitioning the
> > tables into multiple databases would be beneficial for a large
> installation
> > (hypothetically 50,000 blogs).
>
> At some point we decided to have multiple databases on a single
> machine. Maybe it had something to do with the number of files in a
> directory, or the number of hard drives in the machine; I never quite
> understood it. Maybe Barry or Donncha can explain.
>
> > I know that using multiple database server would definitely bring a
> > performance enhancement, but we're starting with one... so is it even
> worth
> > it to implement partitioning at all right now?
>
> There is some measurable overhead to partitioning. So if you had two
> blogs and you put them in separate partitions I assume there would be
> a tiny downside and no measurable upside.
>
> At some point (hopefully) you'll need to use partitioning due to
> hardware limitations. When that time comes, you may or may not benefit
> from any partitioning you did on the one first server. Your scheme
> might have to be thrown out or it might actually make it easier to
> migrate to a scaled-up system. The only guaranteed benefit is having
> the experience of working with the more complex configuration.
>
> Without further information I'd say proceed without partitioning.
> Don't try to do all your scaling in advance; it doesn't work. Run some
> experiments once you have real-world data and traffic.
>
> Andy
> _______________________________________________
> HyperDB mailing list
> HyperDB at lists.automattic.com
> http://lists.automattic.com/mailman/listinfo/hyperdb
>