October 1st, 2004

database update -- new machines, 64-bit, innodb

After a month and a half of vendor and motherboard hell, we now have six new 64-bit database machines on their way:

Two dual 64-bit Intel Xeons (EM64T)
Two dual 64-bit AMD Opteron 246s (2.0 Ghz)
Two dual 64-bit Intel Itanium2 (1.4Ghz 1.5MB cache)

That's a total of 12 new 64-bit processors... the first 12 we've had.

Why is this notable? Because now our user clusters can run InnoDB well. We already run InnoDB on our global machines, and it kicks ass, but we've stuck with MyISAM (which is lame, but has its benefits) on the user clusters because 32-bit machines don't give a single process enough memory to run InnoDB the way we would've liked.

See, InnoDB maintains all its own caches in-process, whereas MyISAM only caches indexes, and the kernel caches data pages. On a 4GB or 8GB box on MyISAM, you can get 2GB of indexed cached in-process (because you only have 3GB of user address space on a 32-bit machine) and the rest of the memory on the box is used by the kernel to cache data pages.

But with InnoDB you only have that 2GB for everything... data and indexed. Sure, the kernel can still help out, but they step on each other's toes.

Plus InnoDB uses twice as much disk space as MyISAM, so MyISAM won there. And MyISAM is easier to sysadmin. But MyISAM has table-level locking, which sucks, but can be mitigated by having multiple databases per machine. (and with memcached it's not a big deal... only sometimes) In a nutshell, MyISAM's worked well enough for us so far, and we've come to be able to deal with (or tolerate) its deficiencies so far. But that's not to say we've been happy with MyISAM.

Anyway, we've been holding out for 64-bit for awhile now, waiting to run InnoDB effectively. Soon we'll be able to.

Also, the new machines feature:

-- 8, 12, or 16 GB of memory
-- twice as many disks in the RAID 10 as we've done in any other machine (instead of RAID 10 on 4 disks we'll have RAID 10 on 8 disks or 10 disks). and that's in addition to the RAID 1 for the operating system and DB logs volume

So it's all very exciting. Can't wait to get all the users moved to this new hardware.