Yeah, CPU #2 is the top of my list of things to do.

Here’s the problem with that, though: that’s it for CPU slots. After that I’m looking at… another server? We just went into the black on this one. And, essentially, only for game threads? That doesn’t seem like a wise use of resources.

I tried searching the site, but I couldn’t find any mention of exactly what box you got. If it’s a recent Intel-based server, though, you can probably replace the CPU with one of the E53xx quad-cores (and eventually have quad-cores in both sockets). Not cheap, though — anywhere from $500 to $1200, each.

A couple other quick hits:
We use xcache as our php bytecode cacher. It’s pretty tight and is friendly with php5.

Run ‘vmstat 10 > /root/vmstat.txt &;tail -f /root/vmstat.txt’ during a game and watch the stats. This is the clearest way I know of to find your bottleneck. If it’s disk, you’ll see pagefile operations here. If it’s cpu, you’ll see it here.

(!– Dynamic Page Served (once) in 0.748 seconds –)
(!– Cached page served by WP-Cache –)

It looks like the wp-cache was actually off before and someone’s tinkering with the on button right now. ;-P

Also, keep in mind that, although the game thread is only 300-400 posts long, those posts don’t come at a regular rate throughout the game.

I’m not saying that this is definitely a disk IO issue (or even that I think that, in this specific situation, it’s likely to be the cause), I’m just saying that “Oh we have query caching on” doesn’t mean you shouldn’t still inspect the IO situation.

A methodical approach of definitively ruling out various potential bottlenecks seems like the best way to approach things.

Derek, I would definitely take a look at installing a byte code cache like the aforementioned Zend Optimizer (which I believe is still free) or the eAccelerator that is mentioned in the WP optimization article. These things really can work wonders for PHP execution times.

Dynamic Page Served (once) in 0.753 seconds

A game thread is a few hundred posts long, made over a few hours. Invalidating the MySQL cache ~300 times in a few hours isn’t going to thrash anything. That’s an average of two or three invalidations a minute. I’d be shocked if MySQL is thrashing the HD.

Anyways… I just pulled this from the bottom of the source HTML of the page right now, when it’s pretty calm and we’re just BS’ing about server stuff –

Even when there’s no game thread it takes WP almost a second just to generate the page it sends you. 0.75 sec/page * 2,000 people slamming refresh = Melted plastic.

WordPress just blows. The only way to make it work under such heavy load would be to throw a lot of resources at it via replication and load balancing — put MySQL on it’s own box and then replicate the webserver. Or chuck WordPress…

Does wp-cache arrange to serve static files, or just produce a highly optimized database result? I would hope it’d be serving static files, because no amount of optimization would save the server when Felix comes out in the first inning. And in that case the results should be plainly apparent in the apache logs — you’ll see lots and lots of the cached .html files being pushed over the wire.

I have, in relative secrecy, experimented with Movable Type and other options, and the results were pretty awful.

What flavor of awful? I know MT a lot better than WordPress, and USSM isn’t doing anything that MT can’t handle. Switching over would still be a pain, though, so if wp-cache can arrange to serve static files with only ~300 or so database updates per game thread, that should be a big win.

#25: Query caching doesn’t fix everything, when the table changes (like when a comment is posted) the cached queries for that table are invalidated. On a game thread, this happens a lot. MySQL could very well still be thrashing the disk leaving a lot of processes in an IOWAIT state. This is by far the #1 most common performance bottleneck on a Apache/MySQL/PHP architecture.

2GB of RAM and a decent CPU should get you into 7 figures in terms of hits/day unless something really complicated is going on with the pages. Disabling swap is…not wise. FreeBSD and other unix systems are a lot better at memory management than windows is. All the memory being in use but none of the swap being in use is an indication of efficient memory management. Unix doesn’t like to waste RAM, so it finds something useful to put in there and doesn’t take it out until it needs to.

Stepping back for a second though, the game threads are pretty much a doomsday scenario in terms of scalability: constantly changing data with teeming throngs pounding the refresh button. If I was going to look at something from a software standpoint, I would look into finding or developing a real simple (but high performance) solution that would handle the game thread discussions exclusively and leave the rest of the site in Wordpress’ capable hands. This could probably even be developed in the context of a WP plugin as something that didn’t even touch MySQL for those discussions.

Looking at the pre-spike table, we see that MySQL seems to be using about 5% of the CPU, and it wouldn’t appear to need much more than that since you still have some available CPU left at that point. Then in the next table, MySQL is nowhere to be found, which would naively seem like a big problem to me.

I would suspect that if MySQL wants 5% of the CPU when the server isn’t choking, it probably still wants at least 5% of the CPU when the server is choking, but isn’t getting it. If it really is happy using less than 0.3% of the CPU during the slowdown, then even if you can get it to run 100 times faster through better caching or whatever, then you’re only going to open up enough space for one more httpd process.

Basically, I guess I’d say that if MySQL is the bottleneck, you might want to look into getting it more CPU time by giving it a higher priority or something. If it isn’t the bottleneck, then it seems like you won’t really gain anything by spending any extra time optimizing the MySQL side of things.

Here’s the problem with that, though: that’s it for CPU slots. After that I’m looking at… another server? We just went into the black on this one. And, essentially, only for game threads? That doesn’t seem like a wise use of resources.