Generalized USSM hosting/server/hair-pulling post
Skip on down if you’re not at all interested in this stuff. Which probably limits this to me.
If you were around yesterday, you saw the server pretty much choke and die twice on the game threads, when Felix hit his grand slam and when he got injured. When McLaren and Bavasi got fired, you got to see some related fireworks including a perfect storm that (as far as I can tell) resulted in a bad redirect page being cached and served to everyone who went to www.ussmariner.com until I went through and peeled that onion.
Anyway, here’s the scoop:
- the Digital Forest folks have been really good to host our box
- I’d planned to put a second processor in there
- but I don’t think that would do it
We have a fairly high-traffic site that has to endure huge, essentially random usage spikes but most of the time sits around twiddling its thumb.
I would also like to get out of the business of supporting the site in the sense of doing things like upgrading Apache versions. I’d much rather be writing snarky articles or many-thousand word discussions with Dave arguing about which Mariner utility man was the best.
And we don’t have a ton of money, obviously. But our experience with shared hosting is that we get kicked off. I don’t know where to go here, if that involves buying another, much bigger box and recruiting someone to run the tech side, or what… you see the dilemma.
So my question to you, since you’ve stuck it out this far: how do we get out of this mess? What’re best practices for something like this?
Now updating with answers! Yayy
Comments
57 Responses to “Generalized USSM hosting/server/hair-pulling post”
Leave a Reply
You must be logged in to post a comment.
USSM uses and recommends
My brother in law is a real wizz with this stuff. I will have him read this and I will ask him to email you a solution.
Would something like this (Amazon’s Cloud Computing) be a possible solution?
Amazon Web Services
I’m of no help with things like this otherwise, sadly.
I’ve a few questions:
What sort of bandwidth needs do you have?
Do you prefer to use your own hardware?
What are the specs of your current hardware?
Is Apache the way you want to go, or are you open to others?
Do you want a managed service (it looks to me like you are open to this)?
I work for AT&T Business and some of my clients do web hosting. I’ll put some feelers out and see what I can find.
Also, have you considered incorporating as a non-profit?
Well, at this point you’re looking at some sort of hosted solution, more something for business than for a personal blog.
Adding another processor won’t help, I don’t think. It’ll speed things along, but the problem really is that the system can only handle so much traffic. You’re probably looking at getting more than one server and a load balancer. Or, you’re looking at some sort of failover solution for moments like OMGWTFFELIX.
Do OMGWTFFELIX moments happen a lot, though? Is it worth all this effort for incidents that aren’t daily?
I hate to say it, honestly, but I think your solution in the long term is to monetize this place and use that to pay for a long-term business solution. I know you don’t want all these ads — and neither do I! — but if you’re generating $3-4K in ad revenue a year you’re going to be in a good position to ask for a decent hosted system with the balancing and failover you’ll need for 90% of the OMGWTFFELIX moments.
$4K/year is $333/month. I’m not an online ad guru (I work at a non-profit, after all) but I think you can get $333/month out of advertisers.
I was about to suggest AWS as well, since I, you know, work for it. The pro- argument is that it handles your exact use-case (it’s metered based on what you use, so if your site is twiddling it’s thumbs 99% of the time, you’re just not racking up any costs). The con- is that there’s pretty much no way to set it up without either being a full-time developer yourself or having one handy.
Sorry. Twiddling its thumbs….
The con- is that there’s pretty much no way to set it up without either being a full-time developer yourself or having one handy.
OTOH, there’s got to be devs who read this who’ll work for beer and pizza setting this up. And when you think about it, “Developed solution using AWS to help leading baseball blog maintain uptime during extremely high demand” is a nice thing to throw on your resume.
I’ve suggested that before. Since you’d only have to be registered in one state the regulatory burden wouldn’t be that high. And then you could do things like accept donated hardware and issue receipts for it (the IRS is remarkably laid-back with regard to charitable tax receipt rules – trust me on this one). You’d effectively be having the government pay people for giving you stuff.
I’m employed by a Canadian charity with a US division (soon to be a US charity in its own right), and I’m the guy who manages the donation database and issues all the receipts. The US rules make running a charity really quite easy. I suspect you’d fall under the educational part of non-profit law (because you do actual research and disseminate it online).
Some of your readers could probably swing corporate donations of hardware or services if there were an official charitable angle (companies like to publicise how many donations they make – you’d count).
Speaking solely as a user of the site:
The vast majority of the readership of the blog are pretty understanding when the site has difficulties during spikes due to something crazy happening (like say, some awesome pitcher of ours being the first AL pitcher in 30+ years to hit a grand slam).
So I wouldn’t worry about it too much except inasmuch as it increases your own frustration; people can wait a few minutes to post WOO FELIX if they have to.
In most cases, you need to issue a diploma/degree to be considered educational. I am in IT sales, and I know we are required by vendors to show this proof before we can sell at an educational discount.
I will say the other points are valid, though. You really should consider becoming a non-profit. The benefits are pretty large, and easy to get.
As far as the hardware goes, you could be talking about a decent sized expense to achieve the results you desire. I just don’t think it is an issue often enough to justify that cost.
If people want to moan and complain about the site getting owned after something like Felix hitting a grand slam, which will likely never happen again, well, give them a full refund.
I don’t know anything about servers or hosting, but I do know about hair-pulling, so I just wanted to add a quick word here – actually, I guess I’m seconding what Jeff said:
When something like the Felix slam happens, or someone gets canned, I have no problem waiting for the site to come back up. It doesn’t kill me to stop refreshing and simply come back in awhile to read up on stuff. After all, what I enjoy most about this site are the info-packed posts and debates; if USSM isn’t structured to handle “breaking news!!” spikes, is that really so bad? I’ll bet many of the readers feel the same way.
That said, if you want to put out a donation call especially to resolve this at least temporarily – I’d be among the first to donate. What does a beer cost at Safeco? $8? Is reading this site not worth an $8 donation from everyone?
Actually I’ll send my $8 right now, dammit. Who’ll match me?
Without going too far above my paygrade here, I know my dad’s firm (a local Seattle business) tried to contact you guys about hosting the site for free, and he never heard back. I’m not sure if you guys just weren’t interested, or if you never received that email, or what, but surely that would help you guys to focus on the fun stuff and not have to worry about bandwidth.
I don’t think you can solve it without spending money: more bandwidth, and a dedicated machine. Granted, that’s really expensive, and not really an option. It seems like the only answer at this point really is A) accepting it, and most of the people who read USSM totally do not mind the occasional spikes and flaming burnination, or B) find a way to raise the money to buy a dedicated server and more bandwidth. I think you might be out of other options.
I think bandwidth and budget are going to decide this, aren’t they? Unless you actually like admin tasks, I’ve never really grokked the urge to own your own server, but I’m not very bright, either.
Companies like pair offer plans where you lease a box from them, and they set it up just like it was one of their FreeBSD/Apache/etc. boxes. They start at about $250/month for a terabyte/month traffic. Add roughly $150 to $200 a month for each additional terabyte of traffic.
So just how much bandwidth are you sucking, here?
Sorry, see http://www.pair.com/services/dedicated/
There are a ton of companies that offer this, of course.
Another option would simply be to have this blog used for your articles and only your articles, and partner up with Lookout Landing to handle the game threads. The amount of traffic that Jeff gets over there during the threads, not to mention all the pictures and animated gifs that get posted, and the mass spam of “OMGWWTFELIX!” that happened yesterday, is handled decently well by the SB network.
Sure, sometimes even the SBN can take a dump, but it’s track record has held up pretty well. If you can spare yourself those massive heart attack moments that happen in game and spark a ton of shock requiring us panicked animals to come refreshing the site for information and post our own spam on the subject, then that would help a bit. Obviously moments like Bavasi and McLaren being fired would still be a concern, but at least the crashes would be farther apart (one would hope).
[ot]
[ot]
Another option would simply be to have this blog used for your articles and only your articles, and partner up with Lookout Landing to handle the game threads.
It’s not just game threads, it also comes up regularly for dramatic announcements that aren’t predictably timed. Firing GMs, managers suddenly resigning, foolish big trades for staff aces, callups of hot prospects, signings of disastrous free agent pitchers, you name it.
You could really reduce traffic for big threads by fetching new comments with AJAX rather than reloading the whole thread. It looks like there’s a ton of AJAX plugins for WordPress.
I should sign in and comment more often. Just noticed the “Add my light to the glowing firmament of discussion” thingy. How funny.
Seriously though, make a donation folks, it’s easy & and you get a warm fuzzy….hey…is that what the “WF” in WFB stands for?
Why not a virtual private server (vps)? slicehost.com is an excellent and relatively inexpensive service, loved by its users (which includes me). If you want someone else to do the admin, railsplayground.com is another vps that for an additional $15/month will …”cover most custom software installs, OS kernal upgrades, and general VPS maintainance.”
If the bandwidth described in their packages is insufficient, I’m sure they’ll give you a custom package. VPS is much cheaper than colocation, while providing similar speed and reliability.
If you’d like, you can contact me for more detailed help (I actually used to work at Expedia)- I’ve got quite a bit of operations experience designing and building/running extremely large sites, and something like this should still be eminently solveable even for a smaller, not-for-profit site.
Some suggestions:
1) Unless you expect the traffic to grow larger and this problem to become more common… let the spikes happen. We all can handle it, this isn’t a for-pay site and we aren’t expecting 99.999% uptime.
2) If you are bandwidth cramped, consider adjusting how you serve content. First, make sure you are caching all static content including images, stylesheets, etc. Second, if you aren’t using http compression, you should- it really helps with heavily compressible content like html documents. Lasty, in game threads, only serve the last N comments with a link to “Show all comments”. If we’re posting as we’re watching, we don’t need the full history of the game “conversation” each time we refresh. Yesterday’s game thread was a light one, with 120 comments at 80K for the page; for every refresh a user does, that’s up to 80K more traffic. What if it was ~10K for the last 20 comments? Your bandwidth is suddenly ~1/8th what it was before.
3) If you are processor cramped, that’s harder to fix. Make sure that you’re using caching where possible to not have to re-generate the page from the database on each refresh. Obviously suggestion 2 above requires more CPU than you have currently, but if you are being hosted and then “cut-off”, it sounds more like you’re exceeding a bandwidth cap than the site is unable to handle serving the pages.
4) Consider co-locating in multiple places; why not take advantage of multiple people offering to host the site? This is more technically sophisticated to implement, but has two advantages: you can scale faster and cheaper since no one place has to have multiple servers, etc, and the $/GB is usually higher the more you use in one location. Plus, if any server crashes/goes down, it can quickly and automatically be routed around and we might not even notice a problem.
The last one would be the most scaleable; you’d have 3-4 trusted USSM friends “hosting” the site, and when we the teeming masses hit the site we’d actually be hitting one of those 3-4 servers; those servers would in turn talk to your current USSM site to get the latest copy of content to then serve to their users. When we refresh frantically, all the bandwidth and CPU is to those sites; 1/4 of us hammering one site, 1/4 hammering another, etc. If you find traffic keeps growing, you just find another trusted hosting partner and suddenly you’ve increased your capacity by say 20% without any refactoring or work.
Basically, it’d be using 3-4 trusted USSM hosting partners to act as somewhat modified reverse proxies for USSM; they’d cache unchanged content and pass along posts, drastically reducing the amount of usage on your current or “main” USSM server.
Seriously though, make a donation folks, it’s easy & and you get a warm fuzzy….hey…is that what the “WF” in WFB stands for?
Why, yes, it is. It’s the feeling you get from riding on a pony.
Derek, check your email……I hope I have a solution for you.
I have a small to mid-sized website and I’m using very little of the overall resources.
I’m not interested in money, and there are no strings attached……other than maybe adding a pony or a few bees to the top of the page.
If the current setup can’t handle only these rare spikes, that’s acceptable, so no big deal if it is unavailable. It is the work that it takes Derek to get it fixed that sucks. So, can you set up something to prevent crashes by cutting off the high volume spikes? I am not a techie, but I’m thinking of either a “circuit breaker” that softly takes the site down and restarts it automatically, or alternatively a flow limiter that restricts input.
In the meantime, I am not going to post nor refresh for a bit after the next likely-to-be-site-swamping play happens.
I don’t think buying a dedicated server and colocating it (what you are doing now) is the best solution for you because, as you say, you end up having to buy hardware to handle the traffic spikes that is way overkill 95% of the time.
Traditional shared hosting (Dreamhost, etc.) is also not going to cut it. The first time you get hammered you’re going to see the lovely “This client has exceeded their (whatever) quota.”
I think the solution is to explore a third type of hosting which is burstable virtual servers. The idea is, hosting company runs monster server (8-32 cores of CPU, 32gb RAM, etc. etc.) and uses virtualization software to divide the server into virtual machines for each client. Every client gets full root access on their slice of the server; you can’t see or touch or feel the other clients on the box.
Typically, you sign up for a guaranteed minimum amount of CPU resources, memory, etc. but as long as there are surplus system resources available, you can burst up to the full capacity of the hardware.
Basically, you would be trading the up-front cost of a new server for an ongoing monthly cost which may or may not be feasible for you.
If you are interested, here are some options for hosts that I have direct professional experience with and would recommend. (No relationship other than as a satisfied customer.)
Joyent Accelerator (OS would be Solaris which takes a little getting used to coming from Linux.)
Slicehost (Linux, several distro options.)
MediaTemple (dv)
Be up front with the host about traffic levels, including the bursts, and see what they recommend in terms of minimum resources. Make sure you find out what happens if you go over your allocated resources. Good hosts will have soft caps and work with you to adjust your setup to handle it. Bad hosts will cut off your site and/or slam you with overage charges.
Feel free to toss me an e-mail at my profile address if you have any questions about my specific experiences with any of the above Derek. Good luck!
As a church? an institute of learning?
How wedded are you to Wordpress? This isn’t anything more than an educated guess, but I suspect that it may be part of your problems. Obviously you can’t solve the “Bavasi is fired and the whole world shows up to post a comment” surges without hardware (and here a virtual host that can smoothly allocate resources as you need them is the best bet) but I suspect that the typical game thread mini-crashes are more the fault of Wordpress, which doesn’t cache optimally and was never really designed for this kind of thing, than your base hardware infrastructure.
As a church? an institute of learning?
Considering they’re keeping the flame alive for the long-suffering fans of the worst team in baseball, I’d call this a charity.
I’m kinda new here, and am not intimately familiar with things, but I thought I would throw out some ideas.
It seems like the game threads might be better served using some kind of chat room, rather than posting to the wordpress comments. I don’t have any specific suggestions for software (perhaps someone else may?).
Have you considered going to a PhPbb style (or similar) message board to deal with all of the ongoing conversation threads? This seems it would be more favorable to having ongoing conversations, and would reduce the amount of off topic posting from carry overs from previous discussions. I’ve seen this done in other places and it seems to work pretty well. Instead of having a comments box, you would have a link to a message board post where folks could leave their comments?
Either one or both of these things would take considerable stress off of the blog itself,and move the dynamic conversations to platforms more suited to the task.
Of course it does mean more administration, at least for the setup. I know from experience that managing a message board isn’t a whole lot of fun, but if you can find some trusted lieutenants to run the boards, it would free up your time to write more (the blog itself would require much less administration).
Anyway, just some thoughts, hope it helps. =)
What sort of bandwidth needs do you have?
Fairly ridiculous. Looking at the server logs, a good day shows as ~2,000,000 kb and a bad day’s 2-2.5 that, throttled by server outages… but that can’t be right. Can it?
Do you prefer to use your own hardware?
Don’t care.
What are the specs of your current hardware?
I’d have to go look up the spec sheet.
Is Apache the way you want to go, or are you open to others?
I don’t really care, but I am partial to Apache.
Do you want a managed service (it looks to me like you are open to this)?
I would love a solution that doesn’t require me to do anything related to troubleshooting or… really, I’d like to get out of the optimization/tech support business.
Also, have you considered incorporating as a non-profit?
No.
Do OMGWTFFELIX moments happen a lot, though? Is it worth all this effort for incidents that aren’t daily?
I struggle with that. In terms of staying up for Bavasi/McLaren-type news, that’d be good — and the server did pretty well. But for Felix-type stuff, we can’t recover for hours… and that’s just not right.
I think your solution in the long term is to monetize this place and use that to pay for a long-term business solution.
If there was a way to run that side of this by outsourcing, I’d be all for it, but I want to do online ad sales even less than I want to do tech support.
I know my dad’s firm (a local Seattle business) tried to contact you guys about hosting the site for free, and he never heard back.
That’s entirely possible, and please, take our word — it’s not because we want to be rude. Our record on answering email is not stellar. Our inbox is generally filled with support requests for failed password resets, link exchange begging, PR spam, random questions from people who own M’s in fantasy leagues… it’s crazy.
Unless you actually like admin tasks, I’ve never really grokked the urge to own your own server,
At the time, the cost to get a server that would handle our load against buying one meant that we broke even on a server in ~8mo, so we solicited donations and bought one. It was the cheapest choice, and spending reader money for the first time I was extremely reluctant to go with a hosting option that would require us to make repeat requests.
I’m… I’m a little resigned to that now.
Another option would simply be to have this blog used for your articles and only your articles, and partner up with Lookout Landing to handle the game threads.
How would that work, though? Just post a “here’s your game thread” with link to LL? That’s kind of amusing.
You could really reduce traffic for big threads by fetching new comments with AJAX rather than reloading the whole thread. It looks like there’s a ton of AJAX plugins for WordPress.
An interesting thought. I tried this once, and it looked slick but was a total disaster on the server side: it just killed, absolutely destroyed, the server.
It may be worth trying again now that the plug ins have matured.
On hincandenza’s comments (Expedia wooo!):
We’re processor cramped more than anything else. We peg the CPU and stall: load goes through the roof and then nothing happens.
On colo in multiple places: That’s a great idea, but I have no idea how I’d set that up, and USSM Labs doesn’t have an IT department.
So, can you set up something to prevent crashes by cutting off the high volume spikes? I am not a techie, but I’m thinking of either a “circuit breaker” that softly takes the site down and restarts it automatically, or alternatively a flow limiter that restricts input.
So– not really, no.
For instance, in yesterday’s thread, at one point I tried to shut down the webserver, and nothing happened… nothing. I had to go through and kill threads off manually, and that… that was painful.
I think the solution is to explore a third type of hosting which is burstable virtual servers.
I like. Except I really do want to get out of the support side.
How wedded are you to Wordpress? This isn’t anything more than an educated guess, but I suspect that it may be part of your problems.
We’re running two levels of cache: people who aren’t logged in get served a static HTML page, while logged in people if possible get a cached version of the page.
I’m not sure that Wordpress is the problem: we’re not killing the database, I feel like it’s a matter of not having enough CPU to serve enough pages fast enough.
Have you considered going to a PhPbb style (or similar) message board to deal with all of the ongoing conversation threads?
We’ve talked about something like this. The problem there is — as you note — the management overhead. It’s a huge cost sink without much of a visible return.
As I understand it, you own the machine that’s hosting USSM, but are being gifted rack space and b/w from digital.forest. However, you’re tipping over during heavy traffic either from the server being unable to handle the load of generating pages, or the bandwidth simply being too much. I’m still not clear on what the source of the problem is- does the server tip over, and require you to log in to reboot/restart it?
Scotje and joser are right; an instantly scaleable solution is the way to go (one where the spikes are easily absorbed, and the cost of absorbing them isn’t punitive in the overall hosting costs), along with considering a migration from Wordpress to another platform that is more efficient for higher volume/frequent change content, such as game threads.
This seems complicated enough that maybe you should just have it be a topic at the next meetup, where you can directly triage with some of the brighter tech minds of the USSM community.
Shoot, the formatting on my last post was wack (is there a preview button and I just don’t see it?)
Actually, the simplest version would be “Find 3-4 trusted people, and have them run Apache reverse proxies to USSM”. Then you’d simply set up the DNS entry for ussmariner.com to be a round robin load balancing to the 3-4 simple reverse proxy hosts. Your current server wouldn’t have any changes made to it; it would simply have most of the bandwidth and page serving offloaded to the trusted hosts.
End result: The USSM server appears to have only 3-4 readers, who regularly refresh requested parts of the site every, say, 15 seconds (main page, active game threads, etc)- no matter how much traffic you get. Such a volume your server, running Wordpress, should easily be able to handle. Those reverse proxies however each handle 1/4 of the traffic, even if it spikes.
You could consider Wordpress.com VIP hosting. The two big drawbacks would be that tags get linked off USSM, and you may not be able to run all your plugins (they don’t allow plugins that modify the database). Oh, and probably all the user accounts would get toasted in the migration.
Another thing to look at if you haven’t already done so – CPU spikes are not *always* an indicator that you are actually CPU bound. Have you also looked at your available RAM at the time of the spikes, and what does your disk subsystem look like? Running out of RAM and needing to fall back to swap can be known to spike your CPU, and a disk subsystem that can’t keep up with your static file reads and/or DB requests can also peg your CPU as things spin waiting for the data to be retrieved from disk…
I can only imagine, totally understandable. I sent you another email about it a couple hours ago. It had “hosting USS Mariner” in the subject line. Is there a better way to get in touch with you for this type of thing?
Appreciate all the authors write has been the highlight until Monday. Then……..
The best and worst of 2008 season within 15 minutes.
A few people are saying “migrate away from Wordpress,” but I really have to disagree.
Right now, there are three options for blogging platforms:
Wordpress
Movable Type
Expression Engine
Movable Type has improved, but you’re either generating static pages or you’re running in PHP just like Wordpress. It’s not an improvement over Wordpress.
Expression Engine is getting to be pretty solid, but I can’t think of any major sites running it as a blog, only some personal sites. It’s really more CMS than blog.
If the problem really is CPU-related, then the solution is probably going to be limiting the amount of processing needed. And that probably means limiting comments to “last 50″ or something similar on the 51+ length threads. I’m not sure cloud computing would help, really, not without actually moving the DB into the cloud itself. Or, at least moving frequently-used files into the cloud to get them off the server, e.g. images.
I had sent a couple emails as well. i have a dual Xeon box with 4GB RAM and 2 x250GB HDD….one of which is basically just sitting there unused…..My server would have no problem hosting my site and the USSM…..and I’m not looking for money or specioal conditions….except maybe a big picture of Bloomquist at the top of the page.
If you really want to get out of the administration business then look into the wordpress.com paid hosting accounts. They host several high traffic blogs and seem open and transparent in their business practices.
I want to compliment the several people who have volunteered to share their servers gratis. One caveat is that Derek and his partners wouldn’t want to risk losing control of the site. All we need is for someone to go ballistic over an opinion that Dave or Derek posts, pull their resources without warning and cripple the site, if not totally destroy USSM.
But if there is a way to safely implement something that allows use of those free resources without risking sabotage, it has promise. The colocating scheme that hincandenza mentions in #46 and earlier looks promising to me. If there were 4 (or more, if possible)proxy hosts in the scheme and one crashed or its owner wanted to pull out of USSM, there would be lots of other capacity.
I reiterate that I am not currently a tech guy, though I was one in my earlier career back when dinosaurs roamed the earth.
Well, I read the site daily, but rarely post…..I don’t agree with everything thats said…..but thats what makes USSM cool…its a different perspective than most sites, and its an educated perspective.
As to “losing control of the site”….that wouldn’t be an issue. I have been thinking of something I could do with the rest of my server resources, and thought this would be a worthwhile contribution.
If you’re interested, hit me up, if not just let me know, either way its cool.
Derek,
Your site is probably similar in traffic to Hardball Times, and I think they have a dedicated server (which I guess they lease, along with all the backups, upgrades, and whatever else they get out of the deal). I’ve never seen their site experience a problem.
Might be worth talking to studes about what he’s doing…
A few people are saying “migrate away from Wordpress,” but I really have to disagree.
Right now, there are three options for blogging platforms:
The trouble is, this is no longer really a blog. Sure, it’s structured like one, and we (including the authors) still think of it as mostly Dave and Derek writing whatever’s on their mind, like a blog, but from a functional standpoint that’s no longer the case. Game threads are much more like any active BB/community site, and the site as a whole falls into a more traditional web-publishing content-management mold. In other words, limiting yourself to looking at alternative blog software misses the point. A site that has thousands of readers and gets hundreds of updates in the course of a three-hour span isn’t a typical blog. You don’t see Slashdot trying to run on Wordpress, no matter how much hardware they have at their disposal.
If the problem really is CPU-related, then the solution is probably going to be limiting the amount of processing needed. And that probably means limiting comments to “last 50″ or something similar on the 51+ length threads.
Right, and traditional CMS/web-publishing systems have support for that kind of “paged” view built in (for their commenting systems and whatever else). It’s been a while since I looked at the state of the art in free software in this category, but I know a couple of years ago Joomla and Drupal were the standard-setters, with the *Nuke (including DotNetNuke on the Windows side) variants trying to unseat them. But I really have no idea now.