Chiffboard Slowdown 2010-12-09

You may have noticed that Chiffboard response has been sluggish for the past few days. We’re working with our hosting provider to identify and fix the source of the problem (usually an issue with the shared back-end mySQL server). We’ll keep you posted. Thanks!

MTGuru / Moderator

Thanks for all of your work.

You’re very welcome. An hour with tech support on Tuesday, a couple of hours today. And things do seem better now. Fingers crossed …

better for a bit…seems to be backsliding again

Yes, it’s worse today, very erratic. We’re working on it.

Makes for a real chore when you’re trying to go about the business of oppressing folks, let me tell you.

Sorry folks, still no joy. And apparently, our live tech support staff go home at 5 PM PST. I guess that’s when the Internet closes down for the night. :imp:

I just go to Wikileaks to see the non-censored version.

Folks, we’re sorry about the unplanned outage this evening, Thursday, from around 1800 to 2100 PST.

To address the recurring slowdown problem, our host decided to move our databases to a different server. Assurances that this would entail no Chiffboard downtime were apparently not so assured, or we would have alerted everyone in advance.

Anyway, we’re running on a new SQL server now, and we’ll keep an eye on the performance issues. Thanks for your patience!

Still slow.

^WHS^

It had seemed to improve just recently but since yesterday (23rd) it’s gone slow again.

Just sayin’.

Example times as of now (all other out-of-cache sites I visit responding promptly, so unlikely to be my connection):

From “View new posts” - click on Board index: 40 seconds to reach Board index.

From Board index to “Mark forums read”: 25 seconds.

From “View new posts” - click on topic: 20 seconds to begin switch to topic thread (then dependent on content, linked pictures, etc.)

Read PM messages - click on message: 50 seconds before display.

Note that I was logged in at all times. Wonder if that is a factor (tracking viewed messages).

(edited, after a 50-second wait, to answer NO). Same performance in Firefox and IE in Windows XP, 100-base-T, DSL)

Thanks for the feedback, everyone.

We’re pretty sure that the slowdown, when it occurs, has very little to do with any normal activities or operations of the board. With our current configuration, when things are working, they’re working just fine.

Without too much detail … The Chiffboard runs on 2 different servers: 1. A front-end http server for phpBB, implemented as a virtual private server (VPS) hosted on a shared CPU; and 2. A back-end mySQL server, also shared, which holds the databases. The board data is fairly large by now, around 3 GB.

There’s a correlation between the responsiveness of the board and CPU memory usage, which skyrockets to 2 or 3 times “normal” during slowdowns. Ordinarily, this might suggest a software memory leak. But re-booting our servers has no effect, so it’s unlikely that’s the root of the problem.

More likely, we think, the cause lies with our shared resources. That is, other users or sites sharing our CPUs and sucking up all available memory, cycles, and bandwidth. This would account for the seemingly random nature of the problem. If another site experiences excessive usage, or their software begins firing off rogue threads and processes that are left unchecked on the physical CPU, the Chiffboard grinds to a crawl. It’s like a DOS (denial of service) attack as a side effect of our VPS configuration.

Yesterday we moved the databases to a different back-end SQL server with smaller network latency (faster response).

Dale and I have been discussing other steps we hope to take after the holidays to address the problem, and we’ll keep you posted. Meanwhile, keep thinking happy, happy thoughts. :wink:

If the other sites are shopping sites, maybe the excess usage will go away after the holidays.

Or Julian Assange’s friends have mistaken us for Amazon or Bank of America :laughing: .

Our old friend Rich Lafferty has agreed to look into our situation. He’s on it, but may not be able to devote much time to it until after the holidays.

Thank you, Rich!

Rich is working the problem and has already found at least one major culprit in our databases. More later…

By the way, Rich tells me that the fixes will involve some downtime.

Remember you can always check the board status at

http://chiffandfipple.posterous.com/

Bookmark that site and visit it when the board goes down.

I’ll try to keep it updated.

Dale

Yes, the main problem turned out to be a severely bloated sessions table which is not auto-pruning as it should. Rich has fixed the immediate problem, and we’ll look into cleaning up other database issues. If the Chiffboard needs to be taken offline for any length of time, we’ll try to announce that in advance.

Many thanks to Rich, and thanks to everyone for your patience.

Rich is gonna have to change his name and move…

Once you work for this company, you can’t not work for this company.

And why would someone want to leave with this salary and these benefits? What? Leave show business?