Major outage 29th October

[jsmcortina]

Yesterday the whole MSEXTRA site was down for close to 24 hrs due to a problem at our web host outside of our control.

The Megasquirt team are very unhappy with this situation. If we do not receive a suitable explanation we may have to move to a different hosting company.

James

[ErnieJones]

I think what happened was that the MySQL server was seeing some really heavy usage and started dropping connections. I saw several 'too many users' errors and these were coming directly from the MySQL server - not the web server.

Usually what happens with shared hosting is that there can be anywhere up to 1,000 sites on one web server, and, depending on the hosting service, they may even run the dB server on the same machine Usually it's a dedicated machine, if the host is worth anything, but it too will be shared with any number of sites. If one of the 1,000 websites (or more) get a big traffic spike then the database machine gets hit hard. MySql has a max number of connections/users setting and so, if there are too many, it starts shedding requests to prevent an overload. This is what I was seeing yesterday, the web server side was working but the dB was shedding traffic due to too many connections.

I run this same forum software on virtualized servers over here on a custom network and I have long experience with shared & dedicated hosting. I always ended up coming back to doing it myself....

[ErnieJones]

BTW, and I'm sure you probably know this, but when I saw the dB server go down my first thought was, I sure hope you have a recent backup. It's essential to back up both the dB and the website files on a daily basis and transfer them to yourself so that when it does get kaffed - and it will - you can roll the clock back and only lose the last 24 hours of posts (or less, depending on how often you back up).

I've seen a lot of sites, with YEARS of important info get wiped out on the dB side with no backup. It's all lost. All the years of posts, the good information, the pictures posted, the forum modifications - all of it.

This was my concern when I saw the dB errors. That there was a chance it was all gone.....

[jsmcortina]

The host claims to have something like 9,000,000 domains on them. Most will have been down yesterday.
Although there were some mysql issues, our front page is (currently) plain HTML and it was not working either. Admin access via cPanel was ironically ok.

James

[subwoofer]

Tapatalk support appears to be down since the incident.

[jsmcortina]

Tapatalk support appears to be down since the incident.

I'll get Ken to take a look as he installed support for it.

James

[DaveEFI]

Was down again earlier this morning here in the UK.

[jsmcortina]

Was down again earlier this morning here in the UK.

Yes, I had to rebuild one of the database tables. I'm not sure if that's related or just an annoying co-incidence.

James

[jsmcortina]

Tapatalk support appears to be down since the incident.

Ken says that it is working ok for him.

James

[subwoofer]

Works now, did not work an hour ago.

[Rick Finsta]

The note that the site is down is still up on the main page.