Major outage 29th October
Moderators: jsmcortina, muythaibxr
-
- Site Admin
- Posts: 39620
- Joined: Mon May 03, 2004 1:34 am
- Location: Birmingham, UK
- Contact:
Major outage 29th October
Yesterday the whole MSEXTRA site was down for close to 24 hrs due to a problem at our web host outside of our control.
The Megasquirt team are very unhappy with this situation. If we do not receive a suitable explanation we may have to move to a different hosting company.
James
The Megasquirt team are very unhappy with this situation. If we do not receive a suitable explanation we may have to move to a different hosting company.
James
I can repair or upgrade Megasquirts in UK. http://www.jamesmurrayengineering.co.uk
My Success story: http://www.msextra.com/forums/viewtopic ... 04&t=34277
MSEXTRA documentation at: http://www.msextra.com/doc/index.html
New users, please read the "Forum Help Page".
My Success story: http://www.msextra.com/forums/viewtopic ... 04&t=34277
MSEXTRA documentation at: http://www.msextra.com/doc/index.html
New users, please read the "Forum Help Page".
-
- Master MS/Extra'er
- Posts: 385
- Joined: Thu Jul 17, 2014 4:44 pm
Re: Major outage 29th October
I think what happened was that the MySQL server was seeing some really heavy usage and started dropping connections. I saw several 'too many users' errors and these were coming directly from the MySQL server - not the web server.
Usually what happens with shared hosting is that there can be anywhere up to 1,000 sites on one web server, and, depending on the hosting service, they may even run the dB server on the same machine Usually it's a dedicated machine, if the host is worth anything, but it too will be shared with any number of sites. If one of the 1,000 websites (or more) get a big traffic spike then the database machine gets hit hard. MySql has a max number of connections/users setting and so, if there are too many, it starts shedding requests to prevent an overload. This is what I was seeing yesterday, the web server side was working but the dB was shedding traffic due to too many connections.
I run this same forum software on virtualized servers over here on a custom network and I have long experience with shared & dedicated hosting. I always ended up coming back to doing it myself....
Usually what happens with shared hosting is that there can be anywhere up to 1,000 sites on one web server, and, depending on the hosting service, they may even run the dB server on the same machine Usually it's a dedicated machine, if the host is worth anything, but it too will be shared with any number of sites. If one of the 1,000 websites (or more) get a big traffic spike then the database machine gets hit hard. MySql has a max number of connections/users setting and so, if there are too many, it starts shedding requests to prevent an overload. This is what I was seeing yesterday, the web server side was working but the dB was shedding traffic due to too many connections.
I run this same forum software on virtualized servers over here on a custom network and I have long experience with shared & dedicated hosting. I always ended up coming back to doing it myself....
Thanks you;
Ernie
Ernie
-
- Master MS/Extra'er
- Posts: 385
- Joined: Thu Jul 17, 2014 4:44 pm
Re: Major outage 29th October
BTW, and I'm sure you probably know this, but when I saw the dB server go down my first thought was, I sure hope you have a recent backup. It's essential to back up both the dB and the website files on a daily basis and transfer them to yourself so that when it does get kaffed - and it will - you can roll the clock back and only lose the last 24 hours of posts (or less, depending on how often you back up).
I've seen a lot of sites, with YEARS of important info get wiped out on the dB side with no backup. It's all lost. All the years of posts, the good information, the pictures posted, the forum modifications - all of it.
This was my concern when I saw the dB errors. That there was a chance it was all gone.....
I've seen a lot of sites, with YEARS of important info get wiped out on the dB side with no backup. It's all lost. All the years of posts, the good information, the pictures posted, the forum modifications - all of it.
This was my concern when I saw the dB errors. That there was a chance it was all gone.....
Thanks you;
Ernie
Ernie
-
- Site Admin
- Posts: 39620
- Joined: Mon May 03, 2004 1:34 am
- Location: Birmingham, UK
- Contact:
Re: Major outage 29th October
The host claims to have something like 9,000,000 domains on them. Most will have been down yesterday.
Although there were some mysql issues, our front page is (currently) plain HTML and it was not working either. Admin access via cPanel was ironically ok.
James
Although there were some mysql issues, our front page is (currently) plain HTML and it was not working either. Admin access via cPanel was ironically ok.
James
I can repair or upgrade Megasquirts in UK. http://www.jamesmurrayengineering.co.uk
My Success story: http://www.msextra.com/forums/viewtopic ... 04&t=34277
MSEXTRA documentation at: http://www.msextra.com/doc/index.html
New users, please read the "Forum Help Page".
My Success story: http://www.msextra.com/forums/viewtopic ... 04&t=34277
MSEXTRA documentation at: http://www.msextra.com/doc/index.html
New users, please read the "Forum Help Page".
-
- Super MS/Extra'er
- Posts: 884
- Joined: Sat Apr 30, 2011 12:34 pm
- Location: Sandefjord, Norway
Re: Major outage 29th October
Tapatalk support appears to be down since the incident.
Joachim
1974 Jensen-Healey
1990 VW Caravelle Syncro - running MS3+X
2014 Ford Fiesta EcoBoost
1974 Jensen-Healey
1990 VW Caravelle Syncro - running MS3+X
2014 Ford Fiesta EcoBoost
-
- Site Admin
- Posts: 39620
- Joined: Mon May 03, 2004 1:34 am
- Location: Birmingham, UK
- Contact:
Re: Major outage 29th October
I'll get Ken to take a look as he installed support for it.subwoofer wrote:Tapatalk support appears to be down since the incident.
James
I can repair or upgrade Megasquirts in UK. http://www.jamesmurrayengineering.co.uk
My Success story: http://www.msextra.com/forums/viewtopic ... 04&t=34277
MSEXTRA documentation at: http://www.msextra.com/doc/index.html
New users, please read the "Forum Help Page".
My Success story: http://www.msextra.com/forums/viewtopic ... 04&t=34277
MSEXTRA documentation at: http://www.msextra.com/doc/index.html
New users, please read the "Forum Help Page".
Re: Major outage 29th October
Was down again earlier this morning here in the UK.
Rover SD1 3.5 EFI
MS2 V3
EDIS
Tech Edge O2
London UK.
MS2 V3
EDIS
Tech Edge O2
London UK.
-
- Site Admin
- Posts: 39620
- Joined: Mon May 03, 2004 1:34 am
- Location: Birmingham, UK
- Contact:
Re: Major outage 29th October
Yes, I had to rebuild one of the database tables. I'm not sure if that's related or just an annoying co-incidence.DaveEFI wrote:Was down again earlier this morning here in the UK.
James
I can repair or upgrade Megasquirts in UK. http://www.jamesmurrayengineering.co.uk
My Success story: http://www.msextra.com/forums/viewtopic ... 04&t=34277
MSEXTRA documentation at: http://www.msextra.com/doc/index.html
New users, please read the "Forum Help Page".
My Success story: http://www.msextra.com/forums/viewtopic ... 04&t=34277
MSEXTRA documentation at: http://www.msextra.com/doc/index.html
New users, please read the "Forum Help Page".
-
- Site Admin
- Posts: 39620
- Joined: Mon May 03, 2004 1:34 am
- Location: Birmingham, UK
- Contact:
Re: Major outage 29th October
Ken says that it is working ok for him.subwoofer wrote:Tapatalk support appears to be down since the incident.
James
I can repair or upgrade Megasquirts in UK. http://www.jamesmurrayengineering.co.uk
My Success story: http://www.msextra.com/forums/viewtopic ... 04&t=34277
MSEXTRA documentation at: http://www.msextra.com/doc/index.html
New users, please read the "Forum Help Page".
My Success story: http://www.msextra.com/forums/viewtopic ... 04&t=34277
MSEXTRA documentation at: http://www.msextra.com/doc/index.html
New users, please read the "Forum Help Page".
-
- Super MS/Extra'er
- Posts: 884
- Joined: Sat Apr 30, 2011 12:34 pm
- Location: Sandefjord, Norway
Re: Major outage 29th October
Works now, did not work an hour ago.
Joachim
1974 Jensen-Healey
1990 VW Caravelle Syncro - running MS3+X
2014 Ford Fiesta EcoBoost
1974 Jensen-Healey
1990 VW Caravelle Syncro - running MS3+X
2014 Ford Fiesta EcoBoost
-
- Master MS/Extra'er
- Posts: 548
- Joined: Fri Dec 10, 2010 4:41 pm
Re: Major outage 29th October
The note that the site is down is still up on the main page.