Jump to content
Sign in to follow this  
Martijn Meerts

Server downtime 31 August 2019

Recommended Posts

Martijn Meerts

Hi!

 

As some of you will have undoubtedly noticed, the forum was down for quite some time.

 

The issue was a power outage at the data center. The backup generators did kick in, but they too started going down again for unknown reasons. It took Amazon quite a while to recover all the server instances, but it seems they managed in the end.

 

While this kind of downtime is something we can protect against by adding a server to a 2nd availability zone, it would increase monthly cost quite a bit. We'd also need a load balancer to be set up in front of the 2 servers that makes sure traffic goes where it should. This would probably increase the cost of running the forum by a factor of 3. And while it would be nice to have the forum never be down, I don't believe it's worth the additional cost. Especially since this kind of major outage is very rare for Amazon.

 

Anyway, we seem to be up and running again, if something seems off, please let us know 🙂

 

  • Like 5

Share this post


Link to post
cteno4

Martijn,

 

thanks for the info! No we can easily go without for a little while, best to keep it simple and cheap. Overall amazon seems to have done well except for this bump. But hopefully they will learn from the outage so as not to repeat.

 

jeff

Share this post


Link to post
EdF

This might be the first time I've seen just 1 as fail, as designed, usually it cascades one way or another and the whole region takes a hit.  But I agree, keep it simple and cheap, a few hours, or even a day down is fine.  Is there a way for us to support the forum in an ongoing way, like patreon.

Share this post


Link to post
cteno4

We do a donation drive once and a while, it’s been a while, I’ll check with Martijn now the funds are.

 

thanks

 

jeff

Share this post


Link to post
Sheffie

I don’t think you need to spend more. Reliability has been fine. 

Share this post


Link to post
katoftw

Agreed. Not worth spending more for a once a year three hour outage.

Share this post


Link to post
Martijn Meerts
12 hours ago, EdF said:

This might be the first time I've seen just 1 as fail, as designed, usually it cascades one way or another and the whole region takes a hit.  But I agree, keep it simple and cheap, a few hours, or even a day down is fine.  Is there a way for us to support the forum in an ongoing way, like patreon.

 

Yeah, Amazon has been very reliable, and this outage where an entire data center is affected I've only seen once before in the 15 or so years I've been using Amazon's web services.

 

As for support, there's a thread about donations that includes a PayPal donations button, which should still work. We try not to really ask for donations, or link to that thread unless we're low on funds. At the moment, the software license has been recently paid for another 6 months, the servers are paid for on a monthly basis for now. Once I set up the new server (so we can more easily use https and more easily upgrade server software and such), I'll pay up front for a year which will drive cost down.

 

I think we currently still have around 150 USD in our donations account, so we should be fine for a while, especially since I tend to pay about half myself, and get the other half from donations. Donations are always welcome though, and if we start getting too many donations (PayPal has limits to how much donations an account can receive unless it's a registered non-profit), we'll find a way to give that money back through for example a small project party where participants do something like super-detailing a building, and then the winner(s) get some small prices or something.

 

 

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×