disroot is a user on social.weho.st. You can follow them or interact with them if you have an account anywhere in the fediverse. If you don't, you can sign up here.

Good news folks. Finally nextcloud is back up!!!!

It was a hell of a night. Time to sleep / nap.

πŸ”₯
if you wonder what happen to nextcloud, its down and we are working to get it back up.

Ufff.... that was near heart attack experience. Everything seems to be back online.

We have some serious issues with our database server. We are working on it and will be back asap.

πŸ”§ πŸ”§πŸ”§
Host server update and reboot;
database maintenance
πŸ”§πŸ”§πŸ”§

Tonight (in about two hours) we are planing to update our host machines which means reboot. So expect a downtime of about 15min (depending how fast machines reboot) somewhere around 1:00 at night Amsterdam Time.

We are also going to do some maintenance on database containers which will means reboot and downtime of some services for few sec/min.

disroot boosted
Today our main server is not happy. You probably noticed that disroot services are less responsive then normally. This is a result of few issue and some hardware issues on the backup server that piled up on eachother at the same time.

Hopefully it will get better in the course of the day today and that we will provide more permanent solution in the coming days.
#disroot #disrootstate

Sorry! :|
https://state.disroot.org

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯
πŸ”₯ The general slowness πŸ”₯
πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

Today our main server is not happy. You probably noticed that disroot services are less responsive then normally. This is a result of few issue and some hardware issues on the backup server that piled up on eachother at the same time.

Hopefully it will get better in the course of the day today and that we will provide more permanent solution in the coming days.

Sorry! :|
state.disroot.org/incident/135

πŸ”₯ No emails sent nor recieved. πŸ”₯

This night we have been updating email server which resulted in our anti-virus not starting properly. This in turn made our mail server not able to push all emails. We have fallen asleep together with our mailserver :P

This morning after realising something is fishy, we've fixed it all up. All your missing emails should be arriving shortly.

state.disroot.org/incident/130

πŸ”© Troubleshooting the cloud πŸ”©

We are trying to pinpoint the issue we have with the nextcloud. You might experience outage, missing apps etc.

Sorry.
state.disroot.org/incident/127

Seems to be fixed now (tested on two different accounts).

πŸ”₯ Cloud - News app issue after update πŸ”₯

Seems like News app update we did few hours ago introduced an issue where the user interface on the browser does not load. We are investigating it.

state.disroot.org/incident/124

πŸ”₯ Nextcloud performance issue πŸ”₯

For last two days our server is reporting higher processor load then normal, caused by nextcloud. We suspect one of the apps causing the issue. We will be investigating it this evening, which may lead to cloud short downtimes, or some apps disappearing for few minutes (we need to go through them all to see which one of them is the rascal).

state.disroot.org/incident/123

Yet again we run out of disk space on the cloud server. We've shuffled some things around and will monitor the situation more carefully (well its a daytime so its easier :P)

πŸ™ƒ Issues with Nextcloud performance πŸ˜‹

We have run into performance issues on the nextcloud server due to exhausted disk space, which was caused by last nights update and maintenance, and our stupidity and ignorance (mainly that).

Things are back to normal now.
state.disroot.org/incident/121

πŸ”₯ War, war never changes.....πŸ”₯
We've friendly-fired spamgorumet

While fighting the nasty spam attacks both in and outbound, it looks like we've friendly-fire spamgourmet.com (at least one of their server ips.). Thanks to report of one of the users, we've stopped blocking it.

We apologize all users that use that service and were unable to send nor recieve emails from it for last days.

...
At this moment we are monitoring closelly the email traffic and we've got in contact with gmx.de, web.de and google.com (who are playing major role in this type of attack, and actually making it happend).

state.disroot.org/incident/113

βœ‰οΈ Emails to gmx.de and web.de blocked for disroot.org users βœ‰οΈ

Due to a spammer attack originating from our servers disroot.org domain has been blocked by gmx.de and web.de email services.

We've spent last days trying to understand and find a way to eliminate that attack. We hope we've finally managed, but we are still blocked.

state.disroot.org/incident/113

βš™οΈ Nextcloud upgrade and maintenance workβš™οΈ

This Friday/Saturday night we will be working on our nextcloud instance. This includes nextcloud upgrade to the latest version, upgrading the operating system and php to version 7.2 as well as some other minor changes. This means that nextcloud will be offline for some time within a span of about two hours. We’ll try to plan our work so that the downtimes are as short as possible.

state.disroot.org/incident/112

πŸ’Ύ Server update (second attempt)πŸ’Ύ

Due to an anomaly that occurred yesterday, where our eyelids closed causing our bodies to go into suspend mode, we have not manage to commence the update operation.

This is why we are attempting to do it today.

So dont get spooked and lets hope @muppeth does not need to go to the datacenter to kickstart the servers like last time :P


state.disroot.org/incident/111

πŸ”© Server update πŸ”©

Tonight we will be commencing routine weekly OS updates. This weeks update to out host server infrastructure require server reboot. This will cause few minutes downtime, so don't freak out when your favourite services are offline for a moment.
(around 1:30 am Amsterdam Local Time)

state.disroot.org/incident/110