Good news folks. Finally nextcloud is back up!!!!
It was a hell of a night. Time to sleep / nap.
🔥
if you wonder what happen to nextcloud, its down and we are working to get it back up.
Ufff.... that was near heart attack experience. Everything seems to be back online.
We have some serious issues with our database server. We are working on it and will be back asap.
🔧 🔧🔧
Host server update and reboot;
database maintenance
🔧🔧🔧
Tonight (in about two hours) we are planing to update our host machines which means reboot. So expect a downtime of about 15min (depending how fast machines reboot) somewhere around 1:00 at night Amsterdam Time.
We are also going to do some maintenance on database containers which will means reboot and downtime of some services for few sec/min.
Hopefully it will get better in the course of the day today and that we will provide more permanent solution in the coming days.
#disroot #disrootstate
Sorry! :|
https://state.disroot.org
🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥
🔥 The general slowness 🔥
🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥
Today our main server is not happy. You probably noticed that disroot services are less responsive then normally. This is a result of few issue and some hardware issues on the backup server that piled up on eachother at the same time.
Hopefully it will get better in the course of the day today and that we will provide more permanent solution in the coming days.
🔥 No emails sent nor recieved. 🔥
This night we have been updating email server which resulted in our anti-virus not starting properly. This in turn made our mail server not able to push all emails. We have fallen asleep together with our mailserver :P
This morning after realising something is fishy, we've fixed it all up. All your missing emails should be arriving shortly.
🔩 Troubleshooting the cloud 🔩
We are trying to pinpoint the issue we have with the nextcloud. You might experience outage, missing apps etc.
Seems to be fixed now (tested on two different accounts).
🔥 Cloud - News app issue after update 🔥
Seems like News app update we did few hours ago introduced an issue where the user interface on the browser does not load. We are investigating it.
🔥 Nextcloud performance issue 🔥
For last two days our server is reporting higher processor load then normal, caused by nextcloud. We suspect one of the apps causing the issue. We will be investigating it this evening, which may lead to cloud short downtimes, or some apps disappearing for few minutes (we need to go through them all to see which one of them is the rascal).
Yet again we run out of disk space on the cloud server. We've shuffled some things around and will monitor the situation more carefully (well its a daytime so its easier :P)
🙃 Issues with Nextcloud performance 😋
We have run into performance issues on the nextcloud server due to exhausted disk space, which was caused by last nights update and maintenance, and our stupidity and ignorance (mainly that).
Things are back to normal now.
https://state.disroot.org/incident/121
🔥 War, war never changes.....🔥
We've friendly-fired spamgorumet
While fighting the nasty spam attacks both in and outbound, it looks like we've friendly-fire spamgourmet.com (at least one of their server ips.). Thanks to report of one of the users, we've stopped blocking it.
We apologize all users that use that service and were unable to send nor recieve emails from it for last days.
#disroot
...
At this moment we are monitoring closelly the email traffic and we've got in contact with gmx.de, web.de and google.com (who are playing major role in this type of attack, and actually making it happend).
✉️ Emails to gmx.de and web.de blocked for disroot.org users ✉️
Due to a spammer attack originating from our servers disroot.org domain has been blocked by gmx.de and web.de email services.
We've spent last days trying to understand and find a way to eliminate that attack. We hope we've finally managed, but we are still blocked.
⚙️ Nextcloud upgrade and maintenance work⚙️
This Friday/Saturday night we will be working on our nextcloud instance. This includes nextcloud upgrade to the latest version, upgrading the operating system and php to version 7.2 as well as some other minor changes. This means that nextcloud will be offline for some time within a span of about two hours. We’ll try to plan our work so that the downtimes are as short as possible.
#disroot
💾 Server update (second attempt)💾
Due to an anomaly that occurred yesterday, where our eyelids closed causing our bodies to go into suspend mode, we have not manage to commence the update operation.
This is why we are attempting to do it today.
So dont get spooked and lets hope @muppeth does not need to go to the datacenter to kickstart the servers like last time :P
🔩 Server update 🔩
Tonight we will be commencing routine weekly OS updates. This weeks update to out host server infrastructure require server reboot. This will cause few minutes downtime, so don't freak out when your favourite services are offline for a moment.
(around 1:30 am Amsterdam Local Time)