Resolved
App databases are not responding

Started
September 18, 2022 at 5:59 AM
Status
Resolved after about 9 hours

Impact

Major outage
Affected
Collections API
Published Apps
Collections
  • Resolved
    September 18, 2022 at 3:22 PM

    On September 18 at 1:39 AM EST, Adalo began experiencing a service interruption that impacted access to your database — in both the Adalo editor inside of your database collections and in your end-apps when trying to view or create new data. At 11:15 AM EST on September 18, access to all your databases was restored.

    We understand this is extremely frustrating, not only for you, as app creators, but also for the users of your apps & for that, we are sorry. Unfortunately, this was an upstream dependency where our hands were tied in bringing this interruption to a quick resolution.

    Here’s what happened. At 1:39 AM EST our database provider AWS started a minor upgrade on our databases. This should have only been a momentary upgrade but an unexpected error made the databases restart itself on a continuous loop — as there was an issue on their end with that process. And this is what caused the issue.

    At the beginning of the interruption, our on-call engineers were paged and immediately tried the options that we had on our end to make the databases accessible, but ultimately realized that only AWS could stop the loop causing the issue. We immediately contacted our support reps at AWS to bring our databases online. At the same time, we began a manual restore of the database to bring everything back online. That process is a lengthy process only used in case of emergency and ultimately that’s what our engineers had to do in order to resolve the issue.

    Again, we apologize for this inconvenience & our team will be doing a retrospective to prevent this from happening again in the future.

    As always, thank you so much for building with no-code and creating with Adalo.

  • Identified
    September 18, 2022 at 1:19 PM

    Our engineers continue to monitor the auto-upgrade of databases through AWS, which is what took them offline. They ultimately are waiting on AWS to bring our databases online at this time — either through the ending of this upgrade cycle or AWS manually turning off the upgrade.

    All of your data previous to this outage will be restored fully, with the exception of any data that was created between 11:26pm EST Sep. 17th and 1:39am EST Sep 18th. It's simply not accessible at this time.

  • Identified
    September 18, 2022 at 6:19 AM

    An auto update of our database instances is keeping them offline. The databases are restarting, but we expect them to be offline until all of the databases reboot.

  • Investigating
    September 18, 2022 at 5:59 AM

    We are currently investigating this incident.