[Resolved] iOS Builds are failing

Started on November 30, 2021 at 5:00 AM. Resolved after 1 day

Affected

Builds
iOS
  • Investigating
    November 30, 2021 at 5:00 AM

    We are currently investigating an issue with iOS builds. We will update here as we have more info!

  • Investigating
    December 1, 2021 at 1:39 AM

    We are still investigating the root cause of these build server errors and will continue to provide updates.

  • Identified
    December 1, 2021 at 2:29 AM

    We were able to get a limited number of servers back up and running. We are holding off restarting all of the failed builds until we have our servers back at full strength, to not overwhelm the limited number back up.

    Newly started builds should succeed, but may be “Queued” for longer than usual. We’ll provide more updates tomorrow as we get more online.

    We appreciate your patience while we worked through this, and apologize for any inconvenience this causes.

  • Monitoring
    December 1, 2021 at 5:49 PM

    We have identified and resolved the root issue to these errors. We have all of our build servers back online. We are continuing to monitor the build queues, but have seen error rates return to normal.

    You may experience longer than normal queued times as previously failed builds are being restarted, leading to higher than typical volume, but it should not be more than ~30-45 minute "Queued" times.

    If you had a failed build in the past 2 days, please submit new builds. We will be working to get the most recent failed build for apps restarted, but you will be quicker to restart these on your apps.

    Please submit a support ticket if you are still facing errors.

  • Resolved
    December 1, 2021 at 7:35 PM

    We are confident iOS builds are operating again with error rates returning to normal. Queued times have also been lower than expected. Resolving this incident!