Previous incidents

We have reverted the Reporting app to the previous release.

We have determined the root cause of the performance degradation as a missed maintenance window on our HL7 processing database. We've since performed the required maintenance and the database has fully recovered. We consider this incident to be resolved, but we're still monitoring the database closely.

We've deployed a fix and are no longer seeing elevated error rates for OmniReport requests.

[Update 7] The issue is now resolved

[Update 6] After deploying the fix to our backend, we are monitoring the systems to check if the solution resolved the issue completely. We will provide a new update by 11:00am PST

[Update 5] On top of the database change previously mentioned, we deployed a new fix to our backend system. We are monitoring the logs to check if it fixed the issue. We will provide a new update by 10:45am PST

[Update 4] We reverted the database change that caused the issue ...

Update 1: Cloudfare has reported the issue is resolved and all internal testing is successful with the Rad AI applications.

Cloudfare currently has a global network outage. We are closely monitoring the situation and will update impacts to Rad AI as soon as our teams complete initial investigation.
https://www.cloudflarestatus.com/

We are stabilized now. We are monitoring closely but message processing is back to normal throughput with no elevated error rates.

Latest AWS Update (Oct 20 2:48 PM PDT):

"We have restored EC2 instance launch throttles to pre-event levels and EC2 launch failures have recovered across all Availability Zones in the US-EAST-1 Regions. AWS services which rely on EC2 instance launches such as Redshift are working through their backlog of EC2 instance launches successfully and we anticipat...