We are stabilized now. We are monitoring closely but message processing is back to normal throughput with no elevated error rates.

Latest AWS Update (Oct 20 2:48 PM PDT):

"We have restored EC2 instance launch throttles to pre-event levels and EC2 launch failures have recovered across all Availability Zones in the US-EAST-1 Regions. AWS services which rely on EC2 instance launches such as Redshift are working through their backlog of EC2 instance launches successfully and we anticipate full recovery of the backlog over the next two hours. We can confirm that Connect is handling new voice and chat sessions normally. There is a backlog of analytics and reporting data that we must process and anticipate that we will have worked through the backlog over the next two hours. We will provide an update by 3:30 PM PDT."

Since the latest AWS update, we are observing customer's HL7 queues catching up.

AWS Update (Oct 20 12:15 PM PDT):

"We continue to observe recovery across all AWS services, and instance launches are succeeding across multiple Availability Zones in the US-EAST-1 Regions. For Lambda, customers may face intermittent function errors for functions making network requests to other services or systems as we work to address residual network connectivity issues. To recover Lambda’s invocation errors, we slowed down the rate of SQS polling via Lambda Event Source Mappings. We are now increasing the rate of SQS polling as we experience more successful invocations and reduced function errors. We will provide another update by 1:00 PM PDT."

We will continue to monitor performance closely and share additional updates as recovery progresses.

Since the latest AWS update, we are seeing improvements in system performance and an uptick in incoming messages for several customers.

AWS Update (Oct 20, 11:22 AM PDT):

"Our mitigations to resolve launch failures for new EC2 instances continue to progress, and we are seeing increased launches of new EC2 instances and decreasing networking connectivity issues in the US-EAST-1 Region. We are also experiencing significant improvements to Lambda invocation errors, especially when creating new execution environments (including for Lambda@Edge invocations). We will provide an update by 12:00 PM PDT."

We will continue to monitor performance closely and share additional updates as recovery progresses.

Some Reporting and Continuity customers maybe experiencing degradation due to the AWS outage (https://health.aws.amazon.com/health/status). We are currently monitoring the situation and will update as we have additional information.