We are now closing this incident after several successful launches.
Posted Mar 29, 2021 - 13:10 EDT
The unresponsive node in the cluster has been recycled and all pods stuck in the Terminating state have been pruned. The operations team will make it a point to reduce the likelihood of this occurring in the future by forcefully removing pods stuck in the terminating state when one of the node instances in the cluster is not responding or in the process of being recycled.
Posted Mar 29, 2021 - 13:04 EDT
We are continuing to work on a fix for this issue.
Posted Mar 29, 2021 - 12:56 EDT
One of the three-node instances in the Virginia region (us-east-1) failed to respond. This led to some pods stuck in the terminating state. We are pruning these pods stuck in the terminating state and are updating the node cluster to have all instances reflect the ready state.