Workflows and API requests to Paragon may be timing out

Incident Report for Paragon

Postmortem

AWS made an automatic security + performance update to ElastiCache this morning at 3:30am PST. Our ElastiCache instances lost connections to some of the nodes in the cluster which led to communication issues between microservices.

No data was lost and all executions continued as expected, but users on the dashboard testing workflows or making http requests to dependent services experienced 5xx status codes.

Enterprise customers were completely unaffected.

Posted Jul 13, 2021 - 12:36 PDT

Resolved

The incident has been resolved and all systems should now be fully operational.

Posted Jul 13, 2021 - 09:03 PDT

Update

We are continuing to monitor for any further issues.

Posted Jul 13, 2021 - 08:54 PDT

Monitoring

A fix has been implemented and we are monitoring the results. All workflows and API requests should now be operational.

Posted Jul 13, 2021 - 08:53 PDT

Update

We are continuing to investigate this issue.

Posted Jul 13, 2021 - 08:24 PDT

Update

We are continuing to investigate this issue.

Posted Jul 13, 2021 - 08:23 PDT

Investigating

Some Workflow executions and API requests to Paragon may be failing or timing out. Our team is currently investigating the issue and we will update here as soon as we have more information.

Posted Jul 13, 2021 - 08:11 PDT

This incident affected: Workflows and Connect API.