This incident has been resolved. High-volume workflows have returned to a normal rate of executions according to our metrics over the past 20 hours (since our fix was released).
Posted May 25, 2023 - 12:27 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted May 24, 2023 - 15:41 PDT
Identified
We have identified the cause of the issue and are working on implementing a fix, which we expect will be live by tomorrow.
Our team will be closely monitoring workflow performance and providing updates once it is live.
Posted May 22, 2023 - 10:30 PDT
Investigating
We are currently investigating an issue where high-volume workflows are experiencing abnormally high queueing latencies.
Workflows affected will appear in Task History as an actively executing workflow that appears to not be actively running a step for several minutes before the next active step appears as executing.
Impact is limited to high-volume workflows (using a Fan Out step over >100 items, having >100 concurrent active executions, and workflows triggered by integration-based triggers).