Resolved -
Some internal RTB calls experienced a delay of over 1s, which could cause a timeout for deeply chained RTB pings. Overall p50 latency for RTB rose to 2s, with a p99 of 5.2s during the incident.
We believe the issue was caused by an upgraded database library included in a release on Jan 13th. The library changed some defaults that can impact performance in certain situations.
We released a hotfix this afternoon to resolve the issue, and will have a subsequent release tomorrow to update other (undocumented) default settings that were also changed.
Jan 21, 17:01 PST
Monitoring -
Targets should no longer be timing out. We will continue to monitor.
Jan 21, 14:56 PST
Identified -
The engineering team has identified the root cause and mitigation is in progress. You may see intermittent timeouts while the system fully stabilizes.
Jan 21, 14:02 PST
Investigating -
Some Ring Tree Targets are currently timing out. The team is actively investigating.
Jan 21, 13:39 PST