This incident is confirmed to be resolved at this time. Thank you very much for your patience which has allowed us to diagnose and remove the malicious traffic from entering our network.
RCA report: At approx. 11:35am CST we experienced an issue with our SIP Proxy reporting issues and not being able to handle incoming voice traffic.
Our engineering team responded to the issue immediately upon being notified and continued to investigate the source of the issue until 12:00 CST.
At approx. 12:05pm CST the engineering team identified the source of the issue. We determined that the issue was related to malicious SIP fragmented messages. We proceeded to disable the SUBSCRIBE handler in our proxies to mitigate the issue in order to allow calls to take place.
At approx. 12:45pm CST the engineering team identified the source ip causing the issue. We determined that the malicious SIP SUBSCRIBE traffic was coming from an IP that hadn't yet made it to the APIBAN blacklist. We proceeded to block the IP manually which allowed us to enable the SUBSCRIBE traffic again on our proxies.
At approx. 1:10pm CST we determined it was safe to say that the malicious traffic had been fully mitigated and we proceeded to mark this issue resolved.
We are very sorry for the inconvenience this has caused your business and we have been working diligently to add more proactive measures in place to prevent this type of event from happening in the future.
Posted Apr 26, 2024 - 15:29 CDT
Monitoring
A fix has been implemented to block the malicious traffic and BLF has now been enabled. We are monitoring the ingress traffic to ensure things are back to normal and will update here as soon as possible.
Posted Apr 26, 2024 - 13:16 CDT
Identified
We've identified the issue to be related to malicious devices sending a malformed SIP SUBSCRIBE message. While we continue to determine the source of the device(s), we have continued to disable the BLF functionality.
Posted Apr 26, 2024 - 12:27 CDT
Investigating
We are currently undergoing an issue with receiving a high amount of SIP SUBSCRIBE requests to our SIP Proxies resulting in degraded voice calling. We are investigating the source and will update here ASAP. We have disabled the BLF functionality while we diagnose the issue so BLF lights on phones will not function right now.
Posted Apr 26, 2024 - 12:22 CDT
This incident affected: Inbound Calling and Outbound Calling.