Voice Services - DDOS (Denial of Services) Attack

Incident Report for Flux Labs - Status Updates

Postmortem

On Wednesday, October 6th at 10:14AM CST, the data center(s) which hosts Clarity Voice was hit with a DDoS attack. This Denial of Service attack over-loaded the network with queries to the systems which host the voice services.

By 10:40AM CST, the Network Team had already started mitigating the attack by adjusting the firewall rules. In order to maintain the attack, additional systems were brought online to load balance the network.

The attack continued throughout the afternoon and into the evening. Mitigation followed the curve. Systems were readily available around 8PM CST on October 6th.

The mitigation processes required engineers to make changes to the way calls were routed and handled. These changes limited the way phones communicated with outbound calls. This impacted Grandstream and Yealink phones. A patch was applied on Thursday, October 7th at 6:30PM CST for Grandstream templates.

A global fix was applied which resolved the outbound calling issues for all models on October 8th at 10:30AM CST.

Posted Oct 08, 2021 - 10:38 CDT

Resolved

All fixes have been deployed and connectivity on all device models have been restored. If you’re still having issues, please open a support ticket with our Helpdesk by emailing support@fluxlabs.net.

Posted Oct 08, 2021 - 10:26 CDT

Update

At this time Clarity’s network continues to function as expected with the majority of calls across the network functioning normally. There are some end users still experiencing issues with outbound dialing. This is due to the necessary implementation of security changes to mitigate the attack. The most commonly affected devices are Grandstreams, however there are others.

Last night we rolled out a fix for Grandstream phones. We are working on a way to deploy this across all such devices..

We also encourage you to use CoreNexa Web, Desktop, or Mobile for outbound dialing while we remediate the affected devices as all services are available and are working without issue.

Posted Oct 08, 2021 - 07:30 CDT

Update

The DDoS attack that hit Clarity's Voice network yesterday has subsided and our services have been restored.

We are receiving reports that some end users may be unable to make outbound calls. Our Cloud Services Team is working to restore services to these devices. We are in the process of rolling out a fix for Grandstream phones.

Posted Oct 07, 2021 - 15:51 CDT

Update

Clarity’s Voice network continues to function as expected, however some end users may still be experiencing residual effects from yesterday’s DDoS attack including issues with making and receiving calls.

We’re working these issues on a case by case basis.

Posted Oct 07, 2021 - 11:19 CDT

Update

At this time Clarity services continue to function as expected.

Clarity’s Cloud Services and Network Services Teams have worked throughout the night to make changes to our infrastructure and security to help prevent future DDoS attacks.

We will continue to diligently monitor the network for signs of new Incidents, while work continues to secure our network for all of our Customers.

Posted Oct 07, 2021 - 08:00 CDT

Update

Customers may still experience a variety of network related symptoms including portal down or slow, voice quality, dropped calls and unable to make or receive calls until services have completely restored.

Posted Oct 06, 2021 - 20:25 CDT

Update

Posted Oct 06, 2021 - 19:25 CDT

Update

Posted Oct 06, 2021 - 18:52 CDT

Monitoring

Clarity Voice Services and Network Services Teams continue working to block all sources of a DDoS attack that began at approximately 10:14AM today. While we have begun to see some traffic restore to normal, our Teams continue working to restore all network and calling services.

Customers may still experience a variety of network related symptoms including portal down or slow, voice quality, dropped calls and unable to make or receive calls until services have completely restored.

Posted Oct 06, 2021 - 18:09 CDT

Update

Clarity’s Voice Services and Network Services Team continue working to block all sources of a DDoS attack that began at approximately 10:14AM today and to restore all network and calling services.

A Denial of Service attack is when hundreds of thousands of queries are directed at a system. This overloads the system causing delays and impacts performance.

CoreNexa Video, Meeting and Room services are still functioning as expected.

Customers may still experience a variety of network related symptoms including portal down or slow, voice quality, dropped calls and unable to make or receive calls until services have completely restored.

Posted Oct 06, 2021 - 16:18 CDT

Update

In order to mitigate the disruptions from a DDoS attack; Clarity's Network Services Team applied a change to our network security devices at the recommendation of our vendor. We are beginning to see some traffic restore to normal. Our Cloud Services and Network Services Team continue working to block all sources of the attack and to restore all network and calling services.

Posted Oct 06, 2021 - 13:50 CDT

Update

Clarity's Cloud Services Team has identified the source of a service disruption of our Calling Services and is working diligently to restore services. Customers will experience a variety of network related symptoms including voice quality, dropped calls and unable to make or receive calls.

Posted Oct 06, 2021 - 11:45 CDT

Identified

As of 10:14AM, Clarity's Cloud Services Team is investigating a service disruption of our Calling Services. Customers will experience a variety of network related symptoms including voice quality, dropped calls and unable to make or receive calls

Posted Oct 06, 2021 - 10:45 CDT

Investigating

At approximately 10:14AM our Network Services Team reported the source of the disruption was a DDOS (Denial Of Service ) attack. Our Cloud Services and Network Services Team are working to block all sources of the attack and to restore all network and calling services.

Customers may still experience a variety of network related symptoms including portal down or slow, voice quality, dropped calls and unable to make or receive calls until services have completely restored.

Posted Oct 06, 2021 - 10:45 CDT

This incident affected: Flux Voice (Inbound & Outbound Calling).