<img height="1" width="1" style="display:none;" alt="" src="https://dc.ads.linkedin.com/collect/?pid=42246&amp;fmt=gif">
status header.jpg

22 October 2018

RESOLVED: Several NL3 customers report slow responses

This problem has been marked as resolved at October 24, 2018 12:31:53 PM CEST.

Original post
October 22, 2018 10:45:34 AM CEST
Several customers report slow response times while working in TOPdesk. We are aware of the problem and working on a solution.

Our apologies for the inconvenience. We aim to update this status blog at least every 30 minutes until the issue has been resolved.

Root Cause Analysis:

Overloaded parts in the network infrastructure of our NL3-hosting location caused some of the traffic to be dropped, resulting in slowness while working in TOPdesk. 

The issue was resolved by replacing the network devices with new higher capacity models. 

Finding the parts that were causing the disruption proved difficult, as it should not become overloaded with this traffic volume according to specifications. Due to the way the disruption manifested, it did not trigger helpful alerts in the monitoring system. Also, this type of malfunction has not occurred before.   

We are working with our hosting partners to ensure disruptions are detected more rapidly and improve monitoring information to better support finding the cause of a disruption.  

Procedures will be reviewed to ensure fewer disruption occur and better status updates are provided towards our customers.  

 

Update(s):
October 24, 2018 12:27:26 PM CEST
While monitoring connections to our hosting provider, we no longer see any delay or packet loss. The equipment replacement executed by the hosting provider yesterday has resolved all slowness issues. We are aware of the impact this disruption has had on the availability of TOPdesk environments. This disruption will be evaluated both internally and with the hosting provider to ensure a similar issue does not reoccur. We expect to update out status blog with a root cause analysis and follow up steps in 2 weeks.

October 24, 2018 10:39:07 AM CEST
Connections to TOPdesk SaaS environments are still being monitored. We no longer see any delay or packet loss, indicating the firewall replacement has resolved all slowness issues. We will continue to monitor the situation for a little longer and will close this incident once we're sure the issue has been resolved.

October 23, 2018 4:26:52 PM CEST
The datacenter replaced the malfunctioning firewall and we do not see any connection problems anymore. As the connection problems are related to the network usage, we will continue to regularly test connections until after our typical peak traffic moment tomorrow morning.

October 23, 2018 3:36:59 PM CEST
Final checks are being executed before replacing the firewall. The datacenter engineers expect this to take 15 more minutes.

October 23, 2018 2:35:27 PM CEST
The preparations are taking longer than expected. The hosting provider expects to be able to change the firewall in 30 minutes.

October 23, 2018 1:17:25 PM CEST
The hosting provider will replace the malfunctioning firewall to resolve the connection issues. The required preparations are expected to take one hour.

October 23, 2018 12:50:50 PM CEST
The hosting provider has identified one of their firewalls as the cause of the current connection issues. Options to resolve this issue are now being investigated and tested.

October 23, 2018 12:24:20 PM CEST
The hosting provider has tested several possible causes for the connection issues, but nothing resolved the current problems. There are still several options being tested. We are aware of the impact this problem has on your ability to work in TOPdesk. We have escalated this problem at the hosting provider in order to make sure resolving this issue is their primary concern.

October 23, 2018 11:18:50 AM CEST
Our hosting provider is still investigating the root cause of the slowness issues. We expect to have more information in 45 minutes.

October 23, 2018 10:15:36 AM CEST
We notice an increase in connection times to our NL3 datacenter, which can result in slow connections to TOPdesk SaaS environments. We've contacted the hosting provider to continue the investigation.

October 23, 2018 9:57:07 AM CEST
All network connections to our datacenters are still as expected. We'll continue to monitor network connections until after our typical peak traffic moment.

October 22, 2018 9:47:41 PM CEST
Emergency maintenance on the firewalls has been completed successfully. At the moment connection speeds in our NL3 datacenter don't show any abnormalities and are back to normal. We will continue monitoring the network connections tomorrow morning.

October 22, 2018 4:42:38 PM CEST
To resolve this issue, maintenance is required on a firewall in the datacenter. This emergency maintenance has been scheduled starting 20:00 CEST tonight. During the maintenance we expect no downtime for TOPdesk SaaS environments. Technicians from the hosting provider and TOPdesk are on stand-by in case a rollback is needed. We will continue to monitor connections to our datacenter, and will update this incident when more information is available.

October 22, 2018 4:11:19 PM CEST
A probable cause for the slowness issues has been found. The issue has been temporarily mitigated and connection speeds are back to normal. We are working with the hosting provider to make sure a permanent solution is in place as soon as possible, without further impact to customer environments.

October 22, 2018 3:01:53 PM CEST
Our engineers are still working on the connection issues. Several possible causes of packet loss issues have been tested, but the root cause has not been found. We've escalated the problem both at TOPdesk and at the hosting provider to resolve this problem as soon as possible.

October 22, 2018 1:48:00 PM CEST
The packet loss issues on connections to our NL3 hosting location are back. We are still working with our hosting provider to resolve this issue.

October 22, 2018 1:13:27 PM CEST
While investigating this issue we saw connection speeds in the NL3 datacenter returning to normal. Also, several customers reported connections are quicker and more stable now. We are continuing our investigation to ensure the root cause of the issue has been resolved.

October 22, 2018 11:40:06 AM CEST
There are no known issues at the hosting provider. We're working with the datacenter engineers to investigate and resolve the issue.

October 22, 2018 11:04:28 AM CEST
We've checked our internal services and servers, but could not find a cause for the slowness. When checking the connection to our datacenters, we noticed a delay in all pings to the NL3 and US1 hosting locations. We're working with the hosting providers to resolve this issue.