nlcp03 downtime

nlcp03 experienced some downtime during the evening, lasting on and off for a total of roughly 1 hour.

Below you'll find a detailed event log:

March 05:

8.34pm: Our monitoring triggered an alert that nlcp03 were unavailable. Upon investigation we saw that the network driver for the Mellanox network card in the server had locked up.

8.36pm: The once again became reachable and started serving traffic.

9.55pm: We started seeing stability issues again, with random timeouts for some connections, while other succeeded. We continued to investigate possible causes and solutions.

9.58pm: This was further confirmed by a customer ticket stating slow loading times as well.

10.05pm: Due to the stability getting worse and worse, we restarted the system, due to the initial driver error, it can put the system into a state where drivers may not fully recover, and reboots may quite often solve this due to unloading and loading the network driver from scratch again.

10.10pm: The server came online again.

11.49pm: The system once again started causing stability issues, to resolve this, we implemented a possible fix by setting `iommu=pt` as one of the boot options, since this has been known to fix similar NIC errors on AMD EPYC 7002 series servers (Which is what the particular server runs).

11.51pm: The system came online, we began our verification.

March 06:

12.05am: The iommu=pt fix didn't get implemented correctly, due to tuned profiles overriding the options, we changed the implementation and rebooted the system again to apply the iommu=pt fix.

12.10am: The system came online, and we began verifying the fix.

12:31am: The iommu=pt fix didn't resolve the issue, making us believe that the cause is a failing network card, we prepared a replacement server, and started our route towards the datacenter.

12:52am: Another reboot was made to bring back the system temporarily after losing the network.

01.15am: We arrived at the datacenter, passed through security, ensured the switch configurations were correct for the replacement hardware.

01:36am: nlcp03 powered down to move the drives to another physical server

01.41am: The server came back online in the new chassis, and we're since monitoring the situation.

The old system will be receiving some temporary drives, so we can perform some additional tests.