RWTH High Performance Computing (HPC)

Due to latent bugs in resource accounting, ssh-ing into nodes where one of your own jobs is running is temporarily disabled.

We are aware of the popularity of this feature and aim to reenable it in a future maintenance.

With the new Rocky 9.8 updates done, we will next deploy cgroups v2 and re-enable ssh into compute nodes. The timeframe for this is still unfortunately unknown.

Due to a critical PHP security vulnerability, the JARDS service has been shut down as a precautionary measure.

Due to open security issues that cannot be mitigated at short-hand, the cluster will be temporarily restricted.
The frontdend nodes are already closed and will be rebooted shortly.

During the security closure, login to all dialog systems is disabled, and batch submissions are henceforth not possible until the issues can be resolved. Already submitted jobs are still queued and will not be affected. At the moment, we do not consider running jobs nor submitted jobs need to be cancelled.

We are actively working on mitigating the security issues and will post updates once available.

The reboot of the frontend systems is scheduled for 12:00 CEST.

The kernel build containing manually backported patches to mitigate the issues was completed after an unexpectedly long build time. The mandatory kernel update is being prepared at the moment.

The built kernel packages appear to be broken and need to be rebuilt again.

The mandatory patches could be applied. All frontend nodes are available again, already patched batch nodes are available for batch submissions as well. However, since some jobs were still running, these nodes must be rebooted after their job completion to fully mitigate the security issues. New jobs will not be scheduled to these nodes prior to their reboot. Afterwards, the nodes will be available as well.

Please note that the running jobs exceed the proposed end date of this maintenance post and will prospectively end within the forthcoming days.

We are aware of issues submitting Jobs. We are working on a solution.

The problem has been solved.

Partial outage has been resolved.

We identified a bug with the latest driver releases for CLAIX-2025 which leads to job failures when starting GPU jobs across multiple nodes. We are rolling out a patch to the affected nodes which requires rebooting the systems after running jobs are finished.

We apologize for any inconveniences caused.

We have identified the issue to be related to a race condition in the module loading behaviour which prevents the network stack to fully initialize. A workaround to resolve the race condition has been implemented to resolve the issue.