Rechner-Cluster

Due to unforeseen circumstances we are forced to fully stop and restart the Slurm batch system infrastructure.
During this short downtime the submission of new jobs wont be possible, Slurm commands will not be available and other Slurm related tasks will not be possible.
Already running jobs should be able to continue and finish without issue.

Updates went without problem from the server sides. Please report via Tickets any issues you might encounter AFTER the downtime.

Der Aachener RegApp wird die NHR RegApp als föderativer Proxy vorangestellt.
Während der Wartungsarbeiten sind keine neuen Anmeldungen beim Cluster, bei Perfmon und bei JupyterHub möglich.
Es wird empfohlen, dass Benutzer sich vorab eine Sitzung sichern.

A failure in several interconnected systems is causing a global outage of the HPC system.
We are working on a solution.

The RegApp service will be updated to a new version.
During the maintenance, no new logins to the cluster, perfmon, and JupyterHub will be possible.
It is recommended that users acquire a session beforehand.

Aktuell gibt es Probleme mit dem Login zum Cluster. Es kommt während des Logins zu längeren Wartezeiten und es kann zu Abbrüchen kommen. An einer Lösung wird gearbeitet.

The HPC JupyterHub is currently down to fix issues with internal updates

We currently have issues with Jobs using the default account landing on _low partitions

Strange submission error might be experienced while we fix the issue. Please re-submit.

Due to various necessary maintenance works, the entire CLAIX HPC System will be unavailable.

The initial phase of the maintenance should last until 12:00. After which filesystem and login nodes should be available.
Jobs will not run until the full maintenance works are completed.
We aim to also upgrade the Slurm scheduler during the downtime.

Please note the following:
- User access to the HPC system through login nodes, HPC JupyterHub or any other connections will not be possible during the initial part of the maintenance the maintenance.
- No Slurm jobs will be able to run during the entire maintenance.
- Before the maintenance, Slurm will only start jobs that guarantee to be finished before the start of maintenance; any running jobs must finish by then or might be terminated.
- Nodes might therefore remain empty leading to the maintenance, as Slurm tries to clear the nodes from user jobs.
- Waiting times before and after the maintenance might be higher than usual, as nodes are emptied before or the queue of waiting jobs increases in size afterwards.
- Files on your personal or project directories will not be available during the initial part of the maintenance.

The first part of the maintenance will take unexpectedly longer. At the moment, we cannot estimate when the maintenance work of the first part will be finished.

The network maintenance is still ongoing.

The maintenance works were sadly delayed due to circumstances out of our control. This will delay the HPC systems availability by a few hours.

Due to issues during the network maintenance and a short-hand failure of the storage backend of one infrastructure server, required in the maintenance, the maintenance tasks were delayed. Due to the delay, some tasks had to be postponed. The Cluster is operational again.

The RegApp service will be temporarily unavailable due to OS updates and to prepare for upcoming changes.
Login to the HPC frontends and to perfmon.hpc.itc.rwth-aachen.de will be unavailable.
It is recommended to login to the services beforehand.

Due to urgent security updates, the batch service will be temporarily suspended for all GPU nodes. After the updates are deployed, the service will be resumed. Running jobs are not affected and will continue. The dialog node login23-g-1 is not affected.

Once the updates are automatically installed upon job finalization, the nodes will be available again on short-notice.

The majority of the GPU nodes' jobs were finished early. The nodes could be upgraded without a noticable interruption of the service.

The users' HOME directories were temporarily unavailable on both the login and the batch nodes. As a consequence, existing sessions trying to access these directories would hang and new connections to the cluster could not be established. There is a chance that batch jobs that started or run during this time frame may have failed as a result. If your job has crashed, please investigate the output and in case of error messages related to files or directories underneath /home, please resubmit the affected jobs. Part of the batch nodes were automatically disabled as a result and are put back in operation as of this second.
We apologize for the inconveniences this has caused.

The underlying problem has turned out to be more persistent than expected and we are consulting with the vendor for a fix. As a result, the majority of batch nodes remain out of operation until we can guarantee stable access to the filesystem. We are working to remedy the situation as soon as possible.

The problem has been solved and the cluster is back in operation.

The issue re-occured and persistes at the moment. We are currently working on a solution.

The issues could be resolved.

Our GPFS global filesystem needs to be updated and will cause the entire CLAIX HPC System to be unavailable.
Please note the following:
- User access to the HPC system through login nodes, HPC JupyterHub or any other connections will not be possible during the maintenance.
- No Slurm jobs, filesystems dependent tasks will be able to run during the maintenance.
- Before the maintenance, Slurm will only start jobs that guarantee to be finished before the start of maintenance; any running jobs must finish by then or might be terminated.
- Nodes might therefore remain empty leading to the maintenance, as Slurm tries to clear the nodes from user jobs.
- Waiting times before and after the maintenance might be higher than usual, as nodes are emptied before or the queue of waiting jobs increases in size afterwards.
- Files on your personal or project directories will not be available during the maintenance.

Unfortunately the maintenance works will have to be extended. We hope to be done as soon as possible. We apologize for the inconvenience.

We must unfortunately postpone the release of the HPC system for normal use until Wednesday. We apologise for the delays.

Within the maintenance, a pending system upgrade due to security issues, a system update is done as well. However, due to the large number of nodes, the update still requires some time. The cluster will be available as soon as possible. Unfortunately, we cannot give an exact estimate when the updates are finished.

All updates should be completed later this evening. We target the cluster to be available tomorrow by 10:00 a.m.: The Frontend nodes should be available earlier prior to the batch service that will prospectively be resumed by 11:00 a.m. We apologize once again for the unforseen inconveniences.

The updates are still not completed and require additional time. We estimate to be finished this afternoon. The Frontends are already available again.

The global maintenance tasks could be completed, and we are starting to put the cluster back to operation starting from now. However, several nodes will temporarily remain under maintenance due to issues that could not be solved yet.

Operation of most of the nodes could be restored. The remaining few nodes will be processed soon.

The CLAIX-2023 copy nodes copy23-1 and copy23-2 will be reinstalled with Rocky Linux 9. During the Reinstallation, the nodes will not be available.

Aktuell drainen alle Knoten aufgrund einer Störung. Wir arbeiten mit dem Hersteller daran.

Das Problem konnte gelöst werden, der Cluster ist wieder in Operation.

Due to issues with the underlying filesystem servers for $HOME and $WORK, the batch nodes are currently unavailable, and access to $HOME and $WORK on the login nodes is not possible.

Access to the filesystems has been restored. We apologize for the inconvenience.

Due to issues with the GPFS filesystem $HOME and $WORK are not available on login23-g-1.
The issue has been resolved.

To resolve issues and fault analysis of the frontend node, the dialog node login23-1 is temporarily unavailable.

Due to a pending kernel update, the CLAIX-2023 copy nodes copy23-1 and copy23-2 will be rebooted on Monday, 2025-08-25.

Due to pending mandatory firmware updates, the maintenance needs to be prolonged.

Due to issues with respect to the firmware update, the end of the maintenance is delayed.

copy23-1 is available again.

The Maintenance is completed.

Due to mandatory maintenance work, the login GPU node n23-g-1 will not be available during the maintenance.
During the maintenance, the node's firmware will be updated which takes several hours to complete.

The Firmware upgrades are completed. However, there is an issue with respect to the SNC configuration. We were unable to enable the cluster-wide standard setting SNC4 on the dialog node. Further investigation is required. Notwithstanding this, the GPU login node can be used until further notice.

Due to issues with the underlying filesystem servers for $HOME and $WORK, the majority of batch nodes are currently unavailable, and access to $HOME and $WORK on the login nodes is not possible.

The issue has been resolved and the affected machines are back in operation. We are actively working with the vendor to remedy the situation and apologize for the repeated downtimes.

Due to issues with the shared filesystems, all services with respect to the HPC cluster are substantially impacted. The issues are under current investigation, and we are working on a solution.

On most of the nodes, the shared filesystems' function could be restored.

Due to issues with the GPFS file system, the $HOME and $WORK directories may currently be unavailable on the login nodes. Additionally, some compute nodes are temporarily not available due to these issues. Our team is working to resolve the problem as quickly as possible.

The issues could be resolved.

Due to issues with the underlying filesystem servers for $HOME and $WORK; GPFS, some batch nodes are currently unavailable and login nodes might be unstable. This may have caused running jobs to fail.
We are working to resolve the problem as quickly as possible and apologize for any inconvenience caused.
EDIT: Systems seem to be back online and operating normally.

Due to cooling issues, the CLAIX-2023 cluster has been shut down. Accessing the cluster is currently not possible.
Please note that running jobs may have been terminated due to the shutdown.
We are actively working on identifying the root cause and resolving the issue as quickly as possible.
We apologize for any inconvenience this may cause and thank you for your understanding.

The cluster is resuming operation now that the cooling system has been cleared for load. All batch nodes should be back in operation shortly.

The remaining Rocky 8 login nodes login23-3 and login23-x-1 will be upgraded to Rocky 9. These nodes will be unavailable during the upgrade process. Please use the other available login nodes in the meantime. The copy nodes are unaffected and will remain Rocky 8 nodes until further notice.

Zurzeit ist eine Anmeldung über RWTH Single Sign-On nicht möglich.
Bestehende Sessions sind nicht davon betroffen.
Wir arbeiten an der Behebung des Problems.
--English version---

Die Störung wurde behoben.

Due to security concerns, the previously installed file archiver "p7zip" (a fork of "7zip"), was replaced by the original program.
p7zip was forked two decades ago from 7zip due to initially lacking Unix and Linux support. Unfortunately, development is lacking severly behind of the upstream code such that issues are only partially fixed if at all. Since the upstream 7zip contains also several functional and performance improvements in addition to the bug fixes, the overall performance should have improved for the users as well. The usage should not change aside of minor changes due to the development history.

Due to issues with the underlying filesystem servers for $HOME and $WORK, all batch nodes are currently unavailable and the login nodes are not usable. This may have caused running jobs to fail, and no new jobs can start during the downtime.
We are working to resolve the problem as quickly as possible and apologize for any inconvenience caused.

The $HOME and $WORK filesystems are back online

Due to issues with the underlying filesystem servers for $HOME and $WORK, all batch nodes are currently unavailable and the login nodes are not usable. This may have caused running jobs to fail, and no new jobs can start during the downtime.
We are working to resolve the problem as quickly as possible and apologize for any inconvenience caused.

The filesystems have been properly re-mounted on all nodes and the system is back in normal operation. If your jobs have crashed due to I/O errors, please resubmit them. We again apologize for the inconvenience.

The Firmware of login23-3 and login23-4 must be upgraded. During the Upgrade, the nodes will not be available. Please use the other dialog nodes in the meantime.

The firmware upgrades are completed.

Im angegeben Zeitraum kam es teilweise zu Störungen des RWTH Single Sign-On. Nach Eingabe der Zugangsdaten lädt der Bildschirm, dann erscheint ein Internal Server Error.
Dies betrifft alle Services, welche den Single Sign-On Login verwenden.
--- English version ---

Due to an open security issue we have deactivated user namespaces on all Rocky 9 systems.
This feature is mainly used by containerization software and affects the way apptainer containers will behave.
Most users should not experience any interruptions. If you experience any problems, please contact us as
usual via servicedesk@itc.rwth-aachen.de with a precise description of the features you are using.

The user namespaces can be activated again once the Rocky 9 nodes are upgraded to Rocky 9.6. The dialog systems (cf. other ticket) were upgraded already. Currently we are procuring the upgrade to Rocky 9.6 in the background to minimize the downtime. Already upgraded nodes have user namespaces enabled by default again.

The remaining Rocky 9.5 nodes are planned to be updated to Rocky Linux 9.6 until 2025-07-04

Due to high-risk security issues within some components, the affected packages had to be removed from the cluster to enable operation until the security issues can be fixed. Unfortunately, the removal breaks the desktop environments XFCE and Mate due to tight dependencies to the respective removed packages. Consequently, the file managers "Thunar" and "Caja" as well as some other utilities cannot be used at the moment when using a GUI login.
If a GUI login is mandatory, please use the IceWM environment instead. However, some of the GUI applications might not work as expected (see above).
The text-based/terminal usage of the frontend nodes is not affected by the temporal change.

The upgraded Rocky 9 login nodes have received patches that solve the security issues. On these nodes, Mate and XFCE can be used again as desktop environment. However, The Rocky 8 login nodes (i.e., login23-3, login23-x-1) are still awaiting the respecitve patches. Until the patches are available, the usage of the respective environments remains limited.

The Rocky 8 frontend nodes received the pending update and can now be used with Mate and XFCE again.

Aktuell kommt es zu Störungen des RWTH Single Sign-On. Nach Eingabe der Zugangsdaten lädt der Bildschirm, dann erscheint ein Internal Server Error.
Dies betrifft alle Services, welche den Single Sign-On Login verwenden.
Die zuständige Fachabteilung ist bereits informiert und arbeitet an der Behebung.
--- English version ---

Die Störung konnte behoben werden. --- English version ---

The HPC JupyterHub will be down for maintenance during the hours of 9:00 and 11:00.

On Monday, June 23rd, the CLAIX-2023 Rocky 9 frontend nodes login23-1, login23-2 and login23-4 will be upgraded to Rocky Linux 9.6.
During the upgrade, these nodes will temporarily not be available. Please use the Rocky 8 frontend nodes in the meantime.

login23-g-1 and login23-x-2 will be reinstalled with Rocky Linux 9.6. Until the reinstallation is finished, these nodes are temporarily unavailable.

The GPU login node login23-g-1 is reinstalled and available with Rocky 9.6. The other nodes require further work.

The login nodes login23-1, login23-2, login23-4 are updated and available again. login23-x-2 requires some further work.

All Rocky 9-login nodes are upgraded to Rocky Linux 9.6.

Our second Slurm controller machine has lost memory modules and is currently pending maintenance.
This machine is responsible for keeping our Slurm DB in a current state.
Without it the submission to some new Slurm projects/ accounts will fail.
Please write a ticket if the problem persists.

login23-2 is currently not available. Please use one of the other dialog systems.

login23-2 is available again.

During the maintenance, the dialog systems will continue to be available
most of the time but without access to the hpcwork directories.
Batch jobs will not be running.

Due to unexpected problems with the filesystem update, we will have to prolong the maintenance until tomorrow. As of now, we presume that the problem will be solved until noon and will release the batch queues as soon as possible. We apologize for the inconvenience.

All maintenance tasks have been completed and both HPCWORK and the batch queues are operational again.

Due to issues with the new RegApp version, changing the HPC password may fail.

The issue has been identified and a fix has been deployed.

The RegApp software will be updated to the newest version. During this time, login to the frontends and perfmon may be unavailable. It is recommended to log in beforehand.

On Rocky Linux 9 systems (especially login23-1 or login23-4) the cp command of hpcwork files may fail with the error "No data available". Current workarounds: either use "cp --reflink=never ..." to copy files or run the cp command on one of the Rocky Linux 8 nodes, e.g. copy23-1 or copy23-2.

During system maintenance on 11.06. we will install a new version of the filesystem client software which will fix the problem. Until then please use "cp --reflink=never ..." to copy hpcwork files or copy files on Rocky 8 systems (e.g. copy23-1 or copy23-2).

In dem Zeitraum wird das Routing der bisherigen XWiN-Router (Nexus 7700) auf die neuen XWiN-Router (Catalyst 9600) umgestellt. Diese Router sind für die Netzverbindung der RWTH unerlässlich. Für diese Umstellung ist auch die Migration der DFN-Anbindung, welche redundant nach Frankfurt und Hannover geschaltet sind, sowie der RWTH-Firewall auf die neuen System erforderlich.
Innerhalb des Wartungsfensters wird es zu Ausfällen oder Teilausfällen der Außenanbindung kommen. Alle Services der RWTH (bspw. VPN, Email, RWTHonline, RWTHmoodle) werden innerhalb dieses Zeitraums nicht zur Verfügung zu stehen. Die Erreichbarkeit von Services innerhalb des RWTH-Netzes ist aufgrund eingeschränkter DNS-Funktionalität zeitweise nicht gegeben.
---english---

Der Uplink nach Frankfurt wurde erfolgreich auf das neue System geschwenkt.

Umbau des Uplinks nach Hannover beginnt.

Uplink nach Hannover auf das neue System umgezogen.

BGP v4/v6 nach Frankfurt und Hannover sind nun über die neue Routern funktional.

Es stehen noch ein paar kleinere Nacharbeiten an.

Wartung ist abgeschlossen. Der Datenverkehr läuft nun vollständig über die neuen Router!

Problematik mit der Anbindung zur Physik identifiziert, Lösung erfolgt morgen früh.

During the maintenance, we will migrate half of the CLAIX-2023 nodes (ML & HPC) and the entire devel partition to Rocky Linux 9. Additionally, the dialog systems login23-1, login23-x-1, and login23-x-2 will be migrated.

Auf login23-2 stehen aktuell keine Home- und Work-Verzeichnisse zur Verfuegung. Bitte weichen Sie auf eines der anderen Dialogsysteme aus.

We will change the MFA backend of the Regapp. Therefore, new logins will not be possible during the maintenance.

We will change the MFA backend of the Regapp. Therefore, new logins will not be possible during the maintenance.

we have to postpone the maintenance

We have a problem with the external coolage. We therefore have to do an emergency shutdown of CLAIX23.

We are now powering on the cluster again. The frontend (login-) nodes will be available soon again. Until further notice, the batchsystem for CLAIX23 keeps stopped. We hope to resolve the issue today.

The HPC JupyterHub will be down for maintenance to update some internal software packages.

Currently the access to the hpcwork directories may hang due to problems with the file servers. The problem is being worked on.

The accesss to $HPCWORK should work again now.

The access to the hpcwork directories may hang again.

The accesss to $HPCWORK should work again.

The installation of HPCJupyterHub Profiles is currently unavailable due to issues with our HPC container system.
A downtime of at least two weeks is expected. We apologize for the inconvenience.
Update: New Profiles can be installed now, after Apptainer changes.

The CLAIX-2023 login node login23-4 will be upgraded to Rocky Linux 9.5 for assisting the migration to Rocky Linux 9. During the version upgrade, the login node will not be available.

The CLAIX-2023 dialog node login23-4 is currently unavalable due to testing with respect to the planned migration to Rocky Linux 9. Please use an other dialog node until access to this node is available again.

The new Rocky 9 login node login23-4 will be available soon. We are resolving the last issues before it can be used for the Rocky 9 pilot phase.

The Rocky 9 login node login23-4 is available now.

Due to a bug in the RegApp, SSH command keys cannot be approved at the moment. We are working on a solulution.
Update 16.04 14:35:
The cause of the bug has been identified and fixed

the bug affecting ssh command key approval has been identified.
A hotfix will be deployed momentarily, two factor logins on the HPC will be temporarily unavailable.

No new logins are possible during the maintenance work. Users who are already logged in will not be disturbed and the maintenance will not affect the rest of the cluster. All frontends are still available, as is the batch system with its computing nodes.

At the moment, some issues with the file systems that impact the performance are observed. File access can be severely delayed consequently. The issue is currently under investigation.

We were not able to identify the root cause of the observed issues and are still working on a solution.

We changed some network settings on the GPFS servers, but that did not change anything. We are still working with the manufacturer to get a solution.

Additional changes have been made to the configuration. Initial tests tend to show an improvement in the GPFS performance. A full qualitative analysis of the performance is pending

The performance of the HOME and WORK filesystems are much better now.

Due to maintenance tasks on a cluster-wide scale that cannot be performed online, the whole batch service will be suspended for the maintenance.
The Regapp is also affected by the maintenance.

The infrastructure of the HOME and WORK filesystems will also be under maintenance. Hence, the frontend nodes will be inacessible as well until the pending tasks are completed.

Due to delays in servicing the file systems, the cluster maintenance needs to be prolonged.

Due to delays in deploying the updated file system, the maintenance hat do be prolonged.

Due to a scheduled kernel update, the jupyterhub node is temporarily unavailable. The node will be available again, as soon as the update is completed.

Heute Abend kam es gegen ca. 20:25 - 21:00 zu einem Performance-Tief des SSO-Service und wurde innerhalb dieser Zeit von der Fachabteilung geprüft. Das Problem ist behoben und schnelle Logins ohne Wartezeiten sind wieder möglich.
---English---

Due to a scheduled Kernelupdate, the login nodes, i.e., login23-*, are temporarily unavailable during the update. The nodes will be available again, as soon as the update is completed.

The maintenance had to be prolonged for few minutes.

During a kernel update issues occured which lead to delays in re-deploying the nodes to the batch service. Consequently, a large number of compute nodes is unavailable at the moment (reserved and/or down). We are currently working on handing the respective nodes and will release them as fast as possible.

The updates could be completed. The nodes are available again.

The HPC JupyterHub is currently unavailable due to unforeseen errors in the filesystems.
A solution is being worked on.

The kernel of the CLAIX23 copy node claix23-2 has to be updated. The node will be available again as soon as the update is finished.

Es ist zurzeit nicht möglich in der RegApp das HPC Passwort zu ändern.
Das anlegen neuer Accounts funktioniert nicht.

In der angegebenen Zeit findet eine kurze Wartung der RegApp statt. In dieser Zeit sind keine Logins möglich.
Update:
Die RegApp ist seit 9 Uhr wieder erreichbar, die Kommunikation mit dem 2. Faktor ist aber gestört. Entsprechend funktioniert der 2FA-Login am HPC noch nicht.
Update 10:10
Die Störung ist behoben, der 2FA-Login am HPC funktioniert wieder.

In der angegebenen Zeit findet eine kurze Wartung der RegApp statt. In dieser Zeit ist kein Login in der RegApp möglich.

We are migrating the slurm controller to a new host. It might come to short timeouts. We try to minimize that as much as possible.

The first try was not successfull, we are on the old master again. We are analyzing the problems that occurred and try again later.

we make another attempt

There may currently be login problems with various login nodes. We are working on a solution.

The observed issues affect the batch service as well. Consequently many batch jobs may have failed.

The observed problems can be concluded from power issues. We cannot exclude that further systems may have to be controlled shut down temporarily due to a problem resolution if required. However, we hope the issues can be resolved without any additional measures.

The cluster can be accessed again. cf. ticket https: maintenance.itc.rwth-aachen.de/ticket/status/messages/14/show_ticket/9521 for further details. Several nodes, however, are still unavailable due to the consequences of the aforementioned issues. We are currently working on resolving the issues.

Due to to be analyzed infiniband problems many nodes including the whole GPU cluster are not reachable at the moment. We are working together with the manufacturer, to solve the problems.

The problem could be fixed

Zurzeit ist der Single Sign-On und die Multifaktorauthentifizierung sporadisch gestört. Wir arbeiten bereits an einer Lösung und bitten um Geduld.
---english---

Dear users, Sunday the 8.12.2024 at 06:53:00 AM the home filesystems of most nodes went offline.
This might have negatively crashed some jobs and no new jobs can start during the downtime.
We are actively working on the issue.

Most nodes are coming back online. Apologies for the troubles. We expect most nodes to be usable by 18:00.

Due to a problem in our network, some nodes lost their connection to the $HOME and $WORK file system. This included the login23-1 and login23-2 nodes. The issue has been resolved now.

CLAIX-2023 was shut-down in emergency to prevent damage to the hardware. Due to severe power issues, the cooling facilities failed and could not provide sufficient heat dissipation.
The cluster will be operational again if the inducting issues can be resolved.

Both CDUs are active again and cooling could be restored. The cluster will be booted again for damage analysis only. Until further notice, the batch service remains suspended until all issues are resolved and all power security checks are positive.

The cooling system is now fully operational again. Additionally, we have implemented further measures to enhance stability in the future. The queues were reopened last night; however, we are currently conducting a detailed investigation into some specific nodes regarding their cooling and performance. Once these investigations are complete, the affected nodes will be made available through the batch system again.

We are putting a new file server for the home and work directories into operation. For this purpose we will carry out a system maintenance in order to finally synchronise all data over the weekend.

The maintenance needs to be extended.

Due to some issues preventing a normal batch service, the maintenance had to be extended.

Wegen anhaltender externer Störungen im RWTH-Netzwerk ist der Cluster nur eingeschränkt erreichbar und funktionsfähig. Die zuständige Netzwerkfachabteilung arbeitet bereits an einer Lösung der Probleme.

The issues could not be resolved until now and may persist trhoughout tomorrow as well.

The issues have been resolved.

Um 17:00 Uhr hat es einen kurzzeitigen Stromausfall im Raum Aachen gegeben. Die Stromversorgung besteht wieder, jedoch ist die Mehrzahl der Compute-Knoten infolgedessen ausgefallen. Es ist unklar, wann der Betrieb wieder aufgenommen werden kann.
Es wird momentan daran gearbeitet, kritische Dienste zu sichern und wiederherzustellen.

Nachdem die kritische Infrastruktur zum Betrieb der Systeme wiederhergestellt werden konnte, wurde der HPC-Cluster wieder bereitgestellt und freigegeben. Allerdings sind durch die Auswirkungen des Stromausfalls eine größere Zahl GPU-Knoten nicht mehr verfügbar. Wir arbeiten an der Behebung der Probleme, können allerdings noch keine Prognose geben, wann und ob die Systeme wieder verfügbar sein werden.

Der Großteil der ML Systeme (GPUs) konnten heute wieder hochgefahren und in den Batchbetrieb übergeben werden.

Our Slurm workload manager crashed due to an unknown reason. Functionality could be restored at short hand. Further investigations are ongoing.

Currently, a GPU of the GPU login node login23-g-1 shows an issue. The node is unavailable until the issue is resolved.

The issues could be resolved.

It is currently not possible to log in to the login23-* frontends. There is a problem with two-factor authentication

We are doing a new top500 run for the ML partition of CLAIX23.
The GPU nodes will not be available during that run.
Other Nodes and login23-g-1 might be also unavailable:
i23m[0027-0030],r23m[0023-0026,0095-0098,0171-0174],n23i0001

Aufgrund von dem abgelaufenen Zertifikat für idm.rwth-aachen.de können keine IdM-Anwendungen und die Anwendungen, die über RWTH Single Sign-On angebunden sind, aufgerufen werden.
- Beim Aufrufen von IdM-Anwendungen wird eine Meldung zur unsicheren Verbindung angezeigt.
- Beim Aufrufen von Anwendungen mit dem Zugang über RWTH Single Sign-On wird eine Meldung zu fehlenden Berechtigungen angezeigt.
Wir arbeiten mit Hochdruck an der Lösung des Problems.
--- English ---

Das Zertifikat wurde aktualisiert und die Anwendungen können wieder aufgerufen werden. Bitte löschen Sie den Browsercache, bevor Sie die Seiten wieder aufrufen.

Leider kam es in dem genannten Zeitraum zu einer Störung der RegApp, so dass man sich nicht auf den Frontends des Clusters einloggen konnte. Bereits bestehende Verbindungen wurden davon nicht beeinflusst. Das Problem ist behoben.

copy23-2 data transfer system will be unavailable for maintenance.

The maintenance is completed.

The firmware of the InfiniBand Gateways will be updated. The firmware update will be performed in background and should not cause any interruption of service.

The updates are completed.

Aufgrund einer Störung des DNS liefern die Nameserver verschiedener Provider aktuell keine IP-Adresse für Hosts unter *.rwth-aachen.de zurück.
Als Workaround können Sie alternative DNS-Server in Ihren Verbindungseinstellungen hinterlegen, wie z.B. die Level3-Nameserver (4.2.2.2 und 4.2.2.1) oder von Comodo (8.26.56.26 und 8.20.247.20). Ggf ist es auch möglich den VPN-Server der RWTH zu erreichen, dann nutzen Sie bitte VPN.

Anleitungen zur Konfiguration eines alternativen DNS-Server unter Windows finden Sie über die folgenden Links: https: www.ionos.de/digitalguide/server/konfiguration/windows-11-dns-aendern/ https: www.netzwelt.de/galerie/25894-dns-einstellungen-windows-10-11-aendern.html Als Alternative können Sie auch VPN nutzen. Wenn Sie den VPN-Server nicht erreichen, können Sie nach der folgenden Anleitung die Host-Datei unter Windows anpassen. Dadurch kann der Server vpn.rwth-aachen.de erreicht werden. Dazu muss der folgenden Eintrag hinzugefügt werden: 134.130.5.231 vpn.rwth-aachen.de https: www.windows-faq.de/2022/10/04/windows-11-hosts-datei-bearbeiten/

Die Host der RWTH Aachen sind nun wieder auch von ausserhalb des RWTH Netzwerkes erreichbar.

Auch nach der Störungsbehebung am 25.8. um 21 Uhr kann es bei einzelnen Nutzer*innen zu Problemen gekommen sein. Am 26.8. um 9 Uhr wurden alle Nacharbeiten abgeschlossen, sodass es zu keinen weiteren Problemen kommen sollte.

Many nodes suffered an issue after our updates on the 19.08.2024, resulting in jobs failing on the CPU partitions.
If your job failed to start or failed on startup, please consider requeuing it if necessary. This list of jobs was identified as possibly being affected by the issue:
48399558,48468084,48470374,48470676,48473716,48473739,48473807,48473831,
48475599,48475607_0,48475607_1,48475607_2,48475607_3,48475607_4,48475607_5,
48475607_6,48475607_7,48475607_8,48475607_9,48475607_10,48475607_11,48475607_12,
48475607_13,48475607_14,48475607_15,48475607_16,48475607_17,48475607_18,48475607_19,
48476753,48482255,48485168,48486404,48488874_5,48488874_6,48488874_7,48488874_8,
48488874_9,48488874_10,48488874_11,48488875_9,48488875_10,48488875_11,48489133_1,
48489133_2,48489133_3,48489133_4,48489133_5,48489133_6,48489133_7,48489133_8,48489133_9,
48489133_10,48489154_0,48489154_1,48489154_2,48489154_3,48489154_4,48489154_5,48489154_6,48489154_7,
48489154_8,48489154_9,48489154_10,48489154_11,48489154_12,48489154_13,48489154_14,48489154_15,
48489154_16,48489154_17,48489154_18,48489154_19,48489154_20,48489154_21,48489154_22,48489154_23,
48489154_24,48489154_25,48489154_26,48489154_27,48489154_28,48489154_29,48489154_30,48489154_31,
48489154_32,48489154_33,48489154_34,48489154_35,48489154_36,48489154_37,48489154_38,48489154_39,
48489154_40,48489154_41,48489154_42,48489154_43,48489154_44,48489154_45,48489154_46,48489154_47,
48489154_100,48489154_101,48489154_102,48489154_103,48489154_104,48489154_105,48489154_106,48489154_107,
48489154_108,48489154_109,48489154_110,48489154_111,48489154_112,48489154_113,48489154_114,48489154_115,
48489154_116,48489154_117,48489154_118,48489154_119,48489154_120,48489154_121,48489154_122,48489154_123,
48489154_124,48489154_125,48489154_126,48489154_127,48489154_128,48489154_129,48489154_130,48489154_131,
48489154_132,48489154_133,48489154_134,48489154_135,48489154_136,48489154_137,48489154_138,48489154_139,
48489154_140,48489154_141,48489154_142,48489154_143,48489154_144,48489154_145,48489154_146,48489154_147,
48489154_148,48489154_149,48489154_150,48489154_151,48489154_152,48489154_153,48489154_154,48489154_155,
48489154_156,48489154_157,48489154_158,48489154_159,48489154_160,48489154_161,48489154_162,48489154_163,
48489154_164,48489154_165,48489154_166,48489154_167,48489154_168,48489154_169,48489154_170,48489154_171,
48489154_172,48489154_173,48489154_174,48489154_175,48489154_176,48489154_177,48489154_178,48489154_179,
48489154_180,48489154_181,48489154_182,48489154_183,48489154_184,48489154_185,48489154_186,48489154_187,
48489154_188,48489154_189,48489154_190,48489154_191,48489154_192,48489154_193,48489154_194,48489154_195,
48489618_1,48489618_2,48489618_3,48489618_4,48489618_5,48489618_6,48489618_7,48489618_8,48489618_9,48489618_10,
48489776,48489806_6,48489806_55,48489806_69,48489806_98,48489842,48489843,48489844,48489845,48489882_1,48489882_2,
48489882_3,48489882_4,48489882_5,48489882_6,48489882_7,48489882_8,48489882_9,48489882_10,48494481,48494490,48494752,
48494753,48494754,48494755,48494756,48494757,48494758,48494759,48494760

Due to updates to our compute nodes, the HPC system will be unavailable for Maintenance.
The login nodes will available at noon without interruptions, but the batch queue for jobs won't be usable during the maintenance work.
As soon as the maintenance work has been completed, batch operation will be enabled again.
These jobs should be requeued if necessary:
48271714,48271729,48271731,48463405,48463406,48463407,48466930,
48466932,48468086,48468087,48468088,48468089,48468090,48468091,
48468104,48468105,48468108,48468622,48469133,48469262,48469404,
48469708,48469734,48469740,48469754,48469929,48470011,48470017,
48470032,48470042,48470045,48474641,48474666,48475362,48489829,
48489831,48489833_2,48489838

Please migrate your notebooks to work with newer c23 GPU Profiles!
The migration of the GPU Profiles to Claix 2023 and the new nodes of c23g has made the old python packages use non optimal settings on the new GPUs.
Redeployment of these old profiles is necessary and will take some time.

Since the cluster maintenance, random MPI job crashes are observed. We are currently investigating the issue and are working on a solution.

We have identified the issue and are currently testing workarounds with the affected users.

After successful tests with affected users, we have rolled out a workaround that automatically prevents this issue for our IntelMPI installations. We advise users to remove any custom workarounds from their job scripts to ensure compatibility with future changes.

The c23i Partition is DOWN due to unforeseen consequences of our Monitoring systems that automatically downs the only node in the partition.
A solution is momentarily unknown and will be investigated.
The HPC JupyterHub will not be able to use it until it is resolved.

Due to a security vulnerability in the Linux Kernel, user namespaces are temporarily deactivated. Upon the kernel update, user namespaces can be used again.

User namespaces are available again.

The quota system on HPCWORK may not work correctly. There may be an error "Disk quota exceeded" if trying to create files although the r_quota command reports that enough quota should be available. The supplier of the filesystem has been informed and is working on a solution.

File quotas for all hpcwork directories were increased to one million.

During the Maintenance, $HPCWORK will be reconfigured, such that RDMA over IB will be possible from the CLAIX23 nodes instead of HPCWORK access over ethernet. At the same time, the Kernel will be updated. After the Kernel Update, the previously deactivated User Namespaces will be re-activated, again.

The maintenance had to be extended for final filesystem tasks

Due to unforseen problems, the maintenance has to be extended to tomorrow 16.07.2024 18.00. We do not expect the manufacturer of the filesystem to take that long, but expect to open the cluster earlier again.

The maintenance could be ended successfully. Once again, sorry for the long delay.

HPCJupyterHub is down after faied update to 5.0.0. will stay until the update is complete.
HPCJupyterHub could not be updated to 5.0.0. Remains at 4.1.5.

The FastX web servers on login18-x-1 and login18-x-2 have been stopped, i.e. the addresses https:
login18-x-1.hpc.itc.rwth-aachen.de:3300 and https:
login18-x-2.hpc.itc.rwth-aachen.de:3300 are not available anymore. Please use login23-x-1 or login23-x-2 instead.

login18-x-1 and login18-x-2 has been decommissioned.

Due to maintenance work on the water cooling system, Claix23 must be empty during the specified period. As soon as the maintenance work has been completed, batch operation will be enabled again. The dialog systems are not affected by the maintenance work.

Additionally, between 10 and 11 o'clock, there will be a maintenance of the RegApp. During this time, new logins will not be possible, existing connections will not be disturbed.

Due to the reached EOL of Rocky 8.9, the MPI nodes of CLAIX23 must be upgraded to Rocky 8.10. The upgrade is performed in background during production to minimize the downtime of the cluster. However, during the Upgrade, free nodes will be removed on a selection-basis and will not be available for job submission until the upgrade is completed.
Please keep in mind that during the update, the library versions installed will likely change. Thus, the performance and application behaviour may vary compared to earlier runs.

Starting now, all new jobs will be scheduled to Rocky 8.10 nodes. The remaining nodes that still need to be updated are unvailable for job submission. These nodes will be upgraded as soon as possible after their jobs' completion.

The update of the frontend and batch nodes is completed. Remaining nodes (i.e. integrated hosting and service nodes) will be updated on the cluster maintanance scheduled for 2024-06-26.

The dialog nodes (i.e. login23-1/2/3/4, login23-x-1/2) will be updated to Rocky 8.10 today within the weekly reboot. The upgrade of copy23-1/2 will follow.

The copy frontend nodes (copy23-1, copy23-2) will be updated to Rocky Linux 8.10 during the cluster maintanance ond 2024-06-26.

The update of the remaining frontend nodes is completed.

Due to technical problems, it is not possible to create/change/delete HPC-Accounts or projects. We are working on that issue

Due to technical problems, it is not possible to create/change/delete HPC-Accounts or projects. We are working on that issue

The issue has been resolved.

During this period no RWTH-S, THESIS, LECTURE or WestAI projects can be granted. We apologize for the inconvenience.

Due to maintenance of the RegApp Identiy Provider, it is not possible to establish new connections to the cluster during the specified period. Existing connections and batch operation are not affected by the maintenance.

Aufgrund eines ausstehenden Sicherheitsproblems müssen wir sogenannte Usernamespaces auf dem Cluster vorübergehend deaktivieren. Dieses Feature wird hauptsächlich von Containervirtualisierungssoftware wie Apptainer genutzt, und die Abschaltung hat einen Einfluss darauf, wie diese Container intern aufgesetzt werden. Die meisten Nutzer sollten von diesen Änderungen nicht direkt betroffen sein und nahtlos weiterarbeiten können. Sollten Sie dennoch Probleme entdecken, kontaktieren Sie uns bitte via servicedesk@itc.rwth-aachen.de und schildern Sie uns, wie konkret Sie Ihre Container starten. Sobald wir einen Patch für die Sicherheitslücke einspielen können, werden wir User Namespaces wieder aktivieren.

A kernel update addressing the issue was released upstream and will be available to the compute cluster, soon. Upon the update, usernamespaces can be enabled, again.

We are planning to re-enable user namespaces on April, 29th after some final adjustments

We currently register recurring performance degradations on HPCWORK directories which might be partly worsened by the on-going migration process leading on to the filesystem migration on April, 17th. The problems cannot be traced back to a single cause but are actively investigated.

Due to technical problems, we will have to postpone the maintenance (and the final lustre migration step) to 23.04.2024 07:00.

During the Claix HPC System Maintenance, the HPC JupyterHub will be updated to a newer version.
This will improve Claix 2023 support as well mandatory security updates.
The whole clusters needs to be updated with a new kernel.

The migration was successfully completed.

In the last weeks, we started migrating all HPCWORK data to a new filesystem. In this Maintenance we will do the final migration step. HPCWORK will not be available during this maintenance.

Due to technical problems, we will have to postpone the maintenance (and the final lustre migration step) to 23.04.2024 07:00.

The whole clusters needs to be updated with a new kernel such that user namespaces can be reenabled again, please compare https:
maintenance.itc.rwth-aachen.de/ticket/status/messages/14/show_ticket/8929
Simultaneously the Infiniband Stack will be updated for better performance and stability.
During this maintenance, the dialog systems and the batchsystem will not be available. The dialog systems are expected to be reopened in the early morning.
We do not believe that the maintenance will last the whole day but expect the cluster to open earlier.

Due to technical problems, we will have to postpone the maintenance to 23.04.2024 07:00.

Unfortunately, unplanned complications have arisen during maintenance, so that maintenance will have to be extended until midday tomorrow. We will endeavor to complete the work by then. We apologize for any inconvenience this may cause.

During the stated time Claix-2023 will not be available due to a benchmark run for the Top500 list[1]. Batch jobs which cannot finish before the start of this downtime or which are scheduled during this time period will be kept in queue and started after the cluster resumes operation.
[1] https:
www.top500.org

The nodes are available now again

There are currently longer waiting times in the ML partition as the final steps of the acceptance process are still being carried out.

The waiting times should be better now

Am 03.04.2024 wird die RegApp aktualisiert. Während des Updatefensters kann der Dienst für kurze Zeit unterbrochen sein. Aktive Sitzungen sollten nicht betroffen sein.

There are currently problems when submitting jobs. We are working on fixing the problems and apologize for the inconvenience.

The problem is solved now.

Aufgrund eines ausstehenden Sicherheitsproblems müssen wir sogenannte User Namespaces auf dem Cluster vorübergehend deaktivieren. Dieses Feature wird hauptsächlich von Containervirtualisierungssoftware wie Apptainer genutzt und die Abschaltung hat einen Einfluss darauf, wie diese Container intern aufgesetzt werden. Die meisten Nutzer sollten von diesen Änderungen nicht direkt betroffen sein und nahtlos weiterarbeiten können. Sollten Sie dennoch Probleme entdecken, kontaktieren Sie uns bitte via servicedesk@itc.rwth-aachen.de und schildern Sie uns, wie konkret Sie Ihre Container starten. Sobald wir einen Patch für die Sicherheitslücke einspielen können, werden wir User Namespaces wieder aktivieren.
Update:
Wir haben einen Bugfix für die betroffene Softwarekomponente installiert und User Namespaces wieder aktiviert.

Zurzeit werden keine Daten auf /hpcwork angezeigt.
Die Fachabteilung ist informiert und arbeitet an der Lösung.
---english---

Die Störung wurde behoben.

Both CLAIX18 copy nodes will be rebooted on Monday, January 29th, 6.00 am (CET) due to a scheduled kernel upgrade. The systems will temporarily unavailable and cannot be used until the kernel update is finished.

Aufgrund von Netzwerkromplemen kann es im angegeben Zeitraum zu Problemen bei der Nutzung des Clusters gekommen sein.

Momentan kann keine Verbindung zum Windows-Cluster hergestellt werden.
Die Kollegen sind informiert und arbeiten an der Behebung des Problems.
-- english --

Die Störung konnte behoben werden. Eine Verbindung mit dem Windows-Cluster ist wieder möglich. --English Version--

The jupyterhub.hpc.itc.rwth-aachen.de DNS is Temporary out of Service for 20 Minutes. Problems accessing the hpc JupyterHub might arise from this failure. Please wait until the system comes back online.