Noc­tua 2 - Tech­ni­cal de­s­crip­ti­on

The overall setup of the Noctua 2 supercomputer with insights into the operation of the cluster from a hardware, software and facility perspective is described in the Noctua 2 paper.

System Atos BullSequana XH2000
Processor Cores 143.872
Total Main Memory 347,5 TiB
Floating-Point Performance CPU: 5.4 PFLOPS DP Peak (4.19 PFlop/s Linpack)
GPU: 2.49 PFLOPS DP Tensor Core Peak (ca. 1.7 PFlop/s Linpack)
Cabinets 12 racks - direct liquid cooling,
7 racks - air cooling, four of them with active backdoor cooling
Compute Nodes

990 nodes, each with

    • 2x AMD Milan 7763, 2.45 GHz, up to 3.5 GHz

       • 2x 64 cores

    • 256 GiB main memory

Compute Nodes (Large Memory) 66 nodes, each with

    • 2x AMD Milan 7763, 2.45 GHz, up to 3.5 GHz

       • 2x 64 cores

    • 1024 GiB main memory

Compute Nodes (Huge Memory)

5 nodes, each with

    • 2x AMD Milan 7713, 2.0 GHz, , up to  3.675 GHz

       • 2x 64 Cores

    • 2 TiB main memory

    • 34 TiB SSD-based memory

       • 12x 3.2 TB NVMe SSDs, ~70 GB/s

GPU Nodes

32 nodes, each with

    • 2x AMD Milan 7763, 2.45 GHz, up to 3.5 GHz

       • 2x 64 cores

    • 512 GiB main memory

    • 4x NVIDIA A100 with NVLink and 40 GB HBM2

NVIDIA DGX-A100

one node

    • 2x AMD EPYC Rome 7742, 2.25 GHz, up to 3.4 GHz

        • 2x64 cores

    • 1024 GiB main memory

    • 8x NVIDIA A100 with NVLink and 40GB HBM2

FPGA Nodes

36 FPGA nodes, each with

    • 2x AMD Milan 7713, 2.0 GHz, up to  3.675 GHz

        • 2x 64 cores

    • 512 GiB main memory

80 FPGA cards

    • 48x Xilinx Alveo U280 FPGA with 8GiB HBM2 and 32GiB DDR memory

    • 32x Intel Stratix 10 GX 2800 FPGA with 32 GiB DDR memory (Bittware 520N cards)

Communication Network CPUs Mellanox InfiniBand 100/200 HDR, 1:2 blocking factor
Communication Network FPGAs

Application-specific interconnect

    • 4x40 Gbps links per FPGA

    • Connected via CALIENT S320 Optical Circuit Switch (OCS)

    • Configurable point-to-point connections to any other FPGA

    • Accessible as serial channels from OpenCL BSP

Storage System

DDN Exascaler 7990X with NVMe accelerator
Lustre File System with 6 PB capacity