Senior Cluster Orchestration Engineer

Tehtävä

Most GPU clusters are split into silos. Capacity sits idle in one silo while another is starved. As a Senior Cluster Orchestration Engineer, you build the layer that turns these silos into one shared cluster, where training, inference and batch workloads run on the same GPUs and capacity moves between them automatically. This is a software development role in our AI Infrastructure team. You write the schedulers, controllers and platform services that make shared capacity work on Kubernetes and SLURM. You work closely with the engineers who run our GPU fleets and with the teams building customer products on top. The team is small, so you have direct influence on the architecture and a short path from idea to production, on state-of-the-art GPU hardware at real scale. Design and build the scheduling layer that lets training, inference and batch workloads share GPU capacity across Kubernetes and SLURM. Build support for perishable compute: capacity that is free only for a short window, such as idle reservations, spot-like capacity or gaps between jobs. Fill it with short-lived and preemptible work, and reclaim it cleanly when higher-priority work needs it. Implement gang scheduling, topology-aware placement and preemption for multi-node training with Volcano, KAI Scheduler, Kueue or Slurm Bridge. Contribute upstream, or build custom components where these tools fall short. Develop Kubernetes operators and controllers for quotas, priorities, fair-share and queueing, and for the cluster lifecycle: creation, scaling, upgrades, health checks and decommissioning. Solve the hard interactions between workload types: elastic and checkpoint-aware training, latency-sensitive inference autoscaling and batch backfill, all competing for the same GPUs. Build observability for scheduling and utilisation, including queue depth, wait times, fragmentation, preemption rates, GPU utilisation and SLOs for key platform services. Deliver the platform through GitOps, with declarative configuration, drift detection and safe rollout and recovery.