Staff Software Engineer, AI RuntimeActive$190K

Databricks
Apply now

The opportunity

P-1930 At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data…

What you'll do

  • Drive the architecture and evolution of AIR's managed GPU training platform,: delivering scalable, high-throughput, and resilient training across fleets that span thousands of accelerators.

  • Solve the hardest problems in large-scale training, including multi-node: orchestration, distributed parallelism strategies, GPU scheduling and dynamic routing, high-throughput data loading, and checkpoint and restore for very long-running jobs.

  • Push GPU efficiency and training performance, raising utilization (such as: model FLOPs utilization and end-to-end throughput) and lowering cost per training run across diverse model architectures and hardware generations.

  • Build the resilience and observability foundations that keep multi-node jobs: healthy, detecting and recovering from hardware and software failures with minimal disruption to customers.

  • Partner with product, research, and platform teams to shape the APIs, CLI,: and developer experience that make it easy to launch, monitor, and debug production training jobs.

  • Lead end-to-end engineering efforts, from design through production rollout,: holding a high bar for performance, correctness, and reliability.

What they're looking for

  • Make direct, high-impact contributions to the core systems behind AIR, and: help bring up support for the latest accelerators and new regions as the fleet grows.
  • Champion engineering excellence, mentor other engineers through design: reviews and technical discussions, and help shape Databricks' long-term technical direction in AI training infrastructure.
  • + years of experience building and operating large-scale distributed systems,: with significant depth in GPU training infrastructure, high-performance computing, or ML systems.
  • Hands-on experience with distributed training frameworks (such as PyTorch,: FSDP, DeepSpeed, or Megatron) and the parallelism strategies (data, tensor, pipeline, and sequence parallelism) used to train large models.