The opportunity
Anthropic's inference engine is the software between the accelerator kernels and the routing layer. It manages the entire token path in between: batching requests, laying the model out across chips, managing memory for weights and activations, coordinating every forward pass, and managing model state across requests.
What you'll do
Keep device utilization high. Accelerators should never be waiting due to other overheads.
Reuse instead of recompute. Keep model state cached and reuse it whenever: that is cheaper than computing it again.
Measure, model, then change. We build the observability to see where the gaps: are, model the impact of potential improvements, deploy them, and go around again, with Claude speeding up every turn of that loop.
Tokens you can trust. Ensuring model quality matters more than efficiency. We: build the infrastructure to ensure Claude maintains its intelligence across platforms and over time.
Safety on every token. We work closely with our safeguards and safety teams.: The inference engine is the backbone behind our production safety systems, ensuring efficiency without compromising robustness.
A working mental model of LLM inference: how prefill and decode land on an accelerator's compute, memory, and interconnect, and what the host is doing meanwhile
What they're looking for
- Experience inside an LLM serving engine and a sense of where its abstractions strain
- GPU/Accelerator programming
- Language modeling with transformers
- Experience building an allocator, cache, scheduler, or high-bandwidth transport
- Fluency in Rust
- Experience making systems reproducible: determinism, replay, property-based tests
