Browse Jobs
Explore open roles across India.Explore active software roles. Sign up to apply and get AI match scores.
Top MatchesTop Job Matches
Loading jobs…
Back to Browse Jobs
Plaud
Machine Learning Engineer, Inference & Serving (Speech LLM) - San Francisco
Plaud
San Francisco•Hybrid•Salary•3-6 Yrs•4 days ago
Job description
AILLMSpeech RecognitionInference OptimizationReal-Time Systems
Overview
Plaud is building the next generation intelligence infrastructure and interfaces to capture, extract, and utilize intelligence from what people say, hear, see, and think. They are looking for a Machine Learning Engineer with expertise in inference & serving for speech LLMs.
Responsibilities
- Build and deploy high-throughput, ultra-low-latency inference engines for large language models or foundational speech models
- Understand latency, throughput, and Time-To-First-Token tradeoffs
- Handle continuous batching, KV cache management, and stateful connections
- Deep understanding of GPU architectures and memory hierarchy
- Communicate clearly and collaborate effectively with ML training team and backend infrastructure team
Qualifications
- Hands-on experience with building and deploying high-throughput inference engines for LLMs or speech models
- Understanding of latency, throughput, and Time-To-First-Token tradeoffs in real-time streaming environments
- Experience with continuous batching and KV cache management (e.g., PagedAttention)
- Deep understanding of GPU architectures (NVIDIA Ampere/Hopper) and memory hierarchy
- Excellent communication and collaboration skills
- Ability to thrive in fast-moving environments
- Passion for developing performance-optimized AI systems
Nice to have
- Experience with frontier serving frameworks like vLLM, TensorRT-LLM, SGLang, or NVIDIA Triton Inference Server
- Real-time audio streaming experience (WebSockets, WebRTC, neural audio codecs)
- Advanced inference techniques (speculative decoding, lookahead decoding, chunked prefill)
- Model compression & quantization (PTQ, FP8, INT8, AWQ, GPTQ)
- Large-scale distributed systems (multi-GPU and multi-node pipelines)
Education
- Bachelor’s or higher degree in Computer Science, Engineering, or relevant technical field required
Benefits
- Competitive compensation ($170K - $320K base salary + bonus + equity)
- Top-tier healthcare for employees and dependents
- 401(k) plan with company matching
- Unlimited PTO plus 13 paid holidays
- 12 weeks of paid parental leave
- Hybrid office environment (minimum 3 days in-office per week)
- Choice of top-of-the-line laptops/workstations
- Annual offsites and fully stocked office