---
title: 'Software Engineer, ML Infrastructure at Realm Labs'
canonical: 'https://feeny.ai/job/software-engineer-ml-infrastructure-realm-labs-sunnyvale-820mmna8cyxn'
type: 'job'
last_seen: '2026-11-03'
---

# Software Engineer, ML Infrastructure at Realm Labs

- **Company:** Realm Labs
- **Location:** Sunnyvale, CA
- **Compensation:** $180k–$250k
- **Employment:** full-time
- **Work type:** onsite
- **Posted:** 2026-01-16
- **Last confirmed live:** 2026-11-03
- **Apply:** https://jobs.gem.com/realm-labs/am9icG9zdDpNJfb0VmgE2M_7gQrMYWDr

## Job description

## Role Overview

We are hiring a Founding ML Infrastructure Engineer to own the end-to-end deployment, optimization, and operation of our suits of models in production. This is a core founding role focused on building and operating production-grade LLM systems. You will apply deep knowledge of model internals to deploy, optimize, and run modern LLMs at scale, owning performance end-to-end across latency, throughput, and reliability.

You will design and operate the full ML serving stack from model artifacts to GPU execution, and work closely with Product and ML teams to ensure our models can support high QPS, strict SLAs, and production correctness. This role is ideal for someone who deeply understands how LLMs work internally, but chooses to specialize in making them fast, stable, and production-ready.

## About Realm Labs

Realm Labs is an AI trust and security startup. We help enterprises detect, debug, and prevent AI’s misbehaviors in production. We are backed by top VCs and serve some of the most iconic global enterprises.

## Key Responsibilities

- Own the end-to-end LLM inference stack, including:- Model loading and execution
- GPU utilization and memory efficiency
- Runtime performance tuning
- Production deployment and scaling
- Design and operate high-performance LLM serving systems using technologies such as:- vLLM, TensorRT / TensorRT-LLM, Triton Inference Server, SGLang
- Optimize inference across:- Latency
- Throughput (QPS)
- GPU memory footprint
- Cost efficiency
- Work hands-on with PyTorch and TensorFlow models, including:- Model graph understanding
- Attention mechanisms, KV cache behavior, batching strategies
- Precision tradeoffs (FP16, BF16, INT8, etc.)
- Build and maintain production-grade GPU services:- Multi-model serving
- Autoscaling strategies
- Fault isolation and graceful degradation
- Collaborate with application and platform teams to:- Define serving APIs
- Ensure correctness and safety of outputs
- Debug production issues end-to-end
- Build a reproducible model training and versioning system for customer deployments
- Establish best practices for:- Model versioning
- Rollouts and rollbacks
- Performance benchmarking
- Production validation

Expected Qualifications

- 5+ years of professional experience in ML infrastructure, systems engineering, or production ML roles.
- Strong software engineering fundamentals; ability to write robust, maintainable production code.
- Deep hands-on experience with LLM inference infrastructure, including:- PyTorch (required)
- TensorFlow (working knowledge)
- Proven experience with GPU inference optimization, including:- TensorRT / TensorRT-LLM
- vLLM
- Triton Inference Server
- SGLang or similar serving runtimes
- Strong understanding of LLM internals, such as:- Transformer architectures
- Attention and KV caching
- Batching, streaming, and token-level generation
- Experience running ML systems in production with high traffic and SLAs
- Comfortable working in Linux-based, cloud production environments

## Preferred Qualifications

- Experience deploying LLMs on Kubernetes and GPU clusters.
- Familiarity with CUDA, NCCL, or low-level GPU performance concepts.
- Experience with:- Model sharding and parallelism strategies
- Multi-GPU inference
- Streaming inference systems
- Knowledge of observability for ML systems (metrics, latency breakdowns, GPU monitoring).
- Experience working at startups or owning systems with minimal abstraction layers.

Additional Information

- This is a founding, high-ownership role with direct impact on core product capabilities.
- You will be expected to build, run, and own systems end-to-end.
- The role may include limited on-call responsibilities aligned with production ownership.

## Compensation & Benefits

- Market aligned compensation and benefits
- Founding engineer equity (Equity is a significant component of this role and will be discussed)
- Medical, Dental, Vision, Life insurance, 401-K, In-office lunch etc.

Visa sponsorship: We do sponsor visas! However, we aren't able to successfully sponsor visas for every role and candidate. But if we make you an offer, we will make all reasonable effort to get you a visa, and we retain an immigration lawyer to help with this.

## About Realm Labs

## Company Overview
- **One-liner**: Realm Labs provides enterprise AI security and monitoring solutions that detect and prevent failures in LLM applications using deep neural inspection.
- **Entity Type**: Private (early stage startup)
- **Headquarters**: United States (specific city not disclosed)
- **Founded**: 2023
- **Founders**: Saurabh Shintre (CEO & Founder), Akash Mukherjee (Head of Engineering), Piotr Mardziel (Head of AI)

## Core Business
- Primary industry: AI Security / Cybersecurity
- Target customers: B2B Enterprise (companies deploying large language models in production)
- Mission: “We exist to ensure that progress doesn’t come at the cost of control, clarity, or accountability.”

## Products & Services
- **Realm Labs Platform**: AI-native SaaS/API that monitors every LLM interaction in real time. Uses patented Deep Neural Inspection (DNI) to read a model’s internal representations and catch hallucinations, prompt injections, jailbreaks, data leakage, and unsafe behavior at the source. Three modules: *Detect* (surface failures), *Respond* (block, rewrite, or route problematic outputs), *Improve* (feedback loop to correct drift and prevent repeat incidents).

## Market Standing
- **Valuation**: Not disclosed (no reported funding rounds)
- **Key Metric**: Headcount ~7 employees (as of late 2025/early 2026)
- **Notable Investors/Partners**: Not publicly listed
- **Growth Signals**: Early-stage research-driven company with a clear product and a team holding 20+ patents and 5,000+ academic citations across adversarial ML, AI safety, and system security.

## Competitive Advantages
- **Deep Neural Inspection** – only inspection point inside an LLM’s hidden representations that the model itself cannot forge, unlike surface-level output analysis.
- Founding team brings decades of combined expertise from Symantec, Splunk, Google, Apple, Snowflake, and leading universities (CMU, Stanford, USC).
- Focus on both detection and automated response creates a closed-loop security posture for enterprise AI.

## Strategic Focus
- Current priorities: Building trust and accountability in AI systems by providing guardrails, monitoring, and incident response. Growth direction: expanding from LLM security to a broader AI security platform that covers multi-modal models and agentic workflows.

## Why Work Here
- **Culture**: Small, agile, research-first environment where engineers and researchers solve hard AI safety problems.
- **Work Model**: Hybrid (remote + on-site in the US) based on Built In profile.
- **Notable Highlights**: Opportunity to work at the intersection of cutting-edge AI and security, influence product direction from day one, and collaborate with founders who are recognised experts in the field.

## Sources
1. [builtin.com](https://builtin.com/company/realm-labs) – Company overview, headcount, hybrid work, founding year
2. [realmlabs.ai/company](https://www.realmlabs.ai/company) – Founding team, mission, expertise
3. [realmlabs.ai](https://www.realmlabs.ai/) – Product details (Detect, Respond, Improve) and Deep Neural Inspection

## Other roles at Realm Labs

- [Machine Learning Research Intern (Winter/Spring 2027)](https://feeny.ai/job/machine-learning-research-intern-winter-spring-2027-realm-labs-sunnyvale-qfv0hsht8cgm) — Sunnyvale, CA
- [Intern: AI Red Teaming (Fall 2026)](https://feeny.ai/job/intern-ai-red-teaming-fall-2026-realm-labs-sunnyvale-cc16r5a6pk2v) — Sunnyvale, CA
- [Marketing Lead](https://feeny.ai/job/marketing-lead-realm-labs-united-states-7kbp1qe6tzxp) — United States
- [ML Research Engineer, Trust & Safety](https://feeny.ai/job/ml-research-engineer-trust-safety-realm-labs-sunnyvale-js9kd2qfaq4s) — Sunnyvale, CA
- [Software Engineer, ML Infrastructure](https://feeny.ai/job/software-engineer-ml-infrastructure-nuro-mountain-view-fycvgkt4gdjp) — Mountain View, CA