Principal Engineer – Distributed Systems (GPU Edge + Inference) at Elloe AI (Austin, TX)

Elloe AI· Austin, TX·2d ago

Role details

Work type

Remote

Employment

Full-Time

Job description

Full-time | Remote | Infrastructure | Reports to CTO

About Elloe

Elloe is the trust layer for AI. We sit between the world’s most powerful language models and the institutions that can't afford to get it wrong — hospitals, banks, regulators. We trace and block failures in real time. That’s not marketing — we’re deployed at the European Commission, with NIH clinical trials, and inside a Top-5 EU bank catching GDPR violations live.

This is the enforcement layer GenAI has been missing. We're not visualizing problems — we're fixing them.

About the Role

You’ll lead our GPU-edge inference systems. From chaos-resilient deployment to SHAP-driven compliance metrics, you’ll own global infra that makes AI safe and performant.

What You’ll Own

Global Edge Routing

Design zone-routing that ensures <50ms SLA in 10+ regions
Build fallback orchestration to handle compliance-aware rollbacks

GPU Infra Ops

Maximize utilization across 100K+ GPUs via mesh & load prediction
Integrate compliance overlays with VaultChain and SHAP triggers

Reliability Telemetry

Ship `/vault/audit`, `/inference/predict`, `/compliance/log` endpoints
Trace every edge request across governance and model layers

Who You Are

Senior systems engineer with GPU fleet experience (KubeRay, Istio, Envoy)
Operated real-time AI infra with 10M+ QPS loads
Comfortable with compliance observability and infra governance

Why This Matters

Our competitive edge isn’t just AI — it’s defensible enforcement. This role turns that into product.

You’ll Leave This Role With

Referenceable contributions to enforcement infra that’s live in EU and US institutions
First-hand product work across legal, engineering, and GTM teams
Influence over how regulatory primitives become systems people trust

Logistics & Application

Start Date: Flexible (Q3–Q4 ideal)
Location: Remote-first; timezone overlap with NY or EU preferred
Compensation: Top of market salary + equity
To Apply: Send your resume and a sentence on the hardest infra problem you'd want to own at scale.

Why work at Elloe AI

Mission-driven culture: The team describes itself as “mission-first, values-led,” focused on making AI safe and trustworthy in critical domains.
Small, global team: With ~10 employees across three continents, new hires can have outsized impact and direct exposure to founders.
Flexible work options: Job postings list “In-Office or Remote” for most roles, with the Austin HQ operating on-site; typical time on-site is listed as “OnSite Workspace” for HQ, but remote arrangements appear available.
High-profile visibility: Being a Top 20 finalist at TechCrunch Disrupt 2025 gives the company industry recognition and a startup event pedigree.
Meaningful impact: The company’s own metrics (100K+ medical errors prevented, $20M+ biased loans blocked) suggest employees contribute to directly measurable social good.
Diverse, distributed team: Team members come from backgrounds including Cerner, SAP, Freshworks, Kenya Power, and others, indicating a mix of enterprise and startup experience.

About Elloe AI

Company Overview

One-liner: Elloe AI provides a real-time compliance layer that acts as an “immune system” for AI, making large language models safe to deploy across regulated industries.
Entity Type: Private (Pre-Seed, $1M total funding)
Headquarters: Austin, Texas, United States
Founded: 2022 (Pre-Seed round closed April 2022)
Founders: Owen Sakawa (CEO & Co-Founder), Jackson Mwaniki (CTO & Co-Founder), Aaron Madolora (CPO & Co-Founder)

Core Business

Primary industry: AI Safety & Governance, Enterprise Compliance, Cybersecurity
Target customers: Governments, hospitals, financial institutions, and other regulated enterprises (B2B)
Mission statement: “Build the immune system for AI—systems that make AI safe, truthful, and compliant in the real world.”

Products & Services

Elloe AI Platform: An API/SDK that sits on top of an LLM’s output layer, providing three “anchors”:
1. Fact-checking anchor – verifies responses against trusted sources.
2. Regulatory compliance anchor – checks for violations of HIPAA, GDPR, PII exposure, and other regulations.
3. Audit trail anchor – logs every decision, source, and confidence score for full traceability. The platform is not built on another LLM, but uses machine learning and human-in-the-loop oversight.

Market Standing

Valuation/Market Cap: Not publicly available
Key Metric: Annual Revenue ~$870K (company-reported); Total Funding $1M (Pre-Seed, led by Mad Ventures in April 2022)
Notable Investors/Partners: Mad Ventures (sole lead investor in Pre-Seed round); also acquired Saada (2022)
Growth Signals: Finalist (Top 20) in TechCrunch Disrupt 2025 Startup Battlefield; headcount growth trending negative (-16.7% YoY) but still hiring for multiple roles as of late 2025; offices opened in Nairobi and London; claims to have prevented 100,000+ medical errors, blocked $20M+ in biased loans, and protected 12M+ AI-driven risks.

Competitive Advantages

Not an LLM judging LLMs: Elloe AI’s system uses its own ML models and human analysts, avoiding the “Band-Aid on a wound” problem of having one LLM check another.
Multi-layer anchor system: Fact-checking, regulatory compliance, and full audit trail are built-in, not bolted on.
Regulatory expertise: The company keeps up with evolving global regulations (HIPAA, GDPR, etc.) and employs humans-in-the-loop to stay current.
Early trust in regulated verticals: Already deployed by governments, hospitals, and enterprises, giving them domain-specific credibility.

Strategic Focus

Deepen compliance capabilities: Expand coverage of regulatory frameworks and jurisdictions.
Global expansion: Offices in Austin, Nairobi, and London suggest a push into North America, Africa, and Europe.
Product development: Building out “anchors” for more use cases, and possibly edge inference for GPU environments (as seen in job postings for Principal Engineer – Distributed Systems).
Talent acquisition: Actively hiring for roles in engineering, design, product, policy, and communications – indicating a growth phase focused on scaling the team despite recent headcount dip.

Sources