---
title: 'Machine Learning Engineer, Speech - Joint Audio-Video Modeling at Cantina'
canonical: 'https://feeny.ai/job/machine-learning-engineer-speech-joint-audio-video-modeling-cantina-united-81jhte7m6h5q'
type: 'job'
last_seen: '2026-09-12'
---

# Machine Learning Engineer, Speech - Joint Audio-Video Modeling at Cantina

- **Company:** Cantina
- **Location:** United States / Europe
- **Employment:** full-time
- **Work type:** remote
- **Posted:** 2026-07-30
- **Last confirmed live:** 2026-09-12
- **Apply:** https://jobs.ashbyhq.com/cantina/0b1ec7c7-ca1f-4d86-9242-f6397f60ba34

## Job description

About Cantina: Cantina Labs is a social AI company, developing a suite of advanced real-time models that push the boundaries of expression, personality, and realism. We bring characters to life, transforming how people tell stories, connect, and create. We build and power ecosystems. Cantina, our flagship social AI platform, is just the beginning. If you're excited about the potential AI has to shape human creativity and social interactions, join us in building the future!

About the Role: We're looking for a Research / ML Engineer to join our Speech Team to build state-of-the-art speech and audio generation systems end-to-end from data specs through production inference with a focus on joint audio-video modeling. You'll own the audio side of multimodal generation: the representations (audio VAEs, neural codecs), the generative backbone (diffusion / flow-matching transformers), and the conditioning and alignment machinery that makes characters speak, sing, and emote in sync with what's on screen. That includes voice cloning and multi-speaker conditioning inside joint AV models, cinematic dialogue with music and sound design, and adjacent speech tasks (controllable TTS, voice conversion) that feed the same stack. You'll drive the model ↔ data ↔ eval flywheel, partnering closely with research, video, data, and infra to ship fast, reliable, and cost-aware models. In this role you'll work at the intersection of cutting-edge research and practical engineering, contributing to the development of safe, steerable, and trustworthy AI systems. You will thrive in this role if you:

- See research and engineering as two sides of the same coin and enjoy owning work end-to-end.
- Are excited to work across modalities and collaborate closely with a video generation team rather than staying inside audio.
- Are results-oriented, flexible, and willing to pick up whatever moves the needle.
- Like collaborating closely with infra, data, and product to ship measurable improvements.
- Enjoy designing experiments, listening tests, and metrics that correlate with user-perceived quality.
- Are eager to learn every day, and to find and solve unique large-scale problems.

What You’ll Do:

- Audio Representations: Design, train, and improve the audio VAEs, neural codecs, and vocoders our generative models sit on top of latent design, reconstruction and perceptual objectives, compression-vs-fidelity tradeoffs.
- Model Building: Architect, implement, pre-train, fine-tune, and post-train/alignment (e.g., GRPO/DPO) diffusion and flow-matching transformers for large-scale audio and video generation.
- Joint Audio-Video Modeling: Design the audio conditioning and cross-modal alignment inside joint AV models, audio latents alongside video latents, reference-audio and multi-speaker conditioning, multi shot generation audio/video modeling.
- Experimental Design: Design, run, and analyze scientific experiments to advance our understanding of the models.
- Data Ownership: Define data requirements and collaborate on acquisition, curation, AV-sync and quality filtering, annotation quality, and synthetic data strategies for paired audio-video and speech corpora.
- Rigorous Evaluation: Design automated objective/subjective evaluations audio fidelity and intelligibility metrics, AV-sync, listening and viewing tests, robustness & bias checks, and red-team studies.
- Inference Efficiency: Drive distillation, step-count reduction, quantization, and kernel/memory optimization to meet interactive latency and cost targets.
- Pipeline Delivery: Harden the training → evaluation → inference pipeline; profile latency, memory, and cost; and meet production SLAs with robust monitoring and rollback.
- GPU Scaling: Partner with infrastructure to run distributed training/inference on cloud fleets and productionize models with reliability and observability.
- Project Leadership: Independently lead small research projects while collaborating on larger team initiatives, including cross-team work with video generation.
- Tool Development: Develop and improve dev tooling to enhance team productivity.
- Safety & Responsibility: Contribute to safety/consent guardrails, watermarking, and misuse/abuse mitigation for responsible voice and likeness technology.

What You’ll Bring:

- Exceptional research/development experience with large-scale audio models (>8B parameters, >500k hours of data).
- Deep hands-on experience with diffusion and/or flow-matching transformers, including practical knowledge of samplers, schedules, conditioning mechanisms, and distillation.
- Deep hands-on experience training audio VAEs, neural audio codecs, and vocoders latent/tokenizer design, reconstruction and perceptual objectives, adversarial training.
- Strong experience with multi-node, multi-GPU distributed training (FSDP/DeepSpeed or equivalent).
- Strong software engineering skills with a proven track record of building complex systems.
- Strong with PyTorch and performance work (profiling, CUDA/Triton/C++ as needed) and writing reliable production-quality code.
- Shipped large-scale speech/audio or multimodal generative models to production.
- Background in working with large-scale ML data, and the ability to iterate on data and triangulate quality using both subjective and objective signals.
- Experience with voice cloning, speech control/steerability, or expressive speech generation.
- Notable publications and/or open-source contributions in speech/audio/ML.
- Strongly preferred:
- Experience with multimodal audio-video modeling: joint AV generation of multi-shot, multi-speaker scenes with dialogue, music, and sound design generated jointly with video, and the cross-modal alignment that keeps them in sync.
- Experience with video generation: video diffusion/flow-matching transformers, video VAEs, conditioned and multi-shot generation, building data pipelines for video models.
- Streaming or real-time generation, causal distillation (e.g., Self Forcing / Self Forcing++).

Compensation: The anticipated annual base salary range for this role is between $200,000-$220,000 (€170,000-€190,000). When determining compensation, a number of factors will be considered, including skills, experience, job scope, location, and competitive compensation market data.

Benefits for U.S.-based roles:

- Competitive salary and generous company equity
- Medical, dental, and vision insurance – 99.99% of premiums covered by Cantina
- 42 days of paid time off, including:
- 15 PTO days
- 10 sick days
- 15 company holidays
- 2 floating holidays
- Generous parental leave & fertility support
- 401(k) retirement savings plan
- Lifestyle spending account – $500/month to use however you’d like
- Complimentary lunch and snacks for in-office employees
- One Medical membership, and more!

## About Cantina

## Company Overview
- **One-liner**: Cantina is a social AI platform that lets users create, chat with, and share AI characters that talk, perform, and interact in real-time.
- **Entity Type**: Private (Seed Stage)
- **Headquarters**: San Francisco, California, United States
- **Founded**: 2023
- **Founders**: Not publicly listed; key leadership includes Co-Founder Prakash Ramakrishna

## Core Business
- **Primary industry**: Social AI / Social Media / Software Development
- **Target customers**: B2C (consumers), with a creator/developer ecosystem for building AI characters
- **Mission or purpose**: "Infinite Creativity Unlocked" — bringing AI characters to life to transform how people tell stories, connect, and create.

## Products & Services
- **Cantina App**: A social media platform where users chat with friends and AI, create video messages, share with friends, and "set bots free." The flagship product is a mobile-first experience focused on real-time AI character interaction.
- **Cantina AI Models**: A suite of advanced real-time models pushing the boundaries of expression, personality, and realism for AI characters.

## Market Standing
- **Valuation/Market Cap**: Not disclosed (private company)
- **Key Metric**: Total Funding — Seed Round (1 investor, amount undisclosed); 148 employees as of mid-2026
- **Notable Investors/Partners**: 1 seed investor (name not publicly disclosed); member of the Family Online Safety Institute (FOSI)
- **Growth Signals**:
  - Headcount grew 7.2% YoY (+29 people) to 148 employees
  - Monthly website traffic growth of +47.7% and yearly growth of +149.8%
  - Active job postings: 13 (yearly job posting growth of +44.4%)
  - Operates in 15 countries with 4 offices (San Francisco HQ, Sunnyvale, Brooklyn NY, and another Brooklyn location)
  - High LinkedIn follower growth (+9.9% yearly) reaching 22,462 followers

## Competitive Advantages
- **Real-time AI character technology**: Builds proprietary real-time models for expression, personality, and realism — a technical moat in the rapidly growing social AI space.
- **Creator ecosystem**: Allows users to build and release their own AI characters, creating a network effect and UGC flywheel.
- **First-mover in social AI**: One of the earliest platforms combining social networking with generative AI characters for mass consumer use.
- **Strong talent pool**: Employees recruited from top tech companies including Airtime (18), Aircore (26), Meta (4), Grammarly (4), TikTok (4), Amazon (3), and BeReal (4).

## Strategic Focus
- **Product expansion**: Actively hiring for Kotlin Multiplatform Engineer, iOS Engineer, Machine Learning Engineer (Images), and Media Software Engineer (Speech) — indicating a push toward cross-platform mobile, image generation, and speech capabilities.
- **Safety and trust**: Joined the Family Online Safety Institute (FOSI) in 2025 and invested heavily in trust & safety infrastructure, signaling a commitment to responsible AI.
- **Monetization and growth**: Hiring a Head of Product Marketing, Creator Partner Manager, and Product Managers for Video and Web — suggesting moves toward monetization, creator partnerships, and web-based experiences.
- **Research-driven**: Actively recruiting ML engineers and research talent, with 10% of workforce in Research roles.

## Why Work Here
- **Cutting-edge AI work**: Engineers work on real-time AI models for speech, images, and character interaction — at the intersection of generative AI and social media.
- **Strong technical culture**: 66 employees (15% of workforce) in Technical roles, with a tech stack including PyTorch, TensorFlow, Kubernetes, Docker, GCP, Snowflake, and modern mobile frameworks (Kotlin, Jetpack Compose, Swift).
- **Growth stage**: At 148 employees and 13 open roles, this is a growth-stage startup where new hires can have outsized impact.
- **Flexible locations**: Offices in San Francisco (HQ), Sunnyvale, and Brooklyn (two locations) — with a distributed workforce across 15 countries.
- **Creative, fun environment**: Company culture emphasizes creativity ("Minister of Bots" is a real title) and viral social experiences.
- **Notable perks**: Team has a dedicated Comedy Director and "Chief Horse Officer" — indicating a playful, unconventional culture.
- **High talent density**: Recruits from top AI and social media companies (Meta, TikTok, Grammarly, Amazon, Apple, Netflix alumni).

## Sources
1. [cantina.com](https://cantina.com/)
2. [LinkedIn - Cantina Labs](https://www.linkedin.com/company/cantinaai)
3. [Cantina Careers (Ashby)](https://jobs.ashbyhq.com/cantina)
4. [Cantina Careers Page](https://cantina.com/careers)

## Other roles at Cantina

- [Research Intern](https://feeny.ai/job/research-intern-cantina-singapore-0t2dg9nfd7y1) — Singapore
- [Machine Learning Intern](https://feeny.ai/job/machine-learning-intern-cantina-singapore-wnsca7rtewty) — Singapore
- [Product Manager, Growth - Lifecycle](https://feeny.ai/job/product-manager-growth-lifecycle-cantina-san-francisco-fs4n56y79qc2) — San Francisco, CA
- [Payments & Risk Operations Manager](https://feeny.ai/job/payments-risk-operations-manager-cantina-bay-area-8vxjey1a9qp4) — Bay Area, OR
- [Media Software Engineer, Speech (Senior-Staff Levels)](https://feeny.ai/job/media-software-engineer-speech-senior-staff-levels-cantina-sunnyvale-amr9rvpwq2gp) — Sunnyvale, CA
- [Machine Learning Engineer - Voice Conversion](https://feeny.ai/job/machine-learning-engineer-voice-conversion-cantina-united-states-europe-fpbamgtxrfs0) — United States / Europe
- [Machine Learning Engineer, Ops](https://feeny.ai/job/machine-learning-engineer-ops-cantina-united-states-europe-zr6h1nf499dk) — United States / Europe
- [Director, Brand Marketing](https://feeny.ai/job/director-brand-marketing-cantina-los-angeles-dhngs7psnehq) — Los Angeles, CA
- [Senior Creative Strategist, Performance Marketing](https://feeny.ai/job/senior-creative-strategist-performance-marketing-cantina-remote-gxzcxj5fv99e)
- [Staff Software Engineer, Backend](https://feeny.ai/job/staff-software-engineer-backend-cantina-los-angeles-y3jk4ycetkqy) — Los Angeles, CA / San Francisco, CA