---
title: 'ML Systems Engineer, Infrastructure & Cloud at Basis Research Institute'
canonical: 'https://feeny.ai/job/ml-systems-engineer-infrastructure-cloud-basis-research-institute-new-york-fmkf7rgb6w6r'
type: 'job'
last_seen: '2026-09-10'
---

# ML Systems Engineer, Infrastructure & Cloud at Basis Research Institute

- **Company:** Basis Research Institute
- **Location:** New York, NY
- **Employment:** full-time
- **Posted:** 2025-11-23
- **Last confirmed live:** 2026-09-10
- **Apply:** https://jobs.ashbyhq.com/basis-research/51ebb571-246c-4913-84da-b924341f8a26

## Job description

## About Basis

[Basis](https://www.basis.ai) is a nonprofit applied AI research organization with two mutually reinforcing goals. The first is to understand and build intelligence. This means to establish the mathematical principles of what it means to reason, to learn, to make decisions, to understand, and to explain; and to construct software that implements these principles. The second is to advance society’s ability to solve intractable problems. This means expanding the scale, complexity, and breadth of problems that we can solve today, and even more importantly, accelerating our ability to solve problems in the future. To achieve these goals, we’re building both a new technological foundation that draws inspiration from how humans reason, and a new kind of collaborative organization that puts human values first.

## About the Role

ML Systems Engineers at Basis ensure training and evaluation infrastructure is fast, reliable, and scalable. You will own the full stack from distributed training frameworks through cloud administration, making it possible for researchers to iterate quickly on complex models while managing computational resources efficiently. We are looking for engineers who combine deep understanding of ML systems with operational excellence. The ideal ML Systems Engineer has experience with distributed training at scale, understands the intricacies of debugging numerical instabilities, and can manage cloud infrastructure that scales from experiments to production. You will be the guardian of training stability, the optimizer of compute costs, and the enabler of reproducible research. This role spans traditional ML engineering and cloud/DevOps responsibilities. You will manage GPU clusters, optimize cloud spending, ensure security and compliance, and build the infrastructure that lets researchers focus on algorithms rather than operations. We seek individuals who aspire to build robust ML infrastructure, maintain “logbook culture” for documenting issues and solutions, and treat operational excellence as a first-class concern.

We expect you to:

- Have demonstrated expertise in ML systems engineering. Examples include:
- Managing distributed training jobs across hundreds of GPUs
- Debugging and fixing numerical instabilities in large-scale training
- Building infrastructure for reproducible ML experiments
- Optimizing training throughput and resource utilization
- Possess deep knowledge of distributed training frameworks including PyTorch/JAX distributed strategies (DDP, FSDP, ZeRO), gradient accumulation, mixed precision training, and checkpoint/recovery systems.
- Have strong cloud administration skills including AWS/GCP/Azure services, infrastructure as code (Terraform), Kubernetes orchestration, cost optimization, security best practices, and compliance requirements.
- Understand the full ML stack from hardware (GPUs, interconnects, storage) through frameworks (PyTorch, JAX) to high-level training loops and evaluation pipelines.
- Be skilled at debugging complex failures across the stack—GPU/NCCL issues, data loading bottlenecks, memory leaks, gradient explosions, and convergence problems.
- Value documentation and knowledge sharing. You maintain comprehensive logs of issues encountered, solutions found, and lessons learned, building institutional knowledge.
- Progress with autonomy while coordinating closely with researchers. You can anticipate infrastructure needs, prevent problems before they occur, and respond quickly when issues arise.

In addition, the following would be an advantage:

- Experience at organizations training large models (OpenAI, Anthropic, Google, Meta).
- Background in both ML research and production systems.
- Contributions to ML frameworks or distributed training libraries.
- Experience with on-premise GPU cluster management.
- Knowledge of optimization theory and numerical methods.
- Understanding of robotics-specific infrastructure requirements.

Responsibilities:

- Own distributed training infrastructure including job launchers, checkpointing systems, recovery mechanisms, and monitoring that ensures experiments run reliably at scale.
- Debug and resolve training failures by diagnosing issues across GPUs, networking, numerics, and data pipelines, maintaining detailed logs of problems and solutions.
- Profile and optimize training performance by identifying bottlenecks in data loading, gradient computation, communication overhead, and implementing solutions that improve step time.
- Manage cloud infrastructure and costs including capacity planning, spot instance strategies, storage optimization, and building tools that give researchers visibility into resource usage.
- Implement security and compliance measures including access controls, data encryption, audit logging, and ensuring infrastructure meets requirements for handling sensitive data.
- Build evaluation and benchmarking infrastructure that enables consistent, reproducible measurement of model performance across different conditions and datasets.
- Develop monitoring and alerting systems that detect anomalies in training metrics, resource utilization, or system health, enabling rapid response to issues.
- Maintain development environments including containerization, dependency management, and tools that ensure researchers can reproduce results across different systems.
- Document and share knowledge through runbooks, post-mortems, and training materials that help the team understand and operate ML infrastructure effectively.
- Collaborate with researchers to understand requirements, suggest infrastructure solutions, and ensure systems support rather than constrain research goals.

## Role Details

Exceptional candidates who may not meet all of the following criteria are still encouraged to apply.

- FT/PT: Full-time.
- In-person Policy: We are in the office four days a week. Be prepared to attend multi-day Basis-wide in-person events.
- Location: New York City or Cambridge, MA.
- Salary range: Competitive salary.

Non-Discrimination Notice Basis Research Institute provides equal employment opportunities without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, or genetics and prohibits discrimination based on all protected characteristics.

Privacy Notice By submitting your application, you grant Basis permission to use your materials for both hiring evaluation and recruitment-related research and development purposes. Your information may be processed in different countries, including the US. You retain copyright while providing Basis a license to use these materials for the stated purposes. Read our full Global Data Privacy Notice [here](https://www.basis.ai/join-us/privacy-notice/).

## About Basis Research Institute

## Company Overview
- **One-liner**: Basis Research Institute is a nonprofit applied research organization advancing artificial intelligence through probabilistic modeling and causal inference to solve society's most intractable problems, building open-source software that draws inspiration from human reasoning.
- **Entity Type**: Private (Nonprofit 501(c)(3) Research Institute)
- **Headquarters**: New York City, New York, USA (with additional office in Cambridge, Massachusetts, USA)
- **Founded**: 2022
- **Founders**: Zenna Tavares, Emily Mackevicius, and Eli Bingham

## Core Business
- **Primary industry**: Artificial Intelligence Research & Development
- **Target customers**: Municipal governments (via consulting), academic collaborators, for-profit companies (via contract work), and the broader open-source AI community
- **Mission**: Two mutually reinforcing goals: (1) advance basic scientific research in artificial intelligence to understand and build intelligence, and (2) help solve society’s intractable problems by expanding the scale, complexity, and breadth of problems that can be solved

## Products & Services
- **Core Technology Development**: Advanced probabilistic modeling and causal inference frameworks implemented as open-source software, including causal probabilistic programming languages
- **Applied Challenge Projects**:
  - **AI-Driven Robot Design**: Using AI to automate and accelerate robot design and control
  - **Dynamical Modeling for Biological Systems**: Modeling complex biological processes (e.g., neuroscience, cellular dynamics)
  - **Participatory City Modeling**: Consulting with municipal governments to model urban systems and support data-driven policy decisions
  - **Modeling, Abstraction, and Reasoning Agent (MARA)**: A research project focused on building AI agents capable of abstract reasoning and world modeling
- **Postdoctoral Fellowship Program**: A fellowship for early-career scientists to direct their own research within Basis’s mission
- **Open-Source Software Releases**: Publishing research tools and libraries for the broader AI community

## Market Standing
- **Valuation/Market Cap**: Not applicable (nonprofit research organization)
- **Key Metric**: Funded through a combination of research grants, private philanthropy, and contract work with partners; specific funding amounts are not publicly disclosed
- **Notable Investors/Partners**: Supported by an advisory board including Armando Solar-Lezama (MIT), Joshua Tenenbaum (MIT), Kevin Ellis (Cornell), Anthony Philippakis (Broad Institute), and Rui Costa (Columbia University). Collaborates with academic institutions (MIT, Columbia, Cornell, NYU) and municipal governments
- **Growth Signals**: Headcount has grown 41.7% year-over-year (LinkedIn data); active job postings increased 107.1% year-over-year; expanding into London with a GTM Lead role; 29 active job openings as of mid-2026

## Competitive Advantages
- **Nonprofit Structure**: Unlike for-profit AI labs, Basis is a 501(c)(3) nonprofit, enabling it to prioritize open-source, public-good research over commercial outcomes
- **Unique Research Cycle**: A structured cycle between core technology development and applied challenge projects, where insights from real-world problems feed back into foundational research
- **Interdisciplinary Approach**: Combines insights from cognitive science (human reasoning), computer science (probabilistic programming), and domain expertise (biology, urban planning, robotics)
- **High-Profile Advisory Board**: Advisors from leading institutions (MIT, Cornell, Broad Institute, Columbia) provide deep academic and scientific credibility
- **Open-Source Commitment**: All software is released as open-source, building community trust and enabling broad adoption

## Strategic Focus
- **Core AI Foundations**: Advancing probabilistic modeling, causal inference, program synthesis, neuro-symbolic methods, reinforcement learning, and world models
- **Challenge-Driven Research**: Using applied projects (robotics, biology, city modeling) to identify shared bottlenecks that guide further methodological research
- **Collaborative Expansion**: Actively seeking partnerships with academia, government, social sector, and industry to tackle high-impact problems
- **Geographic Growth**: Opening presence in London (UK/EU GTM) and expanding roles in Ithaca, NY, while maintaining NYC and Cambridge hubs
- **Open-Source Ecosystem**: Building durable foundations that enable other researchers and practitioners to build upon Basis’s work

## Why Work Here
- **Mission-Driven Culture**: Employees work on problems that matter beyond publications, with impact on AI science and real-world societal challenges
- **Autonomy and Growth**: Organizational structure allows employees to see their autonomy and responsibilities grow with their projects, distinct from traditional career ladders
- **Collaborative Environment**: Work alongside researchers from top institutions (MIT, Columbia, Cornell) and domain experts from academia, government, and industry
- **Open-Source Impact**: Ship high-quality open-source software used by thousands, contributing to the broader AI community
- **Locations**: Offices in NYC (primary) and Cambridge, MA, with remote flexibility for some roles; London expansion underway
- **Notable Perks**: Postdoctoral fellowships available for early-career scientists; focus on human values and collaborative organization
- **Engineering Culture**: Emphasis on building durable foundations, shipping production-quality code, and working at the intersection of research and engineering

## Sources
1. [basis.ai/about](https://www.basis.ai/about/)
2. [basis.ai](https://www.basis.ai/)
3. [jobs.ashbyhq.com/basis-research](https://jobs.ashbyhq.com/basis-research)
4. [basis.ai/join-us](https://www.basis.ai/join-us/)
5. [LinkedIn - Basis Research Institute](https://www.linkedin.com/company/basis-ri)

## Other roles at Basis Research Institute

- [Research Operations Manager](https://feeny.ai/job/research-operations-manager-basis-research-institute-new-york-95tarn9pj8tz) — New York, NY
- [GTM Lead, UK/EU & US](https://feeny.ai/job/gtm-lead-uk-eu-us-basis-research-institute-london-b9sq0m016y20) — London, United Kingdom
- [Hardware Systems & Lab Engineer, Robotics & Operations](https://feeny.ai/job/hardware-systems-lab-engineer-robotics-operations-basis-research-institute-new-g6a4cmtj6jzd) — New York, NY
- [Data Engineer, Platform](https://feeny.ai/job/data-engineer-platform-basis-research-institute-new-york-z7jf6pnk7yr3) — New York, NY
- [Research Engineer, Operations](https://feeny.ai/job/research-engineer-operations-basis-research-institute-new-york-55vvr38p29sn) — New York, NY
- [Research Engineer, Platform](https://feeny.ai/job/research-engineer-platform-basis-research-institute-new-york-3sep3r6cksy6) — New York, NY
- [Research Scientist, Program Synthesis & Neuro-symbolic Methods](https://feeny.ai/job/research-scientist-program-synthesis-neuro-symbolic-methods-basis-research-yw82mzv4975d) — New York, NY
- [Research Scientist, Reinforcement Learning](https://feeny.ai/job/research-scientist-reinforcement-learning-basis-research-institute-new-york-gprjb8ja06q7) — New York, NY
- [Research Scientist, World Models](https://feeny.ai/job/research-scientist-world-models-basis-research-institute-new-york-xwd9d07xy13d) — New York, NY
- [Postdoctoral Fellow, Open Call](https://feeny.ai/job/postdoctoral-fellow-open-call-basis-research-institute-new-york-gpaxsvg2kqhc) — New York, NY