MTS - Pre-Training Data & Acquisition Engineer

THE ROLE

We’re hiring a Pre-Training Data & Acquisition Engineer to build the data systems powering Prometheus’s foundation models for the physical world. You’ll work closely with research and infrastructure teams to acquire, process, and deliver large-scale training datasets across engineering, scientific, and multimodal domains. This role spans distributed crawling, source integration, data processing, and production operation, with end-to-end ownership from raw content to training-ready datasets.

WHAT YOU’LL DO

- Identify and integrate valuable data sources across engineering and scientific domains.

- Build distributed crawlers, API integrations, and bulk ingestion systems with effective scheduling, rate limiting, retries, and incremental updates.

- Develop pipelines for parsing, extraction, normalization, deduplication, quality filtering, and tokenization across heterogeneous formats.

- Optimize throughput and cost across networking, compute, storage, and databases as acquisition and processing workloads scale.

- Build monitoring and tooling to track source coverage, ingestion failures, processing throughput, and usable data yield.

- Work closely with pre-training researchers to translate data requirements into reliable pipelines and deliver datasets ready for large-scale training.

- Own dataset reproducibility, versioning, provenance, and recovery from acquisition through delivery.

WHAT WE’RE LOOKING FOR

- Experience building and operating large-scale distributed systems, web crawlers, or data processing pipelines.

- Strong programming ability in Python and Rust, Go, C++, or a comparable systems language.

- A practical understanding of web infrastructure, including HTTP, DNS, concurrency, caching, and common failure modes.

- Hands-on experience with databases, object storage, and distributed batch or streaming processing.

- Experience designing fault-tolerant systems that handle partial failures, resume interrupted work, and prevent unintended duplication or data loss.

- Ability to profile and debug performance across CPU, memory, disk, and network usage.

- Strong technical judgment when integrating unfamiliar sources, APIs, and file formats.

- Bias toward fast iteration and end-to-end ownership, from initial implementation through reliable production operation.

- Experience with search indexing, document extraction, or foundation-model data pipelines is a plus.

WHY JOIN US

- Work with world-class researchers on frontier AI systems for the physical world.

- Build the acquisition and processing systems that supply engineering, scientific, and multimodal data to large-scale model training.

- Competitive compensation and flexible work arrangements.

- High-impact, mission-driven environment.