Research Engineer, Web Crawling

What you'll do

• Design and scale crawling and ingestion infrastructure, including specialized crawlers for difficult or high-value sources.
• Build extraction, deduplication, and data-quality filtering pipelines; improve reliability and efficiency at petabyte scale.
• Work with the pretraining team to assess how crawled data affects model performance and help set technical direction.

What they ask for

• 8+ years designing, building, and scaling web crawlers, scrapers, or large-scale distributed data-acquisition systems.
• Track record of owning crawler or data-acquisition infrastructure at internet scale.
• Strong software engineering skills in a language such as Python, Go, or Rust, with real experience in distributed systems.
• Working knowledge of practical and legal considerations of large-scale web data collection, including robots.txt, rate limiting, and licensing.

Nice to have

• Experience applying machine learning to crawl selection, extraction, or data-quality classification at internet scale.
• Experience setting technical direction for relevant infrastructure teams; petabyte-scale systems experience, open-source contributions, or search-engine/frontier-AI data-acquisition background.

Pay

$350,000-$475,000 USD (base salary)

Tools

Python, Go, Rust, robots.txt

AI focus

The job prepares Inkling's pretraining data and works with the pretraining team to understand how crawled data affects model performance. Preferred experience includes applying machine learning to crawl selection, extraction, or data-quality classification.