Exa
Software Engineer, Web Crawling
$180K–$350K
San Francisco, California · On-site · Full-time · Posted Jul 23, 2025
- Visa sponsorship
- The company is happy to sponsor international candidates, including STEM OPT, OPT, H1B, O1, and E3; sponsorship is not guaranteed.
About the job
Build and scale web-crawling systems to collect and prioritize web content for Exa’s search engine.
What you'll do
- Build a distributed crawler capable of handling 100M+ pages per day.
- Optimize crawl politeness and rate limiting across thousands of domains.
- Develop systems for dynamic content, JavaScript rendering, anti-bot measures, and crawl scheduling.
What they ask for
- Extensive experience building and scaling web crawlers, or willingness to ramp up very quickly.
- Experience with a high-performance language such as C++ or Rust.
- Familiarity with TypeScript, Playwright, modern web design, and CDP (Chrome DevTools Protocol).
- Comfort optimizing systems to an exceptional degree.
Pay
$180K – $350K (base salary)
Tools
C++, Rust, TypeScript, Playwright, CDP (Chrome DevTools Protocol)
AI focus
Build web-crawling infrastructure that collects information for Exa’s AI-powered search engine.
Model-written summary · the original posting is authoritative