Shield AI
Staff Engineer, Data Platform (R5659)
San Diego, California · Staff · Full-time · Posted Nov 2, 2026
About the job
Build and lead data platform services that help machine-learning engineers discover, version, access, and reproduce datasets across development and deployment workflows.
What you'll do
- Build dataset discovery, catalog, indexing, and retrieval services, including LanceDB integration.
- Develop APIs, metadata, provenance, and schema evolution mechanisms for reliable, authorized access to exact dataset versions.
- Partner with machine-learning engineers to turn data needs into reusable, tested services and lead technical design.
What they ask for
- Experience designing and implementing production data platforms or data-intensive backend services.
- Strong software engineering skills in Python or a comparable backend language.
- Experience with data indexing, versioning, metadata, storage, access control, and evolving schemas or APIs.
- Ability to lead technical design, implement substantial system components, and translate user needs into reliable services.
Nice to have
- Experience with LanceDB, Arrow, Parquet, or comparable columnar and vector-data systems.
- Experience with dataset catalogs, provenance, reproducible machine-learning workflows, domain models or ontologies, or Kubernetes and disconnected environments.
Tools
Python, LanceDB, Arrow, Parquet, Kubernetes
AI focus
Supports AI and machine-learning development by enabling engineers to find trusted training data, reproduce model-training results, and improve autonomy systems.
Model-written summary · the original posting is authoritative