Data Platform Engineer (Member of Technical Staff)
Cogent SecurityVulnerability Management company
San Francisco, United StatesLead
Bain Capital Ventures
Greylock Partners
Definition Capital
Lockstep
OpenAI Group PBC
Abnormal Security Corp.
Data & AI
About the role
TL;DR
Design and implement a data platform for GenAI systems.
- •Design and implement Cogent's data platform and integration pipeline to ingest, normalize, and transform enterprise data for GenAI systems.
- •Key Responsibilities Build data connectors, pipelines, and ETLs to ingest enterprise data securely.
- •Design indexing, storage, and query strategies for semantic search and RAG.
- •Architect scalable data platform components and make build vs buy decisions.
- •Work with Applied AI teams to taxonomize, correlate, and deduplicate data.
- •Requirements 5+ years as a hands-on engineer and technical leader.
- •Experience building data pipelines for information retrieval systems.
- •Familiarity with ETL orchestration (Temporal, Airflow) and streaming (Kafka, Flink).
- •Expertise in SQL, database fundamentals, and distributed computing.
- •Experience with Kubernetes, Terraform, Docker, and multi-cloud deployments.
Required skills
SQLAirflowKubernetesTerraformDockerPython
Domain expertise
cybersecurityai
Tech stack
PythonSQLAirflowKubernetesTerraformDockerAWSGoogle CloudAzure