Lead Pyspark / SQL Engineer - Threat Detection & DataBricks

Job Summary: We are seeking a skilled PySpark / SQL Engineer to support our Threat Detection team in building and migrating security analytics pipelines using Databricks. This role will focus on a platform migration project, moving detection rules and associated content from a legacy system into Databricks' native detection framework. You will be responsible for creating equivalent PySpark log pipelines, rule configuration files, unit tests, and data validation checks, and deploying these pipelines using Terraform. A strong background in data engineering, particularly with large-scale log analytics, is essential.

Requirements:

• 5-10+ years of experience as a PySpark/SQL Engineer, with a strong focus on data engineering and analytics.


• Prior experience in building threat detection or log analytics pipelines using PySpark, SQL, and Databri...