Senior Data Engineer - Unity Catalog

KPI Partners is seeking a skilled Senior Data Engineer with expertise in Databricks, Unity Catalogue, Pyspark, Python, and SQL. As a member of our dynamic team, you will be responsible for designing, developing, and maintaining scalable data pipelines and solutions that support our data-driven decision-making processes. Key Responsibilities: Develop and maintain data pipelines using Databricks and Pyspark to ensure efficient data processing and transformation. Utilize Unity Catalogue to manage and govern data assets and ensure data quality and security. ollaborate with data analysts, data scientists, and other stakeholders to understand data requirements and deliver actionable insights. Write efficient and optimized queries using SQL to extract and manipulate data from various sources. Monitor and troubleshoot data pipeline performance, implement improvements, and provide support for data-related issues. Document data engineering processes, architecture, and workflows to ensure knowled...