Azure Databricks Systems Specialist_2567

Duties and Responsibilities

Build and optimize scalable data pipelines using Databricks, PySpark, Spark, and Azure Data Factory.

• Analyse, clean, and organize raw data from multiple structured and unstructured sources.

• Develop and maintain ETL/ELT workflows following best engineering and DevOps practices.

• Implement CI/CD pipelines using GitHub and Azure DevOps.

• Deploy and orchestrate workloads using Docker and Kubernetes.

• Develop solutions aligned to enterprise data models, DataVault methods, and architectural standards.

• Ensure data quality, data governance, GDPR compliance, and metadata management.

• Use tools like QuerySurge for data testing and validation.

• Collaborate with teams using Agile/Scrum frameworks and tools like JIRA, Confluence, and SharePoint.

• Support production deployments, troubleshooting, and operational...