M13 - Data Engineer

Scope of Work:

Data Pipeline Development & Management

• Design, build, and maintain robust data pipelines using AWS Glue
• Implement ETL/ELT processes for data ingestion from multiple sources
• Optimize data workflows for performance and scalability
• Monitor and troubleshoot data pipeline failures and performance issues

Data Infrastructure & Engineering (with IPOS IT Department)

• Manage and optimize AWS Redshift data warehouse operations
• Configure and maintain data storage solutions (AWS S3, data lakes)
• Implement data partitioning, indexing, and compression strategies
• Support Infrastructure as Code (IaC) for data infrastructure deployment

CI/CD & DevOps for Data (with AWS partners and IPOS IT Department)

• Develop and maintain CI/CD pipelines for data workflows using GitLab
• Implement automated testing for data pipelines and data qua...