Description:
The Azure Data Engineer will be responsible for developing and maintaining the data pipelines, data models and standards for various Data Integration & Data Warehouse projects in Azure. You will ensure new features and subject areas are modelled to integrate with existing structures and provide a consistent view. In addition, you will develop and maintain the documentation of the data architecture, data flow, and data warehouse data models appropriate for both technical and business audiences.
Responsibilities Include But Are Not Limited To
- Provide technical expertise in Azure Data Platform;
- Develop ELT pipelines in and out of data lake using Azure Data Factory, Databricks and Delta Lake.
- Promote changes between environments using Azure Devops Pipelines.
- Integrate on-premise infrastructure with public cloud AZURE infrastructure.
- Partner with the Data Architects, Product Managers, Data Modellers, Business Users and Scrum Masters to deliver data integrations and BI solutions required for various projects.
- Review the work provisioned by vendors and ensure it is compliant with Sydney Water guidelines; and
- Proactively identify improvements to the way of working within the team and with stakeholders to achieve a better outcome for the Sydney Water.
About You
- Bachelor’s Degree in computer science or Engineering or similar (preferred);
- Azure Data Engineer Certification;
- Minimum 4+ years’ experience in IT and 4+ years of hands-on experience working as an Azure Data Engineer, and 4+ years in Data Warehousing, ETL/ELT, BI, and Analytics projects.
- Extensive design and implementation experience in Azure cloud data platform and modern data warehousing, including data security, Azure Logging and Monitoring, Azure Databricks, Azure Blob Storage, Azure Data Lake, Azure Data Factory, Azure SQL Database, Azure Devops;
- Expertise in data modelling, ELT using ADF, implementing complex views, stored procedures and standard DWH and ETL/ELT concepts.