Lead Data Engineer

Description:

The Lead Data Engineer is responsible for delivery of data solutions in Intellihub’s new Enterprise Data Platform. You will also be responsible to ensure the data pipeline design and implementation are maintainable, cost optimised and secured.

As a Lead Data Engineer, your close collaboration with our Data platform team and architects will be instrumental in driving the successful delivery of our highest priority projects while ensuring top-notch quality. Your role within the leadership team of Data and Analytics will be pivotal in shaping our Data and Platform Strategy.

You will hold responsibility for establishing and enhancing Data engineering practices, encompassing principles, standards, templates, and processes, along with necessary governance and review mechanisms. This includes spearheading the design, development, and operationalisation of data pipelines and integration frameworks that form the backbone of Intellihub’s enterprise data platform.

Operating within a robust DataOps framework, you will oversee the maintenance and support of all products and services developed by the team. Moreover, your role involves elevating the team's development and testing standards, amplifying the overall excellence of our Data Engineering practices.

About You
 

• Proactive, positive, flexible, and resilient
• Passionate about data, software engineering and innovation
• Demonstrated capacity in effective leadership, encompassing mentoring, coaching, and cultivating a high-achieving data engineering team.
• Proficient in communication, adept at translating technical concepts for audiences of varying technical backgrounds.
• Experience in Agile Program Increment / Scaled Agile Framework, backlog grooming to enable continuous delivery.
• Highly motivated to identify and solve problems.
• Extensive proficiency in code management and build tools like Git, Bitbucket, GitHub, AWS CodeCommit, Maven, Gradle, and SBT.
• Hands-on expertise in utilising CI/CD deployment tools (e.g. Jenkins, AWS CodePipeline).
• Experience in implementing data pipelines and solutions on cloud platforms (e.g. AWS, Databricks, Spark, DBT)
• Experience in Data Warehouse, Data Lake and/or Data Lakehouse implementation
• Expertise in SQL and Python
• Knowledge in Data modelling techniques and practices used for Data Warehouse and Data Mart application
• In depth knowledge of large-scale data sets including both structured and unstructured data
• Implementation experience with DevOps and agile delivery (code management, CI/CD, modular design)
• Tertiary qualified with a degree in Information Technology or Engineering, or a related discipline