We are seeking a highly skilled Senior Data Engineer with extensive experience in designing, building, and optimizing high-volume data pipelines. The ideal candidate will have strong expertise in Python, Databricks on Azure Cloud services, DevOps, and CI/CD tools, along with a solid understanding of AI/ML techniques and big data processing frameworks like Apache Spark and PySpark.
Responsibilities
-
Adhere to coding and Numerator technology standards
-
Build suitable automation test suites within Azure DevOps
-
Maintain and update automation test suites as required
-
Carry out manual testing, load testing, exploratory testing as required
-
Work closely with Business Analysts and Senior Data Developers to consistently achieve sprint goals
-
Assist in estimation of sprint-by-sprint stories and tasks
-
Pro-actively take a responsible approach to product delivery
Requirements
-
7-10 years of experience in data engineering roles
-
Good Python skills
-
Experience working with Microsoft Azure Cloud
-
Experience in Agile methodologies (Scrum/Kanban)
-
Experience with Apache Spark, PySpark, Databricks
-
Experience working with Devops pipeline, preferably Azure DevOps
Preferred Qualifications
-
Bachelor's or master's degree in computer science, Information Technology, Data Science, or a related field.
-
Experience working in a support focused role
-
Knowledge/experience in AI ML techniques
-
Certification in relevant Data Engineering discipline or related fields.
