Build and operate production-grade batch, streaming, CDC, and transformation pipelines across Azure and Microsoft Fabric. Develop lakehouse layers, dimensional models, SCDs, and data marts using SQL, Python, PySpark, and Spark. Implement data quality, metadata lineage, security, CI/CD, monitoring, and cost optimization. Support production incidents, documentation, runbooks, and knowledge transfer while complying with UAE data regulations.
Location: UAE-based (Travel required)
Duration: 6-month assignment
Summary (Data Engineer)
Timely, reliable data pipelines are what turn a platform design into business value. The Data Engineer builds and operates the pipelines that move data from source systems into the lakehouse and on to analytics and AI consumers, ensuring that the pipelines are automated, performant, secure and governed. The role works alongside the data architect, data quality engineer, business analysts and AI engineers, delivering production-grade pipelines on Azure / Microsoft Fabric, with metadata and lineage captured in Microsoft Purview.
Responsibilities
- Build batch, incremental, CDC and streaming ingestion pipelines from source systems into the Bronze layer, using Azure Data Factory and Fabric pipelines.
- Develop Silver- and Gold-layer transformations in SQL, PySpark and Spark notebooks, including cleansing, conformance, enrichment and business-rule logic.
- Implement dimensional models, SCDs, surrogate keys and data marts according to the architect's designs.
- Integrate DQ checks, validation, quarantine and reconciliation into pipelines in partnership with the Senior Data Quality Engineer.
- Ensure that pipelines publish metadata and lineage to Microsoft Purview.
- Implement scheduling, dependency management, parameterization, error handling and alerting.
- Optimize pipelines, partitioning, file formats (Delta/Parquet) and compute usage for performance and cost efficiency.
- Apply access controls, masking, encryption and secure credential management in line with UAE PDPL, the Dubai Data Law and DESC ISR.
- Maintain code in source control and deploy through CI/CD pipelines, with automated tests and environment promotion.
- Support production operations, incident resolution and runbooks, and coach counterparts so that the pipelines can be sustained after handover.
Must Have
- 4+ years in data engineering, preferably within UAE government or regulated-sector environments.
- Advanced SQL, Python and PySpark/Spark skills.
- Hands-on experience with Azure Data Factory, Microsoft Fabric, Databricks or Synapse.
- Strong understanding of Lakehouse/Medallion architecture, Delta/Parquet formats and dimensional modelling, including SCDs and surrogate keys.
- Experience with CI/CD (Azure DevOps or GitHub), source control and DataOps practices.
- Familiarity with Microsoft Purview metadata and lineage, and with DQ tooling.
- Awareness of UAE data regulation, including UAE PDPL, the Dubai Data Law and DESC ISR.
- Excellent English communication and documentation skills.
Good to Have
- Arabic proficiency is an advantage.
Similar Jobs
Information Technology
Develop and analyze Guidewire insurance data across policy, claims, and billing domains. Build complex SQL queries, perform data extraction, transformation, mapping, cleansing, loading, reconciliation, and validation. Investigate data quality issues, create analytical datasets and reports, and support testing for downstream reporting and analytics. Collaborate with business, Guidewire, and technical data teams.
Top Skills:
Guidewire BillingcenterGuidewire ClaimcenterGuidewire Cloud Data AccessGuidewire Data ExploreGuidewire Data HubGuidewire Data StudioGuidewire InfocenterGuidewire InsurancesuiteGuidewire PolicycenterSnowflakeSQL
Information Technology
Designs and manages scalable data platforms, pipelines, ETL/ELT frameworks, data warehouses, data lakes, and lakehouse architectures. Develops batch and real-time processing solutions using cloud-native services and big data technologies. Ensures data quality, governance, security, and compliance while optimizing large-scale data performance. Supports analytics, reporting, and AI/ML initiatives, contributes to architecture reviews, automates deployments through CI/CD, provides technical leadership, and mentors junior engineers.
Top Skills:
Amazon EmrAmazon RedshiftAmazon S3Apache KafkaSparkAws GlueAzure Data FactoryAzure Data Lake StorageAzure DatabricksAzure DevopsAzure Sql DatabaseAzure Synapse AnalyticsCi/CdDelta LakeGitGoogle BigqueryGoogle Cloud StorageGoogle DataflowHadoopJenkinsAzureMicrosoft FabricMySQLOraclePostgresPysparkPythonScalaShell ScriptingSnowflakeSQLSQL Server
Software
Design, develop, optimize, and productionize scalable ETL/ELT pipelines using Databricks, PySpark, SQL, GCP, BigQuery, and Delta Lake. Build agentic workflows for data engineering automation, implement data quality and monitoring controls, optimize workloads, and support testing, deployment, troubleshooting, and production operations. Collaborate with business, data engineering, and platform teams to identify automation opportunities.
Top Skills:
BigQueryDatabricksDelta LakeGCPPysparkSQL
What you need to know about the Delhi Tech Scene
Delhi, India's capital city, is a place where tradition and progress co-exist. While Old Delhi is known for its rich history and bustling markets, New Delhi is defined by its modern architecture. It's clear the region places a strong emphasis on preserving its cultural heritage while embracing technological advancements, particularly in artificial intelligence, which plays a central role in shaping the city's tech landscape, fueled by investments in research and development.


