Design and optimize data pipelines using Azure Databricks and Spark, ensuring data reliability and quality through ETL/ELT workflows and CI/CD automation.
Key Responsibilities Design, develop, and optimize data pipelines using Azure Databricks. Implement ETL/ELT workflows leveraging Spark (PySpark/Scala). Integrate and manage data from multiple sources such as Azure Data Lake, SQL DB, Synapse, and external APIs. Design and enforce data quality rules, validation checks, and monitoring frameworks. Work with data quality tools to profile, cleanse, and validate data. Ensure data pipelines follow best practices for performance, scalability, and cost optimization. Collaborate with data engineers, analysts, and business stakeholders to ensure data reliability. Implement CI/CD pipelines and automation for Databricks deployments. Maintain documentation for data processes, pipelines, and quality rules.
Similar Jobs
Information Technology • Consulting
Design, build, and optimize scalable data pipelines and analytics solutions on Snowflake. Collaborate with teams for data products and enforce security and governance.
Top Skills:
SnowflakeSQL
Information Technology • Consulting
Design and optimize scalable data pipelines and analytics platforms using Snowflake, Airflow, and SQL. Ensure data quality and drive performance optimization.
Top Skills:
AirflowAWSAzureDbtDebeziumDockerFivetranGCPGitKafkaKinesisKubernetesPub/SubPythonSnowflakeSQLTerraform
Information Technology • Consulting
The role requires leading big data projects with hands-on experience in Azure Databricks, Pyspark coding, advanced SQL, and data warehouse management.
Top Skills:
Azure DatabricksPysparkPythonSQL
What you need to know about the Delhi Tech Scene
Delhi, India's capital city, is a place where tradition and progress co-exist. While Old Delhi is known for its rich history and bustling markets, New Delhi is defined by its modern architecture. It's clear the region places a strong emphasis on preserving its cultural heritage while embracing technological advancements, particularly in artificial intelligence, which plays a central role in shaping the city's tech landscape, fueled by investments in research and development.
