The role involves gathering data requirements, leading a data engineering team, writing complex SQL queries, creating data pipelines in Azure Data Factory, and using Databricks for data processing. The candidate should have strong communication skills and be adaptable to changing project needs.
Responsibilities
- Able to participate in business discussions and assist gathering data requirements. Good analytical and problem-solving skills to help address data challenges by leading a team
- Proficiency in writing complex SQL queries for data extraction, transformation, and analysis. Knowledge of SQL functions, joins, subqueries, and performance tuning. Able to navigate source systems with minimal guidance to understand how data is related and use like data profiling to gain a better understanding of the data. Hands on experience with PySQL/Pyspark etc.
- Hands on Experience in creating and managing data pipelines using Azure Data Factory. Understanding of data integration, transformation, and workflow orchestration in Azure environments.
- Knowledge of data engineering workflows and best practices in Databricks. Able to understand existing templates and patterns for development. Hands on experience with Unity Catalog and Databricks workflow.
- Proficiency in using Git for version control and collaboration in data projects. Ability to work effectively in a team environment, especially in agile or collaborative settings.
- Clear and effective communication skills to articulate findings and recommendations for other team members. Ability to document processes, workflows, and data analysis results effectively.
- Willingness to learn new tools, technologies, and techniques as the field of data analytics evolves. Being adaptable to changing project requirements and priorities.
Skills
- 9+ years of overall experience with more than 7+ years of expertise in Azure technologies
- Ability to envision/lead E2E solution and solve technical issue during offshore.
- Azure Databricks, Data Lakehouse architectures, and Azure Data Factory.
- Expertise in optimizing data workflows and predictive modeling.
- Designing and implementing data pipelines using Databricks, Spark,
- Expertise in batch and streaming data solutions, automating workflows with CI/CD tools like Jenkins and Azure DevOps, and ensuring data governance with Delta Lake
- Spark, PySpark, Delta Lake, Azure DevOps, Python.
Top Skills
Pyspark
Pysql
Python
SQL
Similar Jobs
.png)
Be an Early Applicant
As a Data Engineer, you will develop and optimize data pipelines, design data storage solutions, ensure data quality, monitor workflows, and collaborate with teams to implement comprehensive data solutions using technologies like Java and SQL.
Be an Early Applicant
Design and build data pipelines using Apache Beam in Java and Apache Spark on Google Cloud Platform. Collaborate with stakeholders for data integration, manage cloud infrastructure, tune performance, and document technical processes while staying updated with the latest technologies.
Be an Early Applicant
As a Data Engineer at Equifax, you will focus on extracting, transforming, and loading data from diverse sources using Big Data technologies and GCP services. You will develop and support data ingestion jobs, work on optimizing queries, and apply modern software development practices in a hybrid work environment.
What you need to know about the Delhi Tech Scene
Delhi, India's capital city, is a place where tradition and progress co-exist. While Old Delhi is known for its rich history and bustling markets, New Delhi is defined by its modern architecture. It's clear the region places a strong emphasis on preserving its cultural heritage while embracing technological advancements, particularly in artificial intelligence, which plays a central role in shaping the city's tech landscape, fueled by investments in research and development.
