Design, build, and maintain scalable ELT/data pipelines and connectors using Airflow, Python, and PySpark. Implement DataOps and CI/CD for data workflows, monitor and troubleshoot pipelines, validate data quality, collaborate with cross-functional teams, and document data processes and governance.
Data Engineer :
Job Description :
- Develop and maintain data pipelines, ELT processes, and workflow orchestration using Apache Airflow, Python and PySpark to ensure the efficient and reliable delivery of data.
- Design and implement custom connectors to facilitate the ingestion of diverse data sources into our platform, including structured and unstructured data from various document formats .
- Collaborate closely with cross-functional teams to gather requirements, understand data needs, and translate them into technical solutions.
- Implement DataOps principles and best practices to ensure robust data operations and efficient data delivery.
- Design and implement data CI/CD pipelines to enable automated and efficient data integration, transformation, and deployment processes.
- Monitor and troubleshoot data pipelines, proactively identifying and resolving issues related to data ingestion, transformation, and loading.
- Conduct data validation and testing to ensure the accuracy, consistency, and compliance of data.
- Stay up-to-date with emerging technologies and best practices in data engineering.
- Document data workflows, processes, and technical specifications to facilitate knowledge sharing and ensure data governance.
Responsibilities:
- Bachelor's degree in Computer Science, Engineering, or a related field
- 8 - 10 years experience in data engineering, ELT development, and data modeling.
- Proficiency in using Apache Airflow and Spark for data transformation, data integration, and data management.
- Experience implementing workflow orchestration using tools like Apache Airflow, SSIS or similar platforms.
- Demonstrated experience in developing custom connectors for data ingestion from various sources.
- Strong understanding of SQL and database concepts, with the ability to write efficient queries and optimize performance.
- Experience implementing DataOps principles and practices, including data CI/CD pipelines.
- Excellent problem-solving and troubleshooting skills, with a strong attention to detail.
- Effective communication and collaboration abilities, with a proven track record of working in cross-functional teams.
- Familiarity with data visualization tools Apache SuperSet and dashboard development.
- Understanding of distributed systems and working with large-scale datasets.
- Familiarity with data governance frameworks and practices.
- Knowledge of data streaming and real-time data processing technologies (e.g., Apache Kafka).
- Strong understanding of software development principles and practices, including version control (e.g., Git) and code review processes.
- Experience with Agile development methodologies and working in cross-functional Agile teams.
- Ability to adapt quickly to changing priorities and work effectively in a fast-paced environment.
- Excellent analytical and problem-solving skills, with a keen attention to detail.
- Strong written and verbal communication skills, with the ability to effectively communicate complex technical concepts to both technical and non-technical stakeholders.
Required Skills –
DevOps (Heavy), Python, Pyspark, Sql,Airflow, Trino, Hive, Snowflake, Agile Scrum
Good to have–
Linux,Openshift, Kubernentes, Superset
Similar Jobs
Agency • Information Technology
Develop and maintain reliable data pipelines, ELT processes, workflow orchestration, custom data connectors, and CI/CD pipelines using Airflow, Python, PySpark, Hive/Trino, SQL, and Snowflake. Collaborate with cross-functional teams, monitor and troubleshoot data workflows, validate and test data, document technical processes, and support data governance.
Top Skills:
Apache AirflowCi/CdHivePysparkPythonSnowflakeSQLTrino
Agency • Information Technology
Design, build, and maintain ELT/data pipelines and workflow orchestration using Airflow, Python, and PySpark. Develop custom connectors to ingest diverse data, implement DataOps and data CI/CD pipelines, monitor and troubleshoot pipelines, validate data quality, collaborate with cross-functional teams, document workflows, and work with large-scale distributed datasets and visualization tools.
Top Skills:
Apache AirflowApache KafkaSparkApache SupersetGitHiveKubernetesLinuxOpenshiftPysparkPythonSnowflakeSQLSsisTrino
Agency • Information Technology
Design, build, and maintain ELT/data pipelines and workflow orchestration using Airflow, Python, and PySpark. Develop custom connectors, implement DataOps and CI/CD for data, monitor and validate data flows, collaborate with cross-functional teams, and document data workflows and governance.
Top Skills:
Apache AirflowApache KafkaSparkApache SupersetGitHiveKubernetesLinuxOpenshiftPysparkPythonSnowflakeSQLSsisTrino
What you need to know about the Delhi Tech Scene
Delhi, India's capital city, is a place where tradition and progress co-exist. While Old Delhi is known for its rich history and bustling markets, New Delhi is defined by its modern architecture. It's clear the region places a strong emphasis on preserving its cultural heritage while embracing technological advancements, particularly in artificial intelligence, which plays a central role in shaping the city's tech landscape, fueled by investments in research and development.
