Leads the design, development, and maintenance of scalable ETL/ELT pipelines and data products across distributed, multi-cloud environments. Defines data engineering architectures, resolves complex production issues, and designs efficient data solutions. Provides technical leadership and mentorship while collaborating with architects, engineers, and business stakeholders. Requires expertise in big data, AWS services, data warehousing, databases, data modeling, orchestration, streaming, security, observability, version control, containers, and modern development practices.
Job Title Lead Data Engineer Location (s) Cochin Years of Experience 5-10yrs Job Description
Role & Responsibilities Technical Qualifications Python + AWS EMR + AWS Glue + Snowflake + ETL/ELT + Data Engineering + Data Architecture + Big Data + SQL
Lead Data Engineer
What You’ll Do
- Lead the design, development, and maintenance of scalable ETL/ELT pipelines and data products in multi-cloud, multi-region, distributed environments.
- Define and apply appropriate data engineering patterns and architectures based on business and technical requirements.
- Drive technical investigations and resolve complex data, operational, and production issues.
- Design scalable, flexible, efficient, and supportable solutions using appropriate technologies and disciplined development practices.
- Provide technical leadership, guidance, and mentorship to data engineering teams.
- Collaborate with architects, engineering teams, and business stakeholders to deliver robust data solutions.
Key Skills
- Data Engineering & Architecture: Strong understanding of data engineering patterns and modern architectures such as Data Mesh, Data Fabric, Data Lake, and Data Warehouse.
- Big Data & Cloud: Proficiency in Amazon EMR, AWS Glue, Data Lake, and technologies for processing large datasets.
- Programming: Strong proficiency in Python, Java, or Scala, with solid OOP expertise.
- Data Warehousing: Strong knowledge of data warehousing solutions, preferably Snowflake.
- ETL/ELT & Orchestration: Hands-on experience with ETL/ELT pipelines and data orchestration tools.
- Databases: Strong expertise in relational and NoSQL databases.
- Data Modelling: Experience designing efficient OLAP and OLTP data models.
- Streaming: Knowledge of streaming data technologies and architectures.
- Version Control: Experience with Git and modern development practices.
- Containers & Orchestration: Understanding of Docker and Kubernetes.
- Data Security: Knowledge of encryption, access control, data privacy, and compliance.
- Monitoring & Logging: Experience implementing monitoring, logging, alerting, and observability for data pipelines.
Role & Responsibilities
- Lead Data Engineer
- What You’ll Do
- Lead the design, development, and maintenance of scalable ETL/ELT pipelines and data products in multi-cloud, multi-region, distributed environments.
- Define and apply appropriate data engineering patterns and architectures based on business and technical requirements.
- Drive technical investigations and resolve complex data, operational, and production issues.
- Design scalable, flexible, efficient, and supportable solutions using appropriate technologies and disciplined development practices.
- Provide technical leadership, guidance, and mentorship to data engineering teams.
- Collaborate with architects, engineering teams, and business stakeholders to deliver robust data solutions.
- Key Skills
- Data Engineering & Architecture: Strong understanding of data engineering patterns and modern architectures such as Data Mesh, Data Fabric, Data Lake, and Data Warehouse.
- Big Data & Cloud: Proficiency in Amazon EMR, AWS Glue, Data Lake, and technologies for processing large datasets.
- Programming: Strong proficiency in Python, Java, or Scala, with solid OOP expertise.
- Data Warehousing: Strong knowledge of data warehousing solutions, preferably Snowflake.
- ETL/ELT & Orchestration: Hands-on experience with ETL/ELT pipelines and data orchestration tools.
- Databases: Strong expertise in relational and NoSQL databases.
- Data Modelling: Experience designing efficient OLAP and OLTP data models.
- Streaming: Knowledge of streaming data technologies and architectures.
- Version Control: Experience with Git and modern development practices.
- Containers & Orchestration: Understanding of Docker and Kubernetes.
- Data Security: Knowledge of encryption, access control, data privacy, and compliance.
- Monitoring & Logging: Experience implementing monitoring, logging, alerting, and observability for data pipelines.
Similar Jobs
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Design and build data services and pipelines supporting machine learning products. Develop automation tools for model deployment, maintain production data systems, improve infrastructure, participate in code reviews, and collaborate across engineering, data science, and product teams. The role requires expertise in distributed systems, large-scale data processing, CI/CD, container orchestration, and AI-enabled workflow improvements.
Top Skills:
AirflowAWSAws BatchCi/CdDockerEmrGlueGoKafkaKubernetesKv StoresLinuxPythonRelational DatabasesSagemakerSparkSpinnaker
Information Technology • Database • Consulting
Design and implement data pipelines, optimize processing and storage, ensure data accuracy and performance, provide technical support, and collaborate with data scientists, analysts, and stakeholders to resolve data issues and meet data requirements.
Top Skills:
Data PipelinesData Processing WorkflowsData Storage Systems
Information Technology • Consulting
Designs and maintains scalable data pipelines and ETL/ELT workflows using Python, SQL, and Azure Databricks. Manages large structured and semi-structured datasets, data models, monitoring, quality controls, troubleshooting, and production support. Integrates LLM and Generative AI capabilities, including APIs, embeddings, prompt engineering, and RAG workflows, into data solutions. Collaborates with engineering, analytics, and business teams while following security, CI/CD, documentation, and maintainability best practices.
Top Skills:
Azure Data FactoryAzure Data Lake Storage (Adls)Azure DatabricksAzure DevopsAzure Openai ServiceAzure SqlAzure Synapse AnalyticsCi/CdEmbeddingsEtl/EltGenerative AiGitLarge Language Models (Llms)Llm ApisPrompt EngineeringPythonRest ApisRetrieval-Augmented Generation (Rag)SQLVector DatabasesVector Search
What you need to know about the Delhi Tech Scene
Delhi, India's capital city, is a place where tradition and progress co-exist. While Old Delhi is known for its rich history and bustling markets, New Delhi is defined by its modern architecture. It's clear the region places a strong emphasis on preserving its cultural heritage while embracing technological advancements, particularly in artificial intelligence, which plays a central role in shaping the city's tech landscape, fueled by investments in research and development.
