Design, build, and support production-grade AWS data pipelines that transform operational data into secure, high-quality, AI-ready datasets. Responsibilities include distributed data processing, Parquet curation, privacy-preserving transformations, orchestration, data quality monitoring, schema management, CI/CD, infrastructure as code, metadata and lineage management, troubleshooting, and reliable backfills.
This is a remote position.
Role OverviewWe are looking for Data Engineers at Senior and Mid-Level to join our team in building a privacy-preserving data platform where data engineering meets production-grade software engineering.
You will work on designing, developing, and maintaining reliable data pipelines that transform operational data into high-quality, secure, and AI-ready datasets.
- Build and maintain production-grade data pipelines on AWS.
- Extract, transform, validate, and curate large-scale Parquet datasets.
- Implement data de-identification, masking, and privacy-preserving transformations.
- Design and maintain data pipeline orchestration, scheduling, retries, and backfill mechanisms.
- Implement comprehensive data quality checks, monitoring, and alerting.
- Work with workflow orchestration tools such as Airflow, Dagster, or AWS Step Functions.
- Contribute to CI/CD pipelines and Infrastructure as Code (IaC) practices.
- Manage schema evolution and schema drift across data sources and pipelines.
- Provide production support, troubleshooting, and root cause analysis for data pipeline issues.
- Maintain data catalogs, metadata, and data lineage.
- Follow software engineering best practices including Git, code reviews, automated testing, and maintainable code.
- Build reliable and idempotent data pipelines capable of handling retries and large-scale backfills.
- Strong proficiency in Python and SQL.
- Hands-on experience with AWS data services and production data pipelines.
- Experience with Apache Spark or equivalent distributed data processing technologies.
- Practical experience with Airflow, Dagster, AWS Step Functions, or similar orchestration tools.
- Strong understanding of data pipeline architecture, ETL/ELT, and data transformation.
- Experience working with Parquet and large-scale datasets.
- Understanding of data quality, schema management, monitoring, and alerting.
- Strong software engineering practices including:
- Git and version control
- Code reviews
- Automated testing
- Idempotency
- Error handling
- Retries and backfills
- Git and version control
- Experience supporting and troubleshooting production data pipelines.
- Ability to work effectively with cross-functional engineering and data teams.
Requirements Required Skills
Python | SQL | AWS | Data Engineering | Data Pipelines | ETL/ELT | Apache Spark | Parquet | Airflow | Dagster | AWS Step Functions | Data Orchestration | Data Quality | Schema Management | Data Transformation | Production Support | Git | CI/CD | Automated Testing | Data De-identification | Data Lineage | Data Catalog
Debezium | AWS DMS | Apache Iceberg | Delta Lake | Apache Hudi | Data Masking | Data Tokenization | Terraform | CloudFormation | ML/AI Training Data | Privacy-Preserving Data
Similar Jobs
AdTech
As a Data Engineer, you will design, develop, and maintain data integration solutions, optimize ETL/ELT pipelines, and support data workflows for analytics and reporting.
Top Skills:
Azure Data FactoryAzure DatabricksMicrosoft FabricPysparkPythonSpark SqlSQLT-Sql
Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
Leads complex, cross-functional scientific learning projects for Medical Affairs. Develops curricula, e-learning, and blended learning programs; partners with medical, scientific, agency, and learning systems teams; manages multiple projects, timelines, budgets, and priorities; evaluates emerging e-learning technologies; and updates existing learning resources across therapeutic areas.
Top Skills:
Articulate StorylineDigital Learning TechnologyE-Learning Authoring ToolsLearning Management Systems
.jpeg)
Artificial Intelligence • Hardware • Information Technology • Machine Learning
Lead technicians to install, maintain, calibrate, and troubleshoot semiconductor lab and manufacturing equipment. Ensure ISO/IEC 17025 calibration compliance, manage equipment tracking, drive RCA/RCCA and preventive actions, coordinate vendors and cross-functional teams, support audits, and provide training while maintaining safety, ESD, and cleanroom standards.
Top Skills:
3D X-RayArtificial IntelligenceBend TesterCleanroomCsamEquipment Tracking SystemEsdFibFtirHast ChamberIso/Iec 17025KaizenLean Six SigmaLinuxReflow OvenSemShock TesterSoak ChamberTemp CycleTemperature ChamberTesterThbWindowsX-Section
What you need to know about the Delhi Tech Scene
Delhi, India's capital city, is a place where tradition and progress co-exist. While Old Delhi is known for its rich history and bustling markets, New Delhi is defined by its modern architecture. It's clear the region places a strong emphasis on preserving its cultural heritage while embracing technological advancements, particularly in artificial intelligence, which plays a central role in shaping the city's tech landscape, fueled by investments in research and development.
