Design, build, and support production-grade AWS data pipelines that transform operational data into secure, high-quality, AI-ready datasets. Responsibilities include distributed data processing, Parquet curation, privacy-preserving transformations, orchestration, data quality monitoring, schema management, CI/CD, infrastructure as code, metadata and lineage management, troubleshooting, and reliable backfills.
This is a remote position.
Role OverviewWe are looking for Data Engineers at Senior and Mid-Level to join our team in building a privacy-preserving data platform where data engineering meets production-grade software engineering.
You will work on designing, developing, and maintaining reliable data pipelines that transform operational data into high-quality, secure, and AI-ready datasets.
- Build and maintain production-grade data pipelines on AWS.
- Extract, transform, validate, and curate large-scale Parquet datasets.
- Implement data de-identification, masking, and privacy-preserving transformations.
- Design and maintain data pipeline orchestration, scheduling, retries, and backfill mechanisms.
- Implement comprehensive data quality checks, monitoring, and alerting.
- Work with workflow orchestration tools such as Airflow, Dagster, or AWS Step Functions.
- Contribute to CI/CD pipelines and Infrastructure as Code (IaC) practices.
- Manage schema evolution and schema drift across data sources and pipelines.
- Provide production support, troubleshooting, and root cause analysis for data pipeline issues.
- Maintain data catalogs, metadata, and data lineage.
- Follow software engineering best practices including Git, code reviews, automated testing, and maintainable code.
- Build reliable and idempotent data pipelines capable of handling retries and large-scale backfills.
- Strong proficiency in Python and SQL.
- Hands-on experience with AWS data services and production data pipelines.
- Experience with Apache Spark or equivalent distributed data processing technologies.
- Practical experience with Airflow, Dagster, AWS Step Functions, or similar orchestration tools.
- Strong understanding of data pipeline architecture, ETL/ELT, and data transformation.
- Experience working with Parquet and large-scale datasets.
- Understanding of data quality, schema management, monitoring, and alerting.
- Strong software engineering practices including:
- Git and version control
- Code reviews
- Automated testing
- Idempotency
- Error handling
- Retries and backfills
- Git and version control
- Experience supporting and troubleshooting production data pipelines.
- Ability to work effectively with cross-functional engineering and data teams.
Requirements Required Skills
Python | SQL | AWS | Data Engineering | Data Pipelines | ETL/ELT | Apache Spark | Parquet | Airflow | Dagster | AWS Step Functions | Data Orchestration | Data Quality | Schema Management | Data Transformation | Production Support | Git | CI/CD | Automated Testing | Data De-identification | Data Lineage | Data Catalog
Debezium | AWS DMS | Apache Iceberg | Delta Lake | Apache Hudi | Data Masking | Data Tokenization | Terraform | CloudFormation | ML/AI Training Data | Privacy-Preserving Data
Similar Jobs
Artificial Intelligence • Fintech • Information Technology • Logistics • Payments • Business Intelligence • Generative AI
Designs and implements multi-cloud security controls, policy-as-code guardrails, IAM and network protections, container and Kubernetes security, vulnerability remediation, and incident response tooling. Partners with engineering, IT, compliance, and risk teams on audits and control validation. The role also mentors security engineers, supports root-cause analysis, improves response runbooks, and participates in on-call rotations.
Top Skills:
AWSAzureBashCi/CdContainer SecurityEdrGCPIamInfrastructure As CodeJavaScriptKubernetesNaclsOrganizations/ScpsPythonVpcVulnerability Scanning
Artificial Intelligence • Fintech • Information Technology • Logistics • Payments • Business Intelligence • Generative AI
Supports cloud, application, infrastructure, container, and Kubernetes security engineering. Contributes to multi-cloud controls, IAM policies, policy-as-code guardrails, vulnerability triage, compliance evidence gathering, security automation, and secure AI system monitoring. The role also involves scripting, documentation, root-cause analysis, and collaboration with Engineering, IT, Compliance, and senior security engineers.
Top Skills:
AWSAzureBashCi/CdContainer SecurityFedrampGCPIamInfrastructure As CodeIso 27001JavaScriptKubernetesNetwork AclsOrganizations/ScpsPci-DssPolicy As CodePythonSecurity GroupsSoc 2VpcVulnerability Scanning
Information Technology
Design end-to-end digital product experiences through user research, interaction design, wireframing, prototyping, visual design, and design systems. Collaborate with product, engineering, and business stakeholders to create accessible, responsive web and mobile interfaces. Conduct usability testing, incorporate feedback and analytics, present design concepts, and support accurate implementation across multiple initiatives.
Top Skills:
A/B TestingAdobe Creative SuiteComponent LibrariesCSSDesign SystemsFigjamFigmaHTMLProduct AnalyticsWcag
What you need to know about the Delhi Tech Scene
Delhi, India's capital city, is a place where tradition and progress co-exist. While Old Delhi is known for its rich history and bustling markets, New Delhi is defined by its modern architecture. It's clear the region places a strong emphasis on preserving its cultural heritage while embracing technological advancements, particularly in artificial intelligence, which plays a central role in shaping the city's tech landscape, fueled by investments in research and development.
