Design, build, and optimize end-to-end data pipelines for large structured and unstructured data. Implement near-real-time ETL, data validation, monitoring, and performance optimization. Collaborate with stakeholders, document designs and workflows, and provide technical guidance to the team.
Location: Pune
Responsibilities include:- Design, implement, and optimize end-to-end data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data.
- Develop data pipelines to extract and transform data in near real-time using cloud-native technologies.
- Implement data validation and quality checks to ensure accuracy and consistency.
- Monitor system performance, troubleshoot issues, and implement optimizations to enhance reliability and efficiency.
- Collaborate with business users, analysts, and other stakeholders to understand data requirements and deliver tailored solutions.
- Document technical designs, workflows, and best practices to facilitate knowledge sharing and maintain system documentation.
- Provide technical guidance and support to team members and stakeholders as needed.
- 8+ years of work experience.
- Proficiency in writing complex SQL queries on MPP systems (Snowflake/Redshift).
- Experience in Databricks and Delta tables.
- Data engineering experience with Spark/Scala/Python.
- Experience in Microsoft Azure stack (Azure Storage Accounts, Data Factory, and Databricks).
- Experience in Azure DevOps and CI/CD pipelines.
- Working knowledge of Python.
- Comfortable participating in 2-week sprint development cycles.
Similar Jobs
Agency • Information Technology
Design, build, optimize, and maintain high-performance Spark-based data pipelines using Scala/Java and Hive on Hadoop/CDP. Own full project lifecycle, enforce coding best practices, troubleshoot Spark/Hive/YARN performance, and collaborate with stakeholders to deliver scalable data solutions.
Top Skills:
SparkCloudera Data Platform (Cdp)HadoopHiveJavaScalaYarn
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Senior Salesforce Engineer responsible for designing and building Salesforce solutions: develop Apex, triggers, Lightning Web Components, Aura, and Visualforce; build and modernize integrations and platform experiences; lead Quote-to-Cash implementations; troubleshoot production (L3) issues; follow CI/CD and deployment best practices; participate in Agile ceremonies; monitor org performance and advocate Salesforce best practices.
Top Skills:
AgentforceAgenticApexAuraCopadoData CloudLightning Web ComponentsLwcQuote-To-CashSalesforceSlackVisualforce
Cloud • Information Technology • Productivity • Software • Automation
Provide technical support to customers via meetings, chat, and email; troubleshoot product, configuration, networking and cloud issues; empathize and analyze reported problems; use technical knowledge to resolve or escalate; collaborate with Product and Engineering; participate in on-call rotations and specified shift hours.
Top Skills:
APIsAWSBoomiGoogle Cloud PlatformGroovyHadoopJavaJavaScriptLinuxAzureNetSuiteOauth 2.0PostmanReactSalesforceWindows
What you need to know about the Delhi Tech Scene
Delhi, India's capital city, is a place where tradition and progress co-exist. While Old Delhi is known for its rich history and bustling markets, New Delhi is defined by its modern architecture. It's clear the region places a strong emphasis on preserving its cultural heritage while embracing technological advancements, particularly in artificial intelligence, which plays a central role in shaping the city's tech landscape, fueled by investments in research and development.
