Lead the design and implementation of scalable ETL pipelines using Python and Snowflake, ensuring efficient data processing and architecture for cloud-native solutions.
Embark on an exciting journey into the realm of software development with 3Pillar! We extend an invitation for you to join our team and gear up for a thrilling adventure. At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making.
As a Senior Data Engineer, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries.
If your passion for data engineering solutions that make a real-world impact, consider this your pass to the captivating world of Data Science and Engineering! 🌍🔥
Responsibilities:
- Lead the design and implementation of scalable ETL/data pipelines using Python for data processing.
- Ensure efficient data processing for high-volume clickstream, demographics, and business data.
- Lead the strategic planning, execution, and optimization of large-scale data migration initiatives to Snowflake, ensuring data integrity, security, and minimal business disruption.
- Use Snowflake components/tools - Snowpipe, SnowSQL, SnowPark etc
- Guide the team in adopting best practices for data pipeline development, code quality, and performance optimization.
- Configure, deploy, and maintain AWS infrastructure, primarily AWS EC2, S3, RDS, and EMR, to ensure scalability, availability, and security.
- Support data storage and retrieval workflows using S3 and SQL-based storage solutions.
- Provide architectural guidance for cloud-native data solutions and infrastructure design.
- Oversee legacy framework maintenance, identify improvement areas, and propose comprehensive cloud migration or modernization plans.
- Coordinate infrastructure changes with stakeholders to align with business needs and budget constraints.
- Develop and implement robust monitoring solutions to track system health, performance, and data pipeline accuracy.
- Set up alerts and dashboards for proactive issue detection and collaborate with cross-functional teams to resolve critical issues.
- Lead efforts in incident response, root cause analysis, and post-mortem processes for complex data engineering challenges.
- Document workflows, troubleshooting procedures, and code for system transparency and continuity.
- Provide mentoring and training to team members on best practices and technical skills.
- Foster a culture of continuous learning, knowledge sharing, and technical excellence within the data engineering team.
Qualification
- Experience: 6 years of experience in data engineering or a related technical field, with at least 4 years in a Snowflake projects.
- Cloud Data Warehousing: Proven expertise with Snowflake data warehousing, including schema design, efficient data loading (e.g., Snowpipe, COPY into), performance tuning, and robust access control mechanisms.
- Programming & Scripting: Strong programming skills in Python for automation and data workflows.
- Data Processing & Storage: Expertise in managing SQL databases for data storage and query optimization.
- Monitoring & Alerting Tools: Familiarity with monitoring solutions for real-time tracking and troubleshooting of data pipelines.
- Experience with the AWS Cloud and data engineering services
Top Skills
AWS
Ec2
Emr
Python
Rds
S3
Snowflake
Snowpark
Snowpipe
Snowsql
Similar Jobs
Artificial Intelligence • Edtech • Mobile • Natural Language Processing • Productivity • Software
Lead design initiatives for an AI communications platform, focusing on user experiences, monetization flows, and collaboration with cross-functional teams. Mentor junior designers and advocate for user-driven design decisions.
Top Skills:
Mobile DesignProduct DesignPrototypingUx DesignWeb Design
Financial Services
As a Trading Services Associate II, you will execute trade operations, reconcile accounts, maintain compliance, and contribute to process improvements while collaborating with stakeholders and mentoring junior staff.
Top Skills:
ExcelMicrosoft Office SuiteSQLVBA
Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
Lead the Creative & Graphics Design Team in developing medical communication materials. Oversee project management and drive innovation in design processes. Ensure quality and compliance with standards while fostering a collaborative team environment.
Top Skills:
Adobe Creative SuiteAIAr/VrCorel DrawFigmaIllustrator
What you need to know about the Delhi Tech Scene
Delhi, India's capital city, is a place where tradition and progress co-exist. While Old Delhi is known for its rich history and bustling markets, New Delhi is defined by its modern architecture. It's clear the region places a strong emphasis on preserving its cultural heritage while embracing technological advancements, particularly in artificial intelligence, which plays a central role in shaping the city's tech landscape, fueled by investments in research and development.
