The Senior Lead Data Engineer will shape data platform architecture, ensuring alignment with business needs and best practices while providing technical leadership and driving continuous improvement in data solutions.
We are seeking a highly skilled Senior Lead Data Engineer to join our R&D Data Engineering Team. In this role, you will be a key player in shaping the architecture and technical direction of our data platform, ensuring that it meets the evolving needs of the business while adhering to best practices and industry standards. If you're looking for an opportunity to combine your technical skills with strategic thinking and make a real difference, we want to hear from you.
About You – experience, education, skills, and accomplishments:
- Bachelor’s Degree or equivalent
- At least 7 years of Relevant Experience
- At least 5+ Years in Software Development: Demonstrated experience in software development, with a focus on Big Data technologies.
- At least 3+ Years in Distributed Data Processing: Proven experience in building scalable distributed data processing solutions.
- At least 3+ Years in Database Design: Expertise in database design and development, with a strong focus on data model design.
- Strong Proficiency with Apache Spark and Airflow: Extensive hands-on experience with these technologies, leveraging them for data processing and orchestration.
- Python Proficiency: Advanced proficiency in Python for data processing and building services.
- Experience with Databricks and Snowflake: Practical experience with these platforms, including their use in cloud-based data pipelines.
- Familiarity with Delta Lake or Apache Iceberg: Experience working with these data storage to decouple storage from processing engines.
- Cloud-Based Solutions Expertise: Proven experience in designing and implementing cloud-based data pipelines, with specific expertise in AWS services such as S3, RDS, EMR, and AWS Glue.
- CI/CD Best Practices: Strong understanding and application of CI/CD principles
It would be great if you also had:
- Knowledge of Additional Technologies: Familiarity with Cassandra, Hadoop, Apache Hive, Jupyter notebooks, BI tools: Tableau and Power BI.
- Experience with PL/SQL and Oracle GoldenGate: Additional experience in these areas is advantageous
- Knowledge on any of these technologies/tools: Cassandra, Hadoop, Apache Hive, Snowflake, Jupiter notebook, Databricks stack,
- AWS services, EC2, ECS, RDS, EMR, S3, AWS Glue, Airflow
What will you be doing in this role?
- Provide Technical Leadership: Offer strategic guidance on technology choices, comparing different solutions to meet business requirements while considering cost control and performance optimization.
- Communicate Effectively: Exhibit excellent communication skills, with the ability to clearly articulate complex technical concepts to both technical and non-technical stakeholders.
- Design and Maintain Data Solutions: Develop and maintain the overall solution architecture for the Data Platform, demonstrating deep expertise in integration architecture and design across multiple platforms at an enterprise scale.
- Enforce Best Practices: Implement and enforce best practices in Big Data management, from software selection to architecture design and implementation processes.
- Drive Continuous Improvement: Contribute to the continuous enhancement of support and delivery functions by staying informed about technology trends and making recommendations for improving application services.
- Lead Technical Investigations: Conduct technical investigations and proofs of concept, both individually and as part of a team, including hands-on coding to provide technical recommendations.
- Knowledge Sharing: Actively spread knowledge and best practices within the team, fostering a culture of continuous learning and improvement
About the Team
We are team located in India, US, and Europe.
Hours of Work
Regular working timing in India.
At Clarivate, we are committed to providing equal employment opportunities for all qualified persons with respect to hiring, compensation, promotion, training, and other terms, conditions, and privileges of employment. We comply with applicable laws and regulations governing non-discrimination in all locations.
Top Skills
Airflow
Apache Iceberg
Spark
Aws Emr
Aws Glue
Aws Rds
Aws S3
Databricks
Delta Lake
Python
Snowflake
Clarivate Analytics Noida, Uttar Pradesh, IND Office
Noida, India
Similar Jobs
Aerospace • Information Technology • Cybersecurity • Defense • Manufacturing
Develop, maintain, and integrate software components for Flight Management Systems. Lead software projects, ensure compliance with standards, and mentor team members.
Top Skills:
Atlassian ToolsC/C++CodeEclipseGitLdraLinuxModel-Based DevelopmentPolyspacePythonReal-Time Operating Systems (Rtos)UnixVisual Studio
Aerospace • Information Technology • Cybersecurity • Defense • Manufacturing
Manage a team of software engineers focused on avionics products, develop project plans, provide mentorship, and ensure successful software deliveries.
Top Skills:
AvionicsBattery Management SystemDo178Functional IntegrationModel Based DevelopmentSoftware EngineeringValidation/Verification
Aerospace • Information Technology • Cybersecurity • Defense • Manufacturing
The Lead Software Engineer will develop, lead, and test Java Full Stack applications, ensuring alignment with system requirements while coordinating project management efforts and fostering collaboration in an Agile team environment.
Top Skills:
AngularAWSAzureCSSDockerHibernateHTMLJavaJavaScriptKubernetesNoSQLPivotal Cloud FoundrySpringSQL
What you need to know about the Delhi Tech Scene
Delhi, India's capital city, is a place where tradition and progress co-exist. While Old Delhi is known for its rich history and bustling markets, New Delhi is defined by its modern architecture. It's clear the region places a strong emphasis on preserving its cultural heritage while embracing technological advancements, particularly in artificial intelligence, which plays a central role in shaping the city's tech landscape, fueled by investments in research and development.
