Big Data Lead

Check the JD for details however the primary skills are - Spark expert who have used PDF unstructured data and mapped them into tables, ADB, ADF, strong SQL, Python / Pyspark Job description EYDM is seeking a proactive, dynamic, and adaptable member to join our global, diverse, and inclusive team. Given the continuous growth and evolution of our products, we require candidates who not only embrace change but also possess the experience and capability to hit the ground running. Ideal team members will thrive in ambiguous environments, demonstrating curiosity and proactiveness while effectively contributing from day one. Reporting into the Technical Lead, the new member will play a pivotal role in supporting the Data Ingestion team, Business and Technical Product Owners, to expand and promote our offering throughout EY’s Service Lines, Products, Functions and engagement teams. This role covers a broad area of activity, including technical design and development of near real time data processing, consumer interaction, business documentation and reporting. Your key responsibilities You will be responsible for: Collaboratively identify and ideate opportunities to continuously improve EY.ai Data Marketplace (EYDM) data asset and the services it provides to consumers. Engage in design and development sessions to further EYDM’s data asset creation and ingestion pipelines with a focus on stability, optimisation and traceability. Develop, test and maintain EYDM’s data pipelines & processes. Developing and testing resilient, fault tolerant, modular code, to ensure EYDM’s processes, pipelines and services are robust and highly available. Research, explore and if feasible implement new methods and processes across EYDM to build world class data products and associated services. Actively participate in cross-team collaboration to ensure smooth transition through Ingestion, Consumption and Activation