Hadoop Developer (Spark/Scala)

Choose a partner with intimate knowledge of your industry and first-hand experience of defining its future.Your locationYour locationIndustriesChoose a partner with intimate knowledge of your industry and first-hand experience of defining its future.# Hadoop Developer (Spark/Scala)London About the Job you are considering: This role involves designing, building, and supporting large scale Big Data solutions using Hadoop and Spark technologies. You will primarily develop and debug Spark jobs in Scala, with opportunities to use Java and Python where appropriate. The position focuses on creating scalable data pipelines, optimizing distributed processing, and supporting analytics and data science teams. It also offers exposure to cloud based Big Data platforms and modern data engineering practices in a fast paced, data driven environment. Hybrid working: The places that you work from day to day will vary according to your role, your needs, and those of the business; it will be a blend of Company offices, client sites, and your home; noting that you will be unable to work at home 100% of the time. Your Role: Design and develop Hadoop based applications and scalable data pipelines. Debug, develop, and optimize Spark jobs primarily using Scala. Build, operate, monitor, and troubleshoot Hadoop clusters. Develop robust ETL workflows using Spark, Hive, and Pig. Create and manage data ingestion pipelines using Sqoop, Flume, or Kafka. Optimize MapReduce jobs and efficiently manage HDFS storage. Collaborate with data scientists and analysts to meet data requirements. Ensure data security, compliance, and maintain technical documentation. Your Skills: Strong experience developing and debugging Spark applications in Scala. Hands on Java Big Data development experience. Good knowledge of Python for data processing and scripting (nice to have). Solid understanding of Hadoop ecosystem: HDFS, MapReduce, YARN. Experience with Hive, Pig, and HBase. Knowledge of data ingestion tools such as Sqoop, Flume, and Kafka. Ability to optimize distributed data processing jobs. Exposure to cloud platforms such as AWS, GCP, or Azure is a plus. Declare they have a disability, and Meet the minimum essential criteria for the role. You'd be joining an accredited Great Place to work for Wellbeing in 2024. Employee wellbeing is vitally important to us as an organisation. We see a healthy and happy workforce a critical component for us to achieve our organisational ambitions. To help support wellbeing we have trained 'Mental Health Champions' across each of our business areas, and we have invested in wellbeing apps such as Thrive and Peppy. You will be empowered to explore, innovate, and progress. You will benefit from Capgemini's 'learning for life' mindset, meaning you will have countless training and development opportunities from thinktanks to hackathons, and access to 250,000 courses with numerous external certifications from AWS, Microsoft, Harvard ManageMentor, Cybersecurity qualifications and much more. You will be joining one of the World's Most Ethical Companies(R), as recognised by Ethisphere(R) for 13 consecutive years. We live our values by making ethical business choices every day. Working ethically is at the centre of our culture at Capgemini, meaning you will be helping to create a future we can all be proud of. Why you should consider Capgemini: Growing clients' businesses while building a more sustainable, more inclusive future is a tough ask. When you join Capgemini, you'll join a thriving company and become part of a collective of free-thinkers, entrepreneurs and industry experts. We find new ways technology can help us reimagine what's possible. It's why, together, we seek out opportunities that will transform the world's leading businesses, and it's how you'll gain the experiences and connections you need to shape your future. By learning from each other every day, sharing knowledge, and always pushing yourself to do better, you'll build the skills you want. You'll use your skills to help our clients leverage technology to innovate and grow their business. So, it might not always be easy, but making the world a better place rarely is.