Lead Data Engineer - Energy Trading - (Hybrid) London, UK

Aubay is seeking a highly motivated Lead Data Engineer to join our dynamic team and contribute to the design, development, and optimisation of our client's Data Lakehouse platform. As a Lead Data Engineer, you will provide technical leadership and work closely with cross-functional teams to translate business requirements into robust technical solutions. The ideal candidate will have extensive experience in data engineering, a strong understanding of data modelling and big data technologies, and the ability to lead and mentor a team of data engineers.
Required Skills and Experience: Extensive experience with Databricks, Apache Spark, and data lake architectures. Proficiency in Python and SQL for data manipulation and analysis. Experience with CI/CD processes using Azure DevOps or Git for streamlined development workflows. Strong knowledge of streaming data technologies such as Kafka and KSQL DB. Familiarity with reporting tools like Power BI and Qlik for data visualisation and analytics. Software engineering experience with .NET, particularly for developing event-driven pipelines and automation testing. Experience with Databricks Unity Catalog and understanding of data governance tools. Strong skills in data modelling, including Snowflake modelling, star schema modelling, and other advanced techniques. Ability to optimise big data workloads in Spark to ensure efficient data processing and analytics. Self-driven and motivated with a strong sense of ownership and accountability. Proven experience in managing and inspiring technical teams to achieve project goals. Excellent communication and collaboration skills, with the ability to effectively interact with both business and IT teams. Strong problem-solving and analytical skills, with a keen ability to provide constructive feedback. Ability to foster a positive team environment and encourage continuous improvement.
Desired Skills and Experience: Experience in delivering projects in the streaming data world, particularly with real-time data processing. Understanding of data governance and data quality best practices, ensuring data integrity and compliance. Familiarity with automation testing frameworks to enhance testing efficiency. Experience working on projects related to energy trading, providing insights into industry-specific challenges and solutions. Role Responsibilities:
Lead the design, development, and implementation of data engineering solutions for the Data Lakehouse platform. Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions. Provide technical leadership, mentoring, and coaching to the data engineering team to ensure high-quality deliverables. Ensure best practices in data engineering, including data modelling, ETL processes, and data quality. Optimise big data workloads using Apache Spark and other big data technologies to enhance performance and efficiency. Manage CI/CD pipelines using Azure DevOps or Git, ensuring seamless integration and delivery processes. Develop and maintain event-driven pipelines using .NET and other relevant technologies to support real-time data processing. Implement and manage Databricks Unity Catalog and other data governance tools to ensure data security and compliance. Deliver projects in the streaming data environment using Kafka, KSQL DB, and similar technologies to handle high-volume data streams. Utilise reporting tools such as Power BI and Qlik for data visualisation and reporting to provide actionable insights to stakeholders.
About Aubay UK – Ahead of Innovation! Aubay UK is a recognised InSourcing Partner for client-side deployment delivered across London. Our team, based in Canary Wharf, specialises in hiring IT professionals within London’s Energy and FinTech sectors, helping our clients to expand their operations with top-tier talent who are experts in their fields. We work exclusively with clients who are globally recognized as Energy Super Majors/Financial Services and innovative FinTech players.
Aubay UK is the most recently started branch of Aubay Group www.aubay.com. Aubay Group is an international Digital Services Company, listed on a Euronext Stock Exchange, who have been operating for 25 years in the European market and working alongside some of the biggest names in the Banking, Finance, Insurance, Energy, IT/Digital, Manufacturing, Transport and Telecoms sectors. With over 7,800 employees across 7 countries and 16 offices in England, France, Belgium, Luxembourg, Italy, Spain, and Portugal, Aubay Group generated revenues of €534 million in 2023.
What Aubay UK Offers At Aubay UK, people are at the heart of our business. We offer a competitive remuneration package which includes a range of benefits. You will receive continuous support from our dedicated Talent Acquisition Specialists, who will support your career development and success during your assignment with our client. Other benefits include: 25 Days Annual Leave + Bank Holidays Pension scheme Work-from-home opportunities Access to Udemy training platform and conferences Discount on winter sports holidays at www.aero1607.com Opportunity to work directly for the client at the end of the assignment
Our Client Our client is one of the Super Major global energy companies with around 84,000 employees across 70+ countries who are working to power progress through cleaner energy solutions. Specialities: Upstream/Downstream, Biofuels, Integrated Gas, New Energies, Chemicals, Energy and Trading.
You will have the opportunity to work in a challenging but rewarding environment that is fast-paced and changing fundamentally and work towards transforming the business of a Super Major energy company to meet the ambition to be a net-zero emissions energy business by 2050 whilst delivering a world-class business case that has a robust societal license to operate. In your role, you will be expected to enact change and deliver value globally across business lines and geographies.
Apply via our website - www.aubay.co.uk This role requires you to be based in the UK and have the right to work in the UK.
Aubay UK is proud to be an equal-opportunity employer. Employment decisions will be based on merit, competence, performance, and business needs.