Senior Data Engineer

Job Description

Senior Data Engineer

London (Hybrid)

AI Technology

Salary: £80,000-£110,000

Paradigm Talent is currently working with an AI-driven technology company focused on building next- automation and intelligence systems for complex, high-stakes environments specifically within the HealthTech space.

The team applies advanced machine learning, computer vision, and multimodal AI to solve critical challenges in operational efficiency and decision-making. They work at the cutting edge of deep learning, object recognition, and large-scale AI systems, delivering solutions that drive real-world impact. If you're passionate about research-driven AI innovation and enjoy working on highly technical challenges, this role is for you.

The Role: Data Engineer (Scalable Data & AI Infrastructure)

We’re looking for a Data Engineer with experience in scalable data pipelines, cloud infrastructure, and real-time data processing. You will be responsible for designing, optimising, and maintaining secure, high-performance data architectures that support machine learning, analytics, and automation-driven applications.

This role offers the opportunity to work in a fast-paced, data-rich environment, collaborating closely with ML engineers, software developers, and product teams to ensure data reliability, security, and efficiency at scale.

What You’ll Do

Data Pipeline Development & Optimization

• Design, construct, and maintain large-scale data processing and ETL pipelines for structured and unstructured data.
• Optimize data flow, transformation, and storage, ensuring high efficiency and scalability.
• Develop and maintain data dashboards for real-time insights and analytics.

Cloud & Infrastructure Engineering

• Work with SQL/NoSQL databases and cloud data services (AWS) to manage and process large datasets.
• Optimize data warehousing, modeling, and indexing for performance and scalability.
• Leverage Apache Spark, Airflow, Kafka, or similar technologies to manage and automate workflows.

Data Security & Quality Control

• Ensure data security, compliance, and integrity, implementing best practices for access control and governance.
• Identify and resolve data quality issues proactively, ensuring clean, accurate, and usable data.
• Collaborate with machine learning and application engineering teams to prepare data for AI-driven applications.

Collaboration & Stakeholder Engagement

• Work closely with cross-functional teams, including ML researchers, software engineers, and business analysts, to understand data needs and optimize solutions.
• Support data collection and integration efforts, working with teams across multiple locations to ensure consistency.
• Bring an analytical mindset, ensuring that data-driven insights align with business and technical goals.

Skills & Experience

• 3+ years of experience in data engineering or a related field.
• Strong expertise in ETL development, building and maintaining scalable data pipelines.
• Proficiency in Python for data processing and automation.
• Hands-on experience with SQL/NoSQL databases and cloud data platforms (AWS)
• Understanding of data modelling, data warehousing, and database optimisation.
• Experience with distributed data processing tools (Apache Spark, Airflow, Kafka, or similar).
• Proactive approach to identifying and solving data quality issues.
• Strong project management skills, coordinating with cross-functional teams and data capture staff.