Jobs / Permutable
Senior Data Platform Engineer - Python & AWS
Permutable · London, ENG, United Kingdom
London, ENG, United KingdomRemote
Remuneration
Equity options
Location
London, ENG, United Kingdom
Visa sponsorship
Not specified
Job summary
Build and scale production systems that ingest, validate, process, store, and deliver large volumes of news, market, and proprietary data for real-time market intelligence and systematic signals. Own Python data pipelines and AWS infrastructure, develop event-driven workflows, and support AI and quantitative workloads using services such as S3, PostgreSQL, Lambda, ECS, and EKS.
Benefits
Equity optionsCompany socials and offsitesSports eventsOffice treats
Qualifications
- Strong production experience building and maintaining Python services and data pipelines.
- Hands-on experience building and operating production systems on AWS.
- Substantial experience in data platform, data, or backend engineering with ownership of production data systems.
- Strong SQL and PostgreSQL skills, including data modelling, query performance, and large-dataset processing.
- Understanding of messaging, queues, asynchronous processing, reliable workflows, retries, and failure recovery.
- Experience with Pulumi, Terraform, or equivalent infrastructure-as-code tools and automated testing and deployment pipelines.
- Experience with Linux, containers, monitoring, logging, and diagnosing issues across applications, pipelines, and infrastructure.
- Ability to balance performance, reliability, cost, and delivery speed in architecture decisions.
- Proactive and resourceful, with comfort taking responsibility in a small, fast-moving team.
- Degree in computer science, software engineering, or a related discipline, or equivalent practical experience.
Responsibilities
- Design and develop Python pipelines to ingest, validate, transform, and deliver news, market, and proprietary data.
- Build and scale AWS infrastructure using S3, RDS/PostgreSQL, Lambda, ECS, and EKS.
- Develop event-driven services using messaging, queues, and asynchronous processing.
- Improve pipeline data quality and reliability with validation, deduplication, schema handling, retries, and recovery.
- Improve data storage, querying, and delivery for research, production models, and client-facing applications.
- Support AI and quantitative workloads, including integration of NLP, LLM, and quantitative model outputs.
- Manage infrastructure with Pulumi and build CI/CD workflows with GitHub Actions.
- Monitor pipeline health, data freshness, and service performance; diagnose production issues and improve speed, resilience, and AWS costs.
Skills
AirflowAWSECSEKSGitHubGitHub ActionsKubernetesAWS LambdaLinuxMakePostgreSQLPulumiPythonS3Terraform
Degrees
Computer science, software engineering, or a related discipline; equivalent prac
Work schedule
At least three days per week in the Vauxhall hub; remote work for the remaining
Industry
Financial-market dataFinancial technology
Relocation
No