Data Platform Engineer

Job summary

This hands-on data platform infrastructure role builds, maintains, and optimizes pipelines that replicate production data into an analytics warehouse, with a focus on reliability, scalability, performance, cost efficiency, and observability. The platform currently uses Amazon ECS services and SQS queues to move data from Aurora MySQL into Snowflake; responsibilities include tuning ingestion, defining freshness and lag SLOs, automating infrastructure and deployments, and applying security controls for payment-handling environments.

Qualifications

• 5+ years in platform, infrastructure, or DevOps roles supporting data-intensive systems in production
• Deep hands-on AWS experience: ECS/Fargate (task sizing, autoscaling, deployments), SQS (throughput, DLQs, visibility timeouts, batching), S3, IAM, CloudWatch, and Aurora/RDS from the operator side
• Proven track record tuning message-driven workloads: consumer concurrency, backpressure, retries, idempotency, throughput versus cost
• Experience operating pipelines into Snowflake from the ingestion side (Snowpipe/COPY, stages, load monitoring)
• Infrastructure as code with Terraform/Ansible, containerization with Docker, and CI/CD pipelines
• Observability and alerting for pipeline health and lag using Prometheus/Grafana, CloudWatch, Datadog, or the Elastic Stack
• Enough Python to read, debug, and instrument worker code
• Passion for automation, performance, and clean system design

Responsibilities

• Operate, tune, and scale the ECS/SQS-based ingestion pipelines from Aurora MySQL into Snowflake
• Own pipeline reliability: define SLOs for freshness and lag, build alerting and dashboards, lead incident response and root-cause analysis for pipeline failures
• Build and maintain infrastructure as code, deployment automation, and CI/CD for pipeline services
• Continuously benchmark and tune throughput versus cost across ECS, SQS, and Snowflake ingestion (Snowpipe/COPY, stages)
• Onboard new sources and consumers to the platform in partnership with application, data, and analytics teams
• Apply data-quality and security controls appropriate for PCI-compliant or payment-handling environments: encryption, secrets handling, least-privilege access, audit trails
• Proactively explore and apply AI tools to improve efficiency across the pipeline lifecycle (development, testing, runbooks, and documentation)

Skills

Languages

Industry

Relocation