Data Platform Engineer

Job summary

Build and operate infrastructure that delivers production data to an analytics warehouse, focusing on pipeline reliability, performance, scalability, observability, and cost efficiency. The platform currently uses Amazon ECS and SQS to replicate Aurora MySQL data into Snowflake; the role includes tuning ingestion, defining freshness and lag SLOs, automating deployments, onboarding data sources, and applying security controls for payment-handling environments.

Qualifications

• At least 5 years of platform, infrastructure, or DevOps experience supporting production data-intensive systems.
• Hands-on AWS experience with ECS/Fargate, SQS, S3, IAM, CloudWatch, and Aurora/RDS.
• Experience tuning message-driven workloads, including concurrency, backpressure, retries, idempotency, and throughput versus cost.
• Experience operating data ingestion pipelines into Snowflake using Snowpipe or COPY, stages, and load monitoring.
• Experience with infrastructure as code, Docker, and CI/CD pipelines.
• Experience with pipeline observability and alerting using Prometheus/Grafana, CloudWatch, Datadog, or Elastic Stack.
• Ability to read, debug, and instrument Python worker code.
• Additional experience with CDC tools, MySQL Binlog replication, Kafka or RabbitMQ, or time-series database administration is advantageous.
• Experience in PCI-compliant or other regulated environments, or payments and fintech domain knowledge, is advantageous.

Responsibilities

• Operate, tune, and scale ECS/SQS ingestion pipelines from Aurora MySQL into Snowflake.
• Define freshness and lag SLOs, build alerts and dashboards, and lead incident response and root-cause analysis.
• Build and maintain infrastructure as code, deployment automation, and CI/CD for pipeline services.
• Benchmark and optimize ingestion throughput and cost across ECS, SQS, and Snowflake.
• Onboard data sources and consumers in partnership with application, data, and analytics teams.
• Apply data-quality and security controls, including encryption, secrets handling, least-privilege access, and audit trails.
• Use AI tools to improve pipeline development, testing, runbooks, and documentation.

Skills

Languages

Industry

Relocation