Data Platform Engineer

Job summary

The Data Platform Engineer operates the infrastructure that moves production data into an analytics warehouse, ensuring ingestion pipelines are reliable, timely, observable, and cost-effective. The current platform replicates Aurora MySQL into Snowflake through AWS ECS and SQS; the role tunes pipeline capacity and throughput, defines freshness and lag SLOs, maintains infrastructure as code and CI/CD, and applies security controls for payment-handling environments.

Qualifications

• At least 5 years in platform, infrastructure, or DevOps roles supporting production data-intensive systems.
• Hands-on AWS experience with ECS/Fargate, SQS, S3, IAM, CloudWatch, and Aurora/RDS.
• Experience tuning message-driven workloads, including concurrency, backpressure, retries, idempotency, and throughput-to-cost tradeoffs.
• Experience operating ingestion pipelines into Snowflake using Snowpipe/COPY, stages, and load monitoring.
• Experience with infrastructure as code, including Terraform or Ansible, Docker, and CI/CD pipelines.
• Experience with pipeline health and lag observability using Prometheus/Grafana, CloudWatch, Datadog, or Elastic Stack.
• Ability to read, debug, and instrument Python worker code.
• Experience with CDC tooling, MySQL Binlog replication, or message brokers such as Kafka, Confluent, or RabbitMQ is preferred.
• Experience in a PCI-compliant or other regulated environment, or payments and fintech domain knowledge, is preferred.

Responsibilities

• Operate, tune, and scale ECS/SQS ingestion pipelines from Aurora MySQL into Snowflake.
• Define freshness and lag SLOs; build dashboards and alerts; lead incident response and root-cause analysis.
• Maintain infrastructure as code, deployment automation, and CI/CD for pipeline services.
• Benchmark and optimize ingestion throughput and cost across ECS, SQS, and Snowflake Snowpipe/COPY.
• Onboard data sources and consumers in collaboration with application, data, and analytics teams.
• Apply data-quality and security controls for PCI-compliant or payment-handling environments.
• Use AI tools to improve pipeline development, testing, runbooks, and documentation.

Skills

Languages

Industry

Relocation