Senior Data Platform Engineer

Job summary

The Senior Data Platform Engineer role involves designing, developing, and supporting scalable data processing solutions using Python/PySpark, SQL Server, and Microsoft SSIS. Key responsibilities include creating ETL/data-processing pipelines, optimizing SQL queries, and ensuring database performance across large-scale data platforms while collaborating with engineering and business teams.

Qualifications

• 4+ years of relevant experience in data engineering, application development, database development, or related technology roles.
• Strong hands-on experience with Python and PySpark for large-scale data processing and ETL development.
• Strong proficiency in SQL/T-SQL and relational database technologies, preferably Microsoft SQL Server.
• Experience developing and supporting complex SQL queries, stored procedures, database objects, and high-volume data-processing solutions.
• Strong understanding of SQL Server performance tuning, including query execution plans, indexing, statistics, joins, locking/blocking, and query optimization.
• Experience with Microsoft BI/SSIS or comparable enterprise ETL technologies.
• Experience working with large datasets and distributed data platforms such as Hadoop/HDFS/Hive.
• Strong understanding of ETL architecture, data warehousing, dimensional modeling, and relational/3NF modeling.
• Experience implementing data-quality checks, reconciliation, audit controls, exception handling, and production monitoring.
• Experience with enterprise scheduling/orchestration tools such as AutoSys.
• Experience with Git, CI/CD, and modern development practices.
• Familiarity with Docker/Kubernetes is preferred.
• Familiarity with NoSQL technologies such as MongoDB is beneficial.
• Strong analytical, troubleshooting, communication, and problem-solving skills.
• Ability to work independently on complex technical assignments while collaborating effectively across teams.
• Experience mentoring or providing technical guidance to other developers.

Responsibilities

• Design, develop, test, and support scalable ETL/data-processing pipelines using Python/PySpark, Microsoft BI/SSIS, SQL, and related technologies.
• Develop and optimize complex SQL queries, stored procedures, functions, data transformations, and database objects for high-volume applications.
• Identify and resolve query and database performance issues through execution-plan analysis, indexing, query optimization, statistics, and appropriate database design techniques.
• Develop and support solutions processing large data volumes across SQL Server and distributed data platforms.
• Apply relational, dimensional, and data-warehouse modeling principles to build scalable and maintainable data solutions.
• Analyze failed jobs, data-quality issues, performance problems, and production incidents; perform root-cause analysis and implement sustainable resolutions.
• Develop and support enterprise batch-processing workflows using scheduling/orchestration platforms such as AutoSys.
• Work with Hadoop ecosystem technologies including HDFS, Hive and related distributed-processing technologies.
• Implement validation, reconciliation, exception handling, logging, monitoring, and data-quality controls within data pipelines.
• Participate in code reviews and enforce development standards for SQL, Python/PySpark, ETL, and data-processing components.
• Use Git-based source control and participate in CI/CD processes; work with container technologies such as Docker/Kubernetes where applicable.
• Follow secure data-development practices, access-control requirements, encryption standards, and organizational security policies.
• Work closely with application developers, infrastructure teams, business teams, and other technology groups to design and deliver solutions.
• Evaluate new technologies and leverage approved AI-assisted development tools to improve code quality, troubleshooting, refactoring, and developer productivity.

Skills

Relocation