Data Engineer(ETL, SQL, SSIS, Python, AWS)
Job Description: Senior Data Engineer (AWS & Databricks)
Role Overview
We are looking for a Senior Data Engineer to architect and execute the migration of our data estate from SQL Server to an AWS-based Databricks Lakehouse. You will lead the transition from legacy stored procedures and manual processes into scalable PySpark pipelines. This role is critical for establishing our "Medallion Architecture" (Bronze/Silver/Gold) on AWS.
Key Responsibilities
Migration Leadership: Lead the technical migration strategy for moving data and logic from on-prem/RDS SQL Server to AWS S3 and Databricks.
Pipeline Engineering: Build high-performance ETL/ELT pipelines using PySpark and Scala, leveraging Databricks Workflows and Unity Catalog.
AWS Integration: Configure and optimize data ingestion patterns using AWS Glue (Crawlers), S3, and IAM for secure data access.
Code Quality: Set standards for Python/Scala development, including unit testing and CI/CD integration via AWS CodePipeline or GitHub Actions.
Required Skills & Qualifications
8+ years in Data Engineering with at least 3 years focused on AWS.
Expert level: PySpark and Databricks (Delta Lake, Photon engine, Liquid Clustering).
Strong Proficiency: SQL Server (T-SQL) for reverse-engineering legacy logic and optimizing source extraction.
Languages: Mastery of Python and SQL; professional experience with Scala.
Cloud Infrastructure: Hands-on experience with AWS S3, IAM, and Secrets Manager.
Nice to Have
Experience reading/decoding SSIS packages to extract business logic.
Databricks Certified Data Engineer Professional.
Job Description: Senior Data Engineer (AWS & Databricks)
Role Overview
We are looking for a Senior Data Engineer to architect and execute the migration of our data estate from SQL Server to an AWS-based Databricks Lakehouse. You will lead the transition from legacy stored procedures and manual processes into scalable PySpark pipelines. This role is critical for establishing our "Medallion Architecture" (Bronze/Silver/Gold) on AWS.
Key Responsibilities
Migration Leadership: Lead the technical migration strategy for moving data and logic from on-prem/RDS SQL Server to AWS S3 and Databricks.
Pipeline Engineering: Build high-performance ETL/ELT pipelines using PySpark and Scala, leveraging Databricks Workflows and Unity Catalog.
AWS Integration: Configure and optimize data ingestion patterns using AWS Glue (Crawlers), S3, and IAM for secure data access.
Code Quality: Set standards for Python/Scala development, including unit testing and CI/CD integration via AWS CodePipeline or GitHub Actions.
Required Skills & Qualifications
8+ years in Data Engineering with at least 3 years focused on AWS.
Expert level: PySpark and Databricks (Delta Lake, Photon engine, Liquid Clustering).
Strong Proficiency: SQL Server (T-SQL) for reverse-engineering legacy logic and optimizing source extraction.
Languages: Mastery of Python and SQL; professional experience with Scala.
Cloud Infrastructure: Hands-on experience with AWS S3, IAM, and Secrets Manager.
Nice to Have
Experience reading/decoding SSIS packages to extract business logic.
Databricks Certified Data Engineer Professional.