AWS Data Engineer- Airflow

Data Engineer- Airflow
JD Design develop and maintain scalable ETL pipelines using AWS Glue PySpark and AWS Lambda
Migrate structured and semistructured data from existing data lakes eg S3 Hadoop to Amazon Redshift
Optimize data ingestion and transformation processes for performance and costefficiency
Collaborate with data architects analysts and business stakeholders to understand data requirements and ensure data quality
Implement data validation cleansing and transformation logic
Monitor and troubleshoot data pipelines and workflows
Ensure data security and compliance with organizational and regulatory standards
Document data flows architecture and processes    Responsibility
Design develop and maintain scalable ETL pipelines using AWS Glue PySpark and AWS Lambda                                
Migrate structured and semistructured data from existing data lakes eg S3 Hadoop to Amazon Redshift                                
Optimize data ingestion and transformation processes for performance and costefficiency                                
Primary Skills
AWS Glue jobs
AWS services  Lambda S3
AWS database  Hadoop and Redshift
Communication
Usage of AI code assist tools
Working knowledge of implementing data migration