Python Pyspark Azure

We are seeking a highly skilled and experienced Consultant to join our team. In this role, you will be responsible for designing and developing robust data pipelines using Apache Spark on Databricks. You will also be responsible for optimizing and troubleshooting existing pipelines for performance and reliability. As a Consultant, you will collaborate with data scientists and analysts to understand their requirements and ensure data quality, integrity, and security across various sources. Additionally, you will implement ETL processes and integrate with cloud platforms such as Azure, AWS, and GCP. Other responsibilities include maintaining and monitoring Databricks clusters and notebooks, documenting technical specifications and pipeline workflows, and staying updated on Databricks features and best practices. The ideal candidate will have at least 7 years of experience in a similar role and will work from our Virtusa office. If you are a self-motivated and detail-oriented individual with a passion for data and technology, we would love to hear from you.