Azure Databricks
Job Summary We are seeking a highly skilled Databricks Consultant with extensive experience in AWS to join our consultancy team. The ideal candidate will possess a deep understanding of data engineering and analytics, leveraging Databricks to deliver innovative solutions for our clients. This role requires a strategic thinker who can analyze complex data requirements and translate them into actionable insights.
Responsibilities
Design, develop, and implement data solutions using Databricks on AWS.
Collaborate with clients to understand their data needs and provide tailored solutions.
Optimize data pipelines and workflows for performance and scalability.
Conduct data analysis and visualization to support business decision-making.
Provide technical guidance and mentorship to junior team members.
Stay updated with the latest trends and technologies in data engineering and analytics.
Mandatory Skills
Proven experience with Databricks and its ecosystem.
Strong knowledge of AWS services related to data processing and storage.
Expertise in data modeling, ETL processes, and data warehousing.
Proficiency in programming languages such as Python or Scala.
Experience with SQL and data querying techniques.
Preferred Skills
Familiarity with machine learning frameworks and libraries.
Experience in working with big data technologies such as Apache Spark.
Knowledge of data governance and security best practices.
Strong communication and interpersonal skills.
Must have minimum 3 years of experience in Data Migration and Data Transformation projects on Azure cloud Proven expertise in building large-scale batch and real-time data pipelines with data processing frameworks in AWS and Azure cloud platforms Design and develop pipelines, ETL and analytical processes on Azure cloud using data services like Azure Data Factory, Azure Data Lake, Azure Databricks, Azure Synapse including Spark & SQL Pools Hands-on experience in Microsoft Fabric for implementing end-to-end data pipelines, Real-Time Analytics, OneLake integration, and Data Activator features Demonstrated experience with ETL development both on-premises and in the cloud using IoT, Azure Stream Analytics, Azure Event Hubs, Azure Databricks, Azure Synapse, SQL, Azure Data Lake, Azure Logic Apps, Power BI, and other ETL technologies Design and develop highly available, resilient, scalable, and secured data platforms adhering to best practices Proficient in writing automation scripts using Python and Spark (Scala) for data processing and transformation Strong analytical skills to solve complex issues in both existing and new code Solid experience in building cloud data solutions and deep understanding of Cloud Storage, Big Data Platform Services, Serverless Architectures, Hadoop, RDBMS, DW/DM, NoSQL databases, and Cloud Security Extensive hands-on experience with Azure Data Architecture components: Data Ingestion, Data Integration, Data Pipeline, Data Orchestration tools and streaming services like Event Hubs, Databricks, Stream Analytics, Logic Apps, Data Factory, Data Lakes Experience in Data Governance, including Data Security, Data Quality, and Access Controls Skilled in monitoring, debugging, diagnosing, and troubleshooting complex production systems Self-driven and able to work independently in dynamic and fast-paced environments Responsibility Translate business requirements into actionable technical specifications, defining application components, enhancement needs, data models, and integration workflows Mentor and coach application teams on the adoption of modern tools, technologies, design patterns, and Microsoft Fabric capabilities Design optimal cloud-hybrid architectures for Big Data, Data Warehousing, and orchestration pipelines, enabling smooth migration from on-premise to Azure Cloud and Microsoft Fabric Architect