Job Summary:
As a Azure Databricks Data engineer, you will lead and implement advanced data analytics and engineering solutions using Databricks on Azure. This role requires a deep understanding of big data technologies, cloud services, and data architecture strategies. You will be instrumental in transforming data into actionable insights that drive business decisions.
Key Responsibilities:
• Design and implement scalable, high-performance data solutions using Databricks on Azure platform
• Collaborate with cross-functional teams to integrate big data solutions with existing IT infrastructure.
• Develop and optimize data pipelines, architectures, and data sets.
• Perform data modeling, data validation, and ensure data accuracy and reliability.
• Implement machine learning algorithms and data processing workflows.
• Provide expertise in data storage solutions and manage large-scale data ingestion and transformation.
• Implement CI/CD based application development methodology using tools like Azure DevOps/Jenkins/TFS/power shell etc.
• Ensure compliance with data security and privacy policies.
• Mentor junior team members and lead project segments.
Qualifications:
• Bachelor’s degree in Computer Science, Engineering, or related field.
• 6-10 years of experience in data engineering with a proven track record in using Databricks on Azure
• Strong knowledge of Python, SQL, PySpark and Scala (optional)
• Experience with cloud services such as cloud Databases, storage accounts ADLS Gen2, Azure Key vault, Cosmos DB, Azure Data factory, Azure Synapse is plus
• Experience in building metadata driven ingestion and DQ framework using PySpark
• Strong understanding of Lakehouse, Apache Spark, Delta Lake, and other big data technologies.
• Experience working with data toolsets, including data warehouse, data marts, data lake, 3NF, and dimensional model
• Experience in building pipelines using Delta live tables, autoloader, Databricks workflows for orchestration. Experience with Apache airflow will be plus.
• Experience with Databricks Unity catalog is plus.
• Experience in implementing fine grained access control using Databricks Unity catalog features is plus
• Experience in performance optimization in Databricks/Apache spark
• Demonstrated ability to work collaboratively in a team environment.
• Excellent problem-solving and analytical skills.