DE&A - Core - Data Engineer – Databricks

Job Description

Job Title: Offshore Data Engineer – Databricks

Experience: 6+ Years

Location: India

Employment Type: Full-Time

About the Role:

We are seeking an experienced Data Engineer (Databricks) to join our offshore data engineering team. The ideal candidate will have strong expertise in Azure Databricks, data lake architecture, and ETL/ELT pipeline development. This role involves building, optimizing, and maintaining large-scale data pipelines, collaborating with onshore teams, and ensuring seamless data integration and availability across platforms.

Key Responsibilities:

1. Design, develop, and optimize data ingestion, transformation, and loading (ETL/ELT) pipelines using Azure Databricks and PySpark.
2. Implement data engineering solutions that support analytics, BI, and data science use cases.
3. Integrate data from various sources such as SAP, APIs, Azure Data Lake, SQL databases, and streaming platforms (Kafka, Event Hubs).
4. Work closely with onshore architecture, analytics, and business teams to translate requirements into scalable data models.
5. Optimize Databricks clusters, Spark jobs, and Delta Lake performance.
6. Develop reusable components, frameworks, and standard practices for data pipelines and workflows.
7. Ensure data governance, security, and compliance best practices are followed.
8. Participate in code reviews, testing, and deployment automation using CI/CD tools.
9. Troubleshoot data issues and ensure data quality and integrity.

Required Skills & Experience:

1. 6+ years of hands-on experience in data engineering.
2. Strong working experience with Azure Databricks, PySpark, and Spark SQL.
3. Experience with Azure Data Factory (ADF), Azure Synapse, or Azure Data Lake Storage (ADLS).
4. Solid understanding of Delta Lake, Lakehouse architecture, and data partitioning strategies.
5. Proficiency in SQL and experience working with relational and non-relational databases.
6. Hands-on experience in CI/CD pipelines, Git, DevOps, and data versioning.
7. Experience with data modeling, schema design, and performance tuning.
8. Familiarity with ETL frameworks, data governance, and monitoring tools.
9. Excellent communication and collaboration skills to work effectively with distributed teams.

Preferred / Nice-to-Have Skills:

1. Knowledge of Airflow or Databricks Workflows for orchestration.
2. Exposure to streaming data (Kafka, Spark Streaming).
3. Experience with Python libraries for data processing and automation.
4. Familiarity with Power BI or other visualization tools for data validation and reporting.

Education:

1. Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field.
2. Relevant certifications (e.g., Azure Data Engineer Associate, Databricks Certified Data Engineer) are a plus.