We are currently seeking a Databricks Engineer to join our team in Hyderabad, Telangana (IN-TG), India (IN).
Key Responsibilities:
•Design, build, and maintain robust ETL/ELT data pipelines using Apache Spark on Databricks.
•Implement data Lakehouse architecture using Delta Lake for cost-effective data storage and analytics.
•Use Databricks Workflows for orchestrating batch and streaming pipelines.
•Develop and maintain CI/CD pipelines for data applications using tools such as Azure DevOps, GitHub Actions or Databricks Repos.
•Monitor pipeline performance and troubleshoot data issues in real-time and batch environments.
•Document solutions, workflows, and technical standards.
Required Skills:
•Experience in data engineering with a strong focus on Databricks and Apache Spark.
•Proficiency in PySpark, SQL, and Python.
•Experience with Delta Lake, Databricks SQL, and Unity Catalog.
•Hands-on experience with cloud platforms
•Familiarity with data lakehouse architecture, data warehousing and streaming data
•Strong understanding of ETL best practices, data partitioning, and performance tuning.
•Experience with CI/CD for data pipelines.
•Excellent problem-solving and communication skills.