GCP Data Engineer/Senior Software Engineer

In this role, you will:

1. Design and Develop Scalable Data Pipelines: Architect and implement end-to-end data workflows using Apache Airflow for orchestration, integrating multiple data sources and sinks across cloud and on-prem environments.
2. BigQuery Data Modeling and Optimization: Build and optimize data models in Google BigQuery for performance and cost-efficiency, including partitioning, clustering, and materialized views to support analytics and reporting use cases.
3. ETL/ELT Development and Maintenance: Design robust ETL/ELT pipelines to extract, transform, and load structured and semi-structured data, ensuring data quality, reliability, and availability.
4. Cloud-Native Engineering on GCP: Leverage GCP services like Cloud Storage, Pub/Sub, Dataflow, and Cloud Functions to build resilient, event-driven data workflows.
5. CI/CD and Automation: Implement CI/CD for data pipelines using tools like Cloud Composer (managed Airflow), Git, and Terraform, ensuring automated deployment and versioning of workflows.
6. Data Governance and Security: Ensure proper data classification, access control, and audit logging within GCP, adhering to data governance and compliance standards.
7. Monitoring and Troubleshooting: Build proactive monitoring for pipeline health and data quality using tools such as Stackdriver (Cloud Monitoring) and custom Airflow alerting mechanisms.
8. Collaboration and Stakeholder Engagement: Work closely with data analysts, data scientists, and business teams to understand requirements and deliver high-quality, timely data products.

Requirements

To be successful in this role, you should meet the following requirements:

1. With Overall Working experience more than 2+ Years primary in DWH/ETL background with following skill sets
2. Mandatory 1+ hands on working experience on Python is good to have.
3. Mandatory 1+ hands on working experience on GCP Bigquery (Mandatory)
4. Mandatory 1+ hands on working experience on Apache Airflow (Mandatory)
5. Mandatory 1+ hands on working experience on PL/SQL Scripting (Mandatory)
6. Mandatory 1+ hands on working experience on ETL tools (Mandatory) – Data stage/ Informatica/ Prophecy.
7. GCP Certification on ACE (Associate Cloud Engineer) is added advantage
8. Locations : Pune and Hyd.