Big data Platform Engineer

Your Impact

1. Build Data Processing and Analytics Services: Develop scalable data processing and analytics services utilizing our big data stack, including Spark, Trino, Airflow, and Kafka, to support real-time and batch data workflows.
2. Architect Distributed Systems: Design, develop, and operate resilient distributed systems that manage thousands of compute nodes across multiple data centers, ensuring scalability and high availability.
3. Troubleshoot and Innovate: Resolve complex technical challenges and drive innovations that enhance system resilience, availability, and performance.
4. Service Ownership and Live-Site Management: Manage the full lifecycle of services, balancing live-site reliability, feature development, and technical debt retirement. Maintain a strong focus on service reliability and performance.
5. On-Call Support: Participate in the team’s on-call rotation to address complex, real-time issues, keeping critical services operational and highly available.
6. Mentor and Guide Team Members: Provide mentorship and technical guidance to junior engineers, fostering growth, collaboration, and knowledge-sharing within the team.

Required Qualifications

1. Education and Experience: Bachelor’s or Master’s in Computer Science, Engineering, or a related field with 8+ years of experience in distributed systems, big data, or similar roles.
2. Experience with Cloud Environments: Proficiency in cloud platforms (AWS, GCP, or Azure), containerization (Docker, Kubernetes), and infrastructure-as-code (Terraform, Ansible).
3. Expertise in Big Data Technologies: Hands-on experience with Hadoop, Spark, Trino (or similar SQL query engines), Airflow, Kafka, and related ecosystems.
4. Proficiency in Programming: Strong skills in Python, Java, Scala, or other programming languages relevant to distributed systems.
5. Deep Understanding of Distributed Systems: Solid knowledge of distributed computing principles, data partitioning, fault tolerance, and performance tuning.
6. Analytical and Problem-Solving Skills: Proven ability to troubleshoot complex system issues, optimizing for speed, efficiency, and scale.