Data Engineer (Apache Airflow Orchestration)

Zylun Philippines, Inc.

Bacolod City, Negros Occidental
Permanent
Full-time

16 days ago

Overview:Position Summary
We are seeking an experienced Data Engineer to specialize in the orchestration and operational management of a new data warehouse project. This role is centered around an expert-level mastery of Google Cloud Composer (Apache Airflow) and the art of DAG (Directed Acyclic Graph) authoring. You will be responsible for scheduling, automating, and monitoring all data pipelines, ensuring they run on time, in the correct order, and with robust error handling. Your expertise will be vital in building a resilient, observable, and highly automated data ecosystem that the entire team can depend on. Responsibilities:Key Responsibilities

Design, build, and manage the complete orchestration framework for all data pipelines

using Google Cloud Composer. * Author complex, dynamic, and maintainable DAGs in Python, implementingsophisticated dependency management, triggers, and scheduling logic. * Serve as the team's subject matter expert on all DAG Authoring/Best Practices,including idempotency, task modularity, backfilling, and performance tuning. * Develop a comprehensive monitoring and alerting strategy for all orchestrated pipelinesto proactively identify and resolve issues. * Establish and enforce CI/CD processes for deploying and testing DAGs to ensureoperational stability. * Collaborate with the ingestion and transformation engineers to understand pipelinedependencies and integrate their workflows into the master orchestration plan. * Own the operational health of the data platform, managing retries, handling failures, andcommunicating pipeline status to stakeholders. Qualifications:

Core Required Skills

Expert-Level DAG Authoring: Demonstrable mastery of authoring complex DAGs in
Python. This must include deep knowledge of Airflow operators and sensors, dynamic DAG generation, managing task dependencies and SLAs, and creating idempotent
tasks. * Cloud Composer (Apache Airflow): Deep, hands-on experience managing aproduction Cloud Composer environment, including configuration, security, scaling, and
troubleshooting. * Python for Orchestration: Expert-level Python proficiency as it applies to writing clean,efficient, and testable Airflow DAGs. * CI/CD for Data Pipelines: Experience implementing continuous integration anddeployment workflows for DAGs using tools like Git and Google Cloud Build. * Monitoring & Alerting: Strong knowledge of setting up and using monitoring tools (like Google Cloud's operations suite) to track pipeline performance and alert on failures.

Zylun Philippines, Inc.

Apply Now