We are looking for an experienced Airflow Developer to work for our client in the New York or New Jersey office.
- Develop guidelines for Airflow cluster and DAG's. Performance tuning of the DAG's and task implementation.
- Develop DAG - data pipeline to on-board and change management of datasets.
- Implement CI/CD for Airflow DAG deployment and test automation frameworks.
- Build DAG to orchestrate augmented pipeline implementing models in pipeline, running reconciliation, DQ, etc.
- Orchestrating the Airflow / workflow in hybrid cloud environment is big plus
- Working in a hybrid cloud environment.
- Developing hooks, templates, Alerts and scheduling for pipelines in Airflow.
- Minimum 2-3 years of experience on Apache Airflow in recent project.
- Python & Pandas, with demonstrated experience in ETL.
- Ability to maintain clean and secure data environments.
- Experience in setting and Schedulers: celery, Dask, Mesos.
- Well-versed with messaging queue : RabbitMQ or Redis.
- Understanding of airflow rest services and integration of airflow of platform eco-system.
- Experience working in an Agile or iterative approach to delivery preferred.
- Experience with Apache Airflow, including installing, configuring and monitoring Airflow cluster.