We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Spatial data processing with workflow engines

Formale Metadaten

Titel
Spatial data processing with workflow engines
Serientitel
Anzahl der Teile
351
Autor
Lizenz
CC-Namensnennung 3.0 Unported:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen.
Identifikatoren
Herausgeber
Erscheinungsjahr
Sprache
Produktionsjahr2022

Inhaltliche Metadaten

Fachgebiet
Genre
Abstract
Workflow engines like Apache Airflow are commonly used in data engineering nowadays. They provide an infrastructure for setting up, executing and monitoring a defined sequence of tasks, arranged as a workflow application. Tasks and dependencies are defined in a declarative way or in a programming language like Python. Airflow established using directed acyclic graphs (DAGs) to manage workflow orchestration. This talk compares a selected subset out of the huge number of available Open Source workflow engines, which are especially suited for workflows containing spatial data processing. It compares the well known Apache Airflow engine with Dagster, an other solution using DAGs and a BPMN-based workflow engine using Celery as distributed task queue. In the same space there is the new OGC API - Processes standard which is a modern REST API for wrapping computational tasks into executable processes. This talk gives an overview of the API and shows possible integrations with available workflow engines.
Schlagwörter