We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Spatial data processing with workflow engines

Formal Metadata

Title
Spatial data processing with workflow engines
Title of Series
Number of Parts
351
Author
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language
Production Year2022

Content Metadata

Subject Area
Genre
Abstract
Workflow engines like Apache Airflow are commonly used in data engineering nowadays. They provide an infrastructure for setting up, executing and monitoring a defined sequence of tasks, arranged as a workflow application. Tasks and dependencies are defined in a declarative way or in a programming language like Python. Airflow established using directed acyclic graphs (DAGs) to manage workflow orchestration. This talk compares a selected subset out of the huge number of available Open Source workflow engines, which are especially suited for workflows containing spatial data processing. It compares the well known Apache Airflow engine with Dagster, an other solution using DAGs and a BPMN-based workflow engine using Celery as distributed task queue. In the same space there is the new OGC API - Processes standard which is a modern REST API for wrapping computational tasks into executable processes. This talk gives an overview of the API and shows possible integrations with available workflow engines.
Keywords