STACKIT Workflows – Example DAGs

A collection of example Apache Airflow DAGs demonstrating how to use STACKIT Workflows (STACKIT's managed Airflow service) together with the STACKIT Data Platform.

What is this?

This repository contains ready-to-use example DAGs that show how to:

• Run Apache Spark jobs on Kubernetes using the stackit_workflows provider package
• Read and write data to an S3-compatible object store
• Work with an Apache Iceberg lakehouse via Dremio
• Execute Jupyter Notebooks with Papermill
• Perform Iceberg table maintenance (compaction, snapshot expiry)
• Use the KubernetesPodOperator with custom images and secrets
• Orchestrate dbt models from Airflow
• Run GPU-accelerated workloads on dedicated node pools
• Use the STACKIT Operators (STACKITSparkScriptOperator, STACKITPythonScriptOperator)

Each DAG is self-contained and heavily commented to explain every parameter.

Repository Structure

Demo/
├── stackit-01-taskflow.py                  # Basic TaskFlow API example
├── stackit-02-regular-operators.py         # Classic Airflow operators
├── stackit-03-kubernetes-operator.py       # KubernetesPodOperator basics
├── stackit-04-simple-spark.py              # Simple Spark job (single node & cluster)
├── stackit-05-spark-full-configuration.py  # All Spark operator parameters explained
├── stackit-05b-spark-full-configuration.py # Same, using the legacy stackit_airflow package
├── stackit-06-lakehouse.py                 # Read/write Iceberg tables via Dremio (SNA)
├── stackit-06daas-lakehouse.py             # Read/write Iceberg tables via Dremio-as-a-Service
├── stackit-07-extra-packages.py            # Installing extra Python packages at runtime
├── stackit-07b-extra-packages.py           # Same, using spark_kubernetes decorator
├── stackit-08-table-maintenance.py         # Iceberg table compaction & vacuum
├── stackit-09-dbt.py                       # Orchestrating dbt models
├── stackit-10-notebook.py                  # Running Jupyter notebooks with Papermill
├── stackit-11-custom-secret-podoperator.py # KubernetesPodOperator with custom secrets
├── stackit-12-schedule-on-gpu-node.py      # Scheduling workloads on GPU nodes
├── stackit-13-demo-sales-usecase.py        # End-to-end sales prediction use case
├── stackit-14-stackit-operators.py         # Overview of all STACKIT operator types
├── stackit-15-data-assets.py               # Working with STACKIT Data Assets
├── scripts/                                # Python scripts executed by the DAGs
│   ├── my_spark_job.py
│   ├── my_spark_job_with_imports.py
│   ├── my_spark_notebook.ipynb
│   ├── sales_prediction.ipynb
│   ├── basic_python_script.py
│   └── my_tools/                           # Helper modules imported by scripts
│       └── say_hello.py
├── tools/                                  # Shared utility modules
│   ├── execute_dremio_query.py
│   └── say_hello.py
└── setup-dbt.md                            # Guide for setting up dbt as a git submodule
dbt-demo/                                   # (Optional) dbt project as a git submodule
stackit-airflow-operators.md                # Reference for all STACKIT Airflow operators

Prerequisites

To run these DAGs you need:

Requirement	Details
STACKIT Workflows	A running STACKIT Managed Airflow instance
stackit_workflows provider	Installed in your Airflow environment (provided by STACKIT Workflows)
Object Storage	An S3-compatible bucket (e.g. STACKIT Object Storage) for notebooks/data
Dremio	A Dremio instance for lakehouse examples (optional)
Kubernetes	Provided automatically by STACKIT Workflows

Required Airflow Connections

Some DAGs require Airflow connections to be configured before they can run:

Connection ID	Used by	Description
lakehouse-rest	06-lakehouse, 08-table-maintenance	Dremio REST/Iceberg connection
lakehouse-rest-daas	06daas-lakehouse	Dremio-as-a-Service connection
s3_papermill_output	10-notebook, 13-demo-sales-usecase	S3 bucket for notebook output
dremio-flight	08-table-maintenance	Dremio Arrow Flight connection

See the comments at the top of each DAG file for the exact fields required.

Usage

1. Clone this repository into your STACKIT Workflows DAG repository (or use it directly as your DAG repo).
2. Configure the required Airflow connections listed above in the Airflow UI.
3. Browse the DAGs in the Airflow UI — all example DAGs are tagged with stackit-demo.
4. Trigger a DAG manually to test it.

Running Spark Jobs

The simplest Spark DAG looks like this:

from stackit_workflows.airflow_plugin.decorators import stackit

@stackit.spark_kubernetes_task(cpu=2)
def my_spark_task():
    import stackit_spark
    spark = stackit_spark.get_spark()
    spark.sql("SHOW DATABASES").show()

See Demo/stackit-04-simple-spark.py for a full example with all options.

Spark Images

The examples use the STACKIT Spark image:

schwarzit-xx-sit-dp-customer-artifactory-docker-local.jfrog.io/stackit-spark:0.4.4

You can also bring your own image — see Demo/stackit-11-custom-secret-podoperator.py for an example.

Further Reading

• stackit-airflow-operators.md — Full reference for all STACKIT Airflow operators and their parameters
• Apache Airflow Documentation
• Apache Iceberg Documentation

License

Copyright 2026 Schwarz Digits Cloud GmbH & Co. KG

Licensed under the Apache License, Version 2.0. See LICENSE for details.