Introduction to Kestra

• ETL, batch data pipelines, scheduled & event-driven workflows
• works with many languages (we’ll be using python)
• monitoring for workflows and executions

Objectives

1. ETL pipelines
   1. postgres
   2. google cloud
2. parameterizing execution
   1. make our workflow dynamic by being able to pass parameters at execution without having to create different workflows
3. scheduling and backfills
   1. backfills (for offline, fill in unsuccessful executions)
4. install kestra on cloud and sync flows with Git

Quick set-up

• using docker run command

  docker run --pull=always --rm -it -p 8080:8080 --user=root \\
  	-v /var/run/docker.sock:/var/run/docker.sock \\
  	-v /tmp:/tmp kestra/kestra:latest-full server local
  	
  # now you go to localhost:8080 on browser 
  

Docker Compose set up

use Docker compose to set up Kestra

• helpful for configuring settings, having a persistent DB…etc.
• you need a postgres db to persist the config data

step 1 - create the docker-compose.yaml file by running command in gitbash terminal

curl -o docker-compose.yml \\
<https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.yml>

# this will copy the docker-compose.yml file from the kestra repo