Goal: deploy kestra to the cloud and use gitsync plugin to sync together

IDs:

Installation guide to set up storage, compute engine to run kestra, set up postgres db to store data related to kestra

Goal: set up instance of kestra running in prod

Create a VM instance where we’ll run kestra

  1. create VM instance and open SSH-in-browser

    1. install docker in VM for Ubuntu (also install docker compose)

      1. https://docs.docker.com/engine/install/ubuntu/

    2. install kestra using the kestra repo

      1. find the docker-compose.yml

        1. https://github.com/kestra-io/kestra/blob/develop/docker-compose.yml
        2. use curl command to create docker-compose file. make sure you grab the raw file (not html)
        curl -o docker-compose.yml \\
<https://raw.githubusercontent.com/kestra-io/kestra/refs/heads/develop/docker-compose.yml>

        1. open it with vim to configure it

          1. authentication
                  kestra:
          server:
            basicAuth:
              enabled: true
              username: "a.guan610@gmail.com" # it must be a valid email address
              password: kestra

    3. go back to your VM instance on console

      1. view network details > firewall > create firewall rule
        1. name it (e.g. kestra-ui-port)
        2. change target to “all instances in the network”
        3. set Source IPv4 range to “0.0.0.0/0” which means anything on the machine will available
        4. set Port > TCP to 8080
        5. create

    4. now we can go back to terminal and spin up $ sudo docker compose up (don’t detach yet so we can see logs in case of error)

    5. we can now access it via the external IP of our VM with port 8080 (login with prev set up auth)

      1. xx.xxx.xxx.xxx:8080

Postgres DB is by default running in the VM and we don’t have enough storage. We’re going to separate postgres and use cloud sql.

  1. Cloud > create postgres instance
  2. change the connection to “private IP” so ONLY our VM can talk to the db
  3. create user
  4. change kestra config - delete the postgres service so it won’t create a postgres db
    1. also delete the depends-on at the bottom of the yml file

image.png