Anchor your .gitignore entries

Posted on June 24, 2026 · #gitops #kubernetes #debugging

Preface

I am an avid selfhoster, and I follow the gitops pattern with ArgoCD for controlling the state of my Kubernetes (k3s) cluster. Something which I probably do different than most folks is that I vendor charts by helm pull-ing and --untar-ring them in-tree -

# in the gitops repo root
$ tree charts -L 3
charts
├── cert-manager
│   └── 1.19.1
│       ├── Chart.yaml
│       ├── templates
│       └── values.yaml
├── forgejo
│   └── 15.0.2
│       ├── Chart.lock
│       ├── Chart.yaml
│       ├── charts
│       ├── docs
│       ├── LICENSE
│       ├── README.md
│       ├── templates
│       └── values.yaml
├── genapp
│   └── 0.1.0
│       ├── Chart.yaml
│       ├── templates
│       └── values.yaml
# ... output elided

I picked up this patten at one of the previous places I worked at. I like it because it gives me git worktree like abilites -

I can edit the charts (and their different versions) in-place
I can just create my own charts in-tree without having to upload them on a helm/OCI repository
I don't have to worry about unreachable networks or hitting OCI pull rate-limits

Upgrading Forgejo

Last night, I was checking the settings of my 6 month old Forgejo deployment and I noticed I am 3 major versions behind (shudders!)

I checked the changelog, and there were no upgrade notes between v15 and v17, and so I went ahead and pulled the new chart in my gitops repo -

$ helm pull --untar --untardir charts/forgejo/17.1.1 \
    oci://code.forgejo.org/forgejo-helm/forgejo
$ mv charts/forgejo/17.1.1/{forgejo/*,}         # move the chart one dir up
$ rm -rf charts/forgejo/17.1.1/forgejo          # delete the empty dir

This is how it looked after

$ tree charts/forgejo -L 2
charts/forgejo
├── 15.0.2
│   ├── Chart.lock
│   ├── Chart.yaml
│   ├── charts
│   ├── docs
│   ├── LICENSE
│   ├── README.md
│   ├── templates
│   └── values.yaml
└── 17.1.1
    ├── Chart.lock
    ├── Chart.yaml
    ├── charts
    ├── docs
    ├── LICENSE
    ├── README.md
    ├── scripts
    ├── templates
    ├── uv.toml
    ├── values.schema.json
    └── values.yaml

After changing Forgejo's Argo app values to use the new helm chart, committing changes to git and pushing them, I eagerly waited on finding a new and healthy Forgejo deployment on my cluster. Only to find the new pod crashing, specifically the init-directories container -

$ kubectl get deployments forgejo
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
forgejo   0/1     1            0           230d

$ kubectl get pod forgejo-64454d8779-mgvzv
NAME                       READY   STATUS                  RESTARTS      AGE
forgejo-64454d8779-mgvzv   0/1     Init:CrashLoopBackOff   6 (20s ago)   8m

$ kubectl logs forgejo-64454d8779-mgvzv -c init-directories
exec /usr/sbin/init_directory_structure.sh: exec format error

Debugging

Wrong architecture ?

At first, I thought maybe, just somehow maybe, a wrong architecture's image got pulled, and so I confirmed that -

$ sudo nerdctl -a /run/k3s/containerd/containerd.sock -n k8s.io \
    image inspect \
    code.forgejo.org/forgejo/forgejo:15.0.3-rootless \
    --format '{{.Architecture}} {{.Os}} {{.Variant}}'
amd64 linux

Nope

What is `init_directory_structure.sh` ?

$ kubectl debug forgejo-64454d8779-mgvzv -it \
    --copy-to=d-forgejo \
    --container=init-directories -- sh
# got dropped into the debug container ...
/var/lib/gitea $ file /usr/sbin/init_directory_structure.sh
sh: file: not found
/var/lib/gitea $ ls -lah /usr/sbin/init_directory_structure.sh
lrwxrwxrwx 1 root git 34 Jun 24 15:37 /usr/sbin/init_directory_structure.sh -> ..data/init_directory_structure.sh
/var/lib/gitea $ cat /usr/sbin/init_directory_structure.sh
/var/lib/gitea $

The symlink was intact but the file was empty! I quickly checked the secret which was volume-mounted for this file in container, and as expected it didn't get populated.

$ kubectl get secret forgejo-init -o yaml
apiVersion: v1
kind: Secret
# .. elided content
data:
  configure_gitea.sh: ""
  configure_gpg_environment.sh: ""
  configure_ssh_signing.sh: ""
  init_directory_structure.sh: ""

Must be some drift in the chart that I didn't notice

$ diff -w \
    <(helm template forgejo charts/forgejo/15.0.2 --values environments/alnitak/forgejo/values.yaml) \
    <(helm template forgejo charts/forgejo/17.1.1 --values environments/alnitak/forgejo/values.yaml)

noisy output

    diff
<     helm.sh/chart: forgejo-15.0.2
---
>     helm.sh/chart: forgejo-17.1.1
13,14c13,14
<     app.kubernetes.io/version: "13.0.2"
<     version: "13.0.2"
---
>     app.kubernetes.io/version: "15.0.3"
>     version: "15.0.3"
54c54
<     helm.sh/chart: forgejo-15.0.2
---
>     helm.sh/chart: forgejo-17.1.1
58,59c58,59
<     app.kubernetes.io/version: "13.0.2"
<     version: "13.0.2"
---
>     app.kubernetes.io/version: "15.0.3"
>     version: "15.0.3"
69c69
<       printf "${1}\n"
---
>       printf '%b\n' "$1"
82c82,83
<       local setting="$(awk -F '=' '{print $1}' <<< "${line}" | xargs echo -n)"
---
>       local setting
>       setting="$(awk -F '=' '{print $1}' <<< "${line}" | xargs echo -n)"
124c125,126
<         local setting="$(awk -F '=' '{print $1}' <<< "${line}" | xargs echo -n)"
---
>         local setting
>         setting="$(awk -F '=' '{print $1}' <<< "${line}" | xargs echo -n)"
151c153,154
<       local section="$(basename "${config_file}")"
---
>       local section
>       section="$(basename "${config_file}")"
171c174
<         while read -d '' configFile; do
---
>         while read -r -d '' configFile; do
185,188c188,195
<       export FORGEJO__SECURITY__INTERNAL_TOKEN=$(gitea generate secret INTERNAL_TOKEN)
<       export FORGEJO__SECURITY__SECRET_KEY=$(gitea generate secret SECRET_KEY)
<       export FORGEJO__OAUTH2__JWT_SECRET=$(gitea generate secret JWT_SECRET)
<       export FORGEJO__SERVER__LFS_JWT_SECRET=$(gitea generate secret LFS_JWT_SECRET)
---
>       FORGEJO__SECURITY__INTERNAL_TOKEN=$(gitea generate secret INTERNAL_TOKEN)
>       export FORGEJO__SECURITY__INTERNAL_TOKEN
>       FORGEJO__SECURITY__SECRET_KEY=$(gitea generate secret SECRET_KEY)
>       export FORGEJO__SECURITY__SECRET_KEY
>       FORGEJO__OAUTH2__JWT_SECRET=$(gitea generate secret JWT_SECRET)
>       export FORGEJO__OAUTH2__JWT_SECRET
>       FORGEJO__SERVER__LFS_JWT_SECRET=$(gitea generate secret LFS_JWT_SECRET)
>       export FORGEJO__SERVER__LFS_JWT_SECRET
209c216
<     if [ -f ${GITEA_APP_INI} ]; then
---
>     if [ -f "${GITEA_APP_INI}" ]; then
222c229
<     environment-to-ini -o $GITEA_APP_INI
---
>     environment-to-ini -o "${GITEA_APP_INI}"
223a231
> 
232c240
<     helm.sh/chart: forgejo-15.0.2
---
>     helm.sh/chart: forgejo-17.1.1
236,237c244,245
<     app.kubernetes.io/version: "13.0.2"
<     version: "13.0.2"
---
>     app.kubernetes.io/version: "15.0.3"
>     version: "15.0.3"
245a254,262
>     
>   configure_ssh_signing.sh: |-
>     #!/usr/bin/env bash
>     set -eu
>     
>     install -m 600 /raw/ssh-signing-key /data/git/.ssh-signing/key
>     ssh-keygen -y -f /data/git/.ssh-signing/key > /data/git/.ssh-signing/key.pub
>     chmod 644 /data/git/.ssh-signing/key.pub
>     
261a279
>     
337a356
> 
346c365
<     helm.sh/chart: forgejo-15.0.2
---
>     helm.sh/chart: forgejo-17.1.1
350,351c369,370
<     app.kubernetes.io/version: "13.0.2"
<     version: "13.0.2"
---
>     app.kubernetes.io/version: "15.0.3"
>     version: "15.0.3"
373c392
<     helm.sh/chart: forgejo-15.0.2
---
>     helm.sh/chart: forgejo-17.1.1
377,378c396,397
<     app.kubernetes.io/version: "13.0.2"
<     version: "13.0.2"
---
>     app.kubernetes.io/version: "15.0.3"
>     version: "15.0.3"
402c421
<     helm.sh/chart: forgejo-15.0.2
---
>     helm.sh/chart: forgejo-17.1.1
406,407c425,426
<     app.kubernetes.io/version: "13.0.2"
<     version: "13.0.2"
---
>     app.kubernetes.io/version: "15.0.3"
>     version: "15.0.3"
412,415c431
<     type: RollingUpdate
<     rollingUpdate:
<       maxUnavailable: 0
<       maxSurge: 100%
---
>     type: Recreate
423c439
<         checksum/config: 2b6d6032d93735b7ef2b8d9f42a609b67854bd87d7bd26f165a603f3c1b0d108
---
>         checksum/config: 7e25be8665026fb35742251f68fafbc357ebc5907669fc5a21759d98ae5daf2d
425c441
<         helm.sh/chart: forgejo-15.0.2
---
>         helm.sh/chart: forgejo-17.1.1
429,430c445,446
<         app.kubernetes.io/version: "13.0.2"
<         version: "13.0.2"
---
>         app.kubernetes.io/version: "15.0.3"
>         version: "15.0.3"
438c454
<           image: "code.forgejo.org/forgejo/forgejo:13.0.2-rootless"
---
>           image: "code.forgejo.org/forgejo/forgejo:15.0.3-rootless"
466c482
<           image: "code.forgejo.org/forgejo/forgejo:13.0.2-rootless"
---
>           image: "code.forgejo.org/forgejo/forgejo:15.0.3-rootless"
518c534
<           image: "code.forgejo.org/forgejo/forgejo:13.0.2-rootless"
---
>           image: "code.forgejo.org/forgejo/forgejo:15.0.3-rootless"
562c578
<           image: "code.forgejo.org/forgejo/forgejo:13.0.2-rootless"
---
>           image: "code.forgejo.org/forgejo/forgejo:15.0.3-rootless"
640c656
<     helm.sh/chart: forgejo-15.0.2
---
>     helm.sh/chart: forgejo-17.1.1
644,645c660,661
<     app.kubernetes.io/version: "13.0.2"
<     version: "13.0.2"
---
>     app.kubernetes.io/version: "15.0.3"
>     version: "15.0.3"
672c688
<     helm.sh/chart: forgejo-15.0.2
---
>     helm.sh/chart: forgejo-17.1.1
676,677c692,693
<     app.kubernetes.io/version: "13.0.2"
<     version: "13.0.2"
---
>     app.kubernetes.io/version: "15.0.3"
>     version: "15.0.3"
694c710
<     helm.sh/chart: forgejo-15.0.2
---
>     helm.sh/chart: forgejo-17.1.1
698,699c714,715
<     app.kubernetes.io/version: "13.0.2"
<     version: "13.0.2"
---
>     app.kubernetes.io/version: "15.0.3"
>     version: "15.0.3"

There was nothing that seemed problematic in all that noise

Let me locally verify if the secret is even getting rendered

$ helm template forgejo charts/forgejo/17.1.1 --values environments/alnitak/forgejo/values.yaml \
    | grep Secret -A 100 \
    | grep init_directory_structure -A 10
  init_directory_structure.sh: |-
    #!/usr/bin/env bash

    set -euo pipefail

    set -x
    mkdir -p /data/git/.ssh
    chmod -R 700 /data/git/.ssh
    [ ! -d /data/gitea/conf ] && mkdir -p /data/gitea/conf

    # prepare temp directory structure

It was getting rendered correctly! I was perplexed. Why would there be a difference between my local helm template rendering and ArgoCD's helm rendering pipeline ?

`diff`-ing the charts

Everything seemed to be in the right place in the last stage of the helm pipeline (rendering), so my focus shifted to the precursors - the charts. I would have diff'd the chart early on if the difference between the versions was not so huge. But atleast with the context gathered about what's failing, I could narrow the diff-set down and look specifically for that part of the chart.

$ rg init_directory_structure charts/forgejo
15.0.2/templates/gitea/init.yaml
15:  init_directory_structure.sh: |-

15.0.2/templates/gitea/deployment.yaml
65:          command: ["/usr/sbin/init_directory_structure.sh"]

17.1.1/templates/gitea/init.yaml
14:  init_directory_structure.sh: |-
15:{{ tpl (.Files.Get "scripts/init_directory_structure.sh.tpl") . | indent 4 }}

17.1.1/templates/gitea/deployment.yaml
65:          command: ["/usr/sbin/init_directory_structure.sh"]

So, there was a change in how the init_directory_structure.sh secret key is rendered in the final yaml manifest. In the new chart, it referenced from the scripts/init_directory_structure.sh.tpl file within the chart.

Is the file present in-tree ?

$ fd "init_directory_structure.sh.tpl" charts/forgejo/17.1.1
charts/forgejo/17.1.1/scripts/init_directory_structure.sh.tpl

Of course it is, I had just rendered the chart locally with the secret populated.

.. And is it tracked in git (for ArgoCD) ?

$ git ls-files charts/forgejo/17.1.1/scripts/init_directory_structure.sh.tpl

# ... crickets ...

I quickly checked my .gitignore -

$ cat .gitignore
crash.log
*.DS_Store
.idea
.metals
settings.json
manifests
scripts

Ah >:

Fix

A year-older me had added some scripts within a directory called scripts in the gitops repo's root. To avoid tracking them on git, I appended .gitignore with scripts.

Since I did not anchor this entry to the repo root, git was faithfully also ignoring the scripts directory inside the Forgejo chart.

Quickly, I anchored the relevant entries, ..

diff --git a/.gitignore b/.gitignore
index 74b080f..29b8872 100644
--- a/.gitignore
+++ b/.gitignore
@@ -3,5 +3,5 @@ crash.log
 .idea
 .metals
 settings.json
-manifests
-scripts
+/manifests
+/scripts

.. git added the helm chart again (staging the files ignored before), committed the changes and pushed it upstream.

After a few minutes, I checked the deployment again

$ kubectl get deployments forgejo
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
forgejo   1/1     1            1           230d