When using k3s and the built-in local persistent storage provider, once in a while you have to edit those files. While that usually works, sometimes you have to replace a 150kB binary file and when containers usually don’t have scp installed, there’s a problem…
The fix is to modify the storage from outside the container. That depends on the persistent storage provider. If it’s NFS, mount by NFS from another machine. If it’s an S3 bucket, edit it directly etc.
k3s has a local persistent storage driver called “local-path”. But where are those files so I can replace one of them? Turns out they are on /var/lib/rancher/k3s/storage/ on a node. Which node and what directory inside storage/ ?
Finding Your PVC
To find the PVC named “grafana-lib”, do
❯ kubectl describe persistentVolumeClaim grafana-lib
Name: grafana-lib
Namespace: default
StorageClass: local-path
Status: Bound
Volume: pvc-a89cee51-0000-47d7-a095-2d48400768e3
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: rancher.io/local-path
volume.kubernetes.io/selected-node: knode5
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 3Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: grafana-deployment-669fc6d658-l78z7
Events: <none>
and the volume shows where it is: knode5:/var/lib/rancher/k3s/storage/pvc-a89cee51…
A bit of jq magic and you get a complete list of all PVCs:
❯ kubectl get persistentVolumeClaims -o json | jq '[.items[] | { "Name": .metadata.name, "Volume": .spec.volumeName, "Node": .metadata.annotations."volume.kubernetes.io/selected-node" }]'
[
{
"Name": "influxdb-data",
"Volume": "pvc-bbac2312-0000-450e-aee1-41a0d5517adb",
"Node": "knode6"
},
{
"Name": "grafana-log",
"Volume": "pvc-22814c7b-0000-4b8b-99b6-ab4a4ca6c65c",
"Node": "knode5"
},
{
"Name": "grafana-lib",
"Volume": "pvc-a89cee51-0000-47d7-a095-2d48400768e3",
"Node": "knode5"
}
]