I am using step-ca for over 2 years now on a small little NanoPi R2S (1GB RAM). And I am monitoring it too, e.g. for the last 6 months memory is very stable and more importantly: not increasing over time. Memory leaks are real, but not on this baby:
At the very end it’s changing, and it’s changing very suddenly too. Here the last 7 days:
What did not work:
Reset the server
Reboot the server
Actually it worked for less than a minute. By then memory exhaustion happened and the server was busy swapping. Connecting via ssh became a gamble at this point.
After a reboot there’s about 1min time to stop step-ca. A simple kill won’t do because systemd would restart it, so a
systemctl stop step-ca
did the job. Just have to be fast enough to execute it.
What happened?
It seems that the internal DB (BadgerV2), which by default is in ~/.step/db/ increased over time to much that its management consumed rather suddenly so much memory that swapping happened. One parameter I did not use (left it empty):
badgerFileLoadingMode [optional]: can be set to FileIO (instead of the default MemoryMap) to avoid memory-mapping log files. This can be useful in environments with low RAM. Make sure to use badgerV2 as the database type if using this option.
MemoryMap: default.
FileIO: This can be useful in environments with low RAM
Needless to say, the default works fine as long as the DB does not get too big. In my case it was 4.7GB in size:
I find it impressive that the default of MemoryMapped worked that well on a 1 GB RAM machine, but I guess in the last days it stopped working.
The Fix
Starting step-ca manually worked with no error messages, but it used more and more memory. I downgraded from v0.23.0 to 0.17.2 as I upgraded some week ago, but it made zero difference. Finding the 4 GB db solved the problem: since the config and all secret keys/certificates are not in the DB, I tried to simply wipe it out, and that worked as expected: step-ca created a new DB. When started it used about 10% memory, and then even less. Back to normal.
Lesson I learned: watch your monitoring. This behavior started on 15th and it took me 3 days to realize.
I wish I had seen this a long time ago: https://overthewire.org/wargames/ are a set of challenges in the area of security: buffer overflows, command injection, web server security, plus fun Linux command line skills.
I learned a lot about how to detect and exploit buffer overflows and shellcode in the Narnia wargame. Good fun and educational too.
Natas is nice since it’s using a web server while the other ones use ssh. The harder ones are…hard, but there’s solutions all over the web in case you get stuck.
All in all, highly recommended to see how easy it is to exploit badly written code and how to write code which does not repeat those known security problems.
All the buzz about Twitter and Elon made me look at my Twitter profile. I saw the missing date of birth, so I added it. Now my account is locked:
Here I became less than 13 years old
I am quite confident I put in a 19xx year in there, so I should be at least 22 years old, but I cannot check that now since my account is locked.
At this point I wonder if I should care. At worst in 13 years I’ll get my access back. If you send me a message in Twitter, you might have to wait 13 years for a reply…
Update 2022-07-31: Well, they like me again. It seems I managed to become 13 quite fast:
About a year ago I found out that the Yubikey Neo can be used as a SmartCard which can keep a secret key on-board. You can also use an actual SmartCard if you have one. But the setup procedure is quite involved and you need gpg.
Yubikey to the rescue! Or maybe OpenSSH in this case: As this explains, most Yubikeys, including the cheap blue ones which can only do U2F or FIDO2, can work with OpenSSH 8.2 to provide the private key without storing the secret key unencrypted on disk. Similar to using a SmartCard, but much easier.
More important is that those keys are supported by GitHub since May 2021 and GitLab 14.8+ since March 2022.
Create a key:
❯ ssh-keygen -t ecdsa-sk
Generating public/private ecdsa-sk key pair.
You may need to touch your authenticator to authorize key generation.
Enter PIN for authenticator:
You may need to touch your authenticator (again) to authorize key generation.
Enter file in which to save the key (/home/harald/.ssh/id_ecdsa_sk):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/harald/.ssh/id_ecdsa_sk
Your public key has been saved in /home/harald/.ssh/id_ecdsa_sk.pub
The key fingerprint is:
SHA256:2xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx8 harald@m75q
The key's randomart image is:
+-[ECDSA-SK 256]--+
| .ooo |
[...]
| .*+ . . |
+----[SHA256]-----+
❯ ls -la .ssh/id_ecdsa_sk*
-rw------- 1 harald users 626 Apr 11 20:35 .ssh/id_ecdsa_sk
-rw-r--r-- 1 harald users 224 Apr 11 20:35 .ssh/id_ecdsa_sk.pub
Once the public key part is added on the target system in its ~/.ssh/authorized_keys file, you can connect to it like this:
❯ ssh -i .ssh/id_ecdsa_sk t621.lan
Confirm user presence for key ECDSA-SK SHA256:2xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx8
Welcome to Ubuntu 21.10 (GNU/Linux 5.13.0-39-generic x86_64)
[...]
❯
Note that the private key ~/.ssh/id_ecdsa_sk is encrypted by the Yubikey, so this is a complete 2-factor authentication here, plus it checks for the user presence. And the maybe best part: that works on old U2F-only keys as well as new FIDO2 security keys. Love it!
Some FIDO2 keys can store the private key on the key directly which is convenient, but unfortunately less secure. Unlike SmartCards which have a limit for unsuccessful attempts, Yubikeys miss that feature for the U2F/FIDO2 part.
In case of errors…
I found 2 potential problems:
ssh-keygen failed with a “Key enrollment failed: invalid format”. If you run ssh-keygen with -vvv, you’ll see a line “debug1: sk_probe: 0 device(s) detected”. What this means is that /dev/hidrawX either does not exist or has the wrong permissions. Default is “0600” and owner is “root:root”.
Quick fix: sudo chmod a+rw /dev/hidrawX
Better fix: edit /lib/udev/rules.d/60-fido-id.rules and add a ‘, MODE=”0666″‘ to the line which starts with SUBSYSTEM==”hidraw”, then do a sudo udevadm control –reload and when you plug in the Yubikey, /dev/hidrawX will have 0666 permissions. You might need to install libyubikey-udev.
The above problem is likely only an issue when you do not use a graphical UI. If you log in via a graphical UI, all input devices should be owned by the logged in user.
Your OpenSSH version is older than 8.2. Check with “ssh -V”
U2F works well and easily via a web browser, but you can also use it directly on the command line. You “just” have to implement the USB protocol part of U2F, namely talk to /dev/hidrawX.
harald@r2s2:~/git$ git clone git@github.com:mdp/u2fcli.git
Cloning into 'u2fcli'...
remote: Enumerating objects: 57, done.
remote: Total 57 (delta 0), reused 0 (delta 0), pack-reused 57
Receiving objects: 100% (57/57), 19.26 KiB | 1.20 MiB/s, done.
Resolving deltas: 100% (21/21), done.
harald@r2s2:~/git$ cd u2fcli
harald@r2s2:~/git/u2fcli$ go mod init u2fcli
go: creating new go.mod: module u2fcli
go: to add module requirements and sums:
go mod tidy
harald@r2s2:~/git/u2fcli$ go mod tidy
go: finding module for package github.com/flynn/u2f/u2ftoken
go: finding module for package github.com/flynn/hid
go: finding module for package github.com/mdp/u2fcli/cmd
go: finding module for package github.com/flynn/u2f/u2fhid
go: finding module for package github.com/spf13/cobra
go: found github.com/mdp/u2fcli/cmd in github.com/mdp/u2fcli v0.0.0-20180327171945-2b7ae3bbca08
go: found github.com/flynn/hid in github.com/flynn/hid v0.0.0-20190502022136-f1b9b6cc019a
go: found github.com/flynn/u2f/u2fhid in github.com/flynn/u2f v0.0.0-20180613185708-15554eb68e5d
go: found github.com/flynn/u2f/u2ftoken in github.com/flynn/u2f v0.0.0-20180613185708-15554eb68e5d
go: found github.com/spf13/cobra in github.com/spf13/cobra v1.2.1
harald@r2s2:~/git/u2fcli$ go build
harald@r2s2:~/git/u2fcli$ ls
cmd go.mod go.sum LICENSE main.go README.md u2fcli
For my home Kubernetes installation I guess it’s time to enable TLS. Can’t use Let’s Encrypt for this as my internal network is not reachable and while I have workaround for this problem, I’d rather use my internal Certificate Authority via step-ca.
It’s actually simpler than I thought it is, mainly because the documentation I found first included options which were not explained at all. Turns out they are indeed fully optional… Thus on the CA server do:
step ca provisioner add acme --type ACME
This adds the ACME provisioner to the ~/.step/config/ca.json file:
The first 2 items were added by above command. The next were added by me. They are optional. Restart step-ca:
harald@r2s1:~$ sudo systemctl restart step-ca
harald@r2s1:~$ systemctl status step-ca
● step-ca.service - Step Certificates
Loaded: loaded (/etc/systemd/system/step-ca.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2021-05-11 18:27:39 JST; 15s ago
Main PID: 547880 (step-ca)
Tasks: 8 (limit: 998)
Memory: 10.5M
CGroup: /system.slice/step-ca.service
└─547880 /usr/local/bin/step-ca /home/harald/.step/config/ca.json --password-file /home/harald/.step/pass/key_pass.txt
To create a new certificate on a different machine which runs no HTTP server on port 80:
❯ sudo REQUESTS_CA_BUNDLE=$(step path)/certs/root_ca.crt \
certbot certonly --standalone \
--server https://ca.lan:8443/acme/acme/directory
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Enter email address (used for urgent renewal and security notices)
(Enter 'c' to cancel): my.mail@some.mail.server
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please read the Terms of Service at None. You must agree in order to register
with the ACME server. Do you agree?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: y
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing, once your first certificate is successfully issued, to
share your email address with the Electronic Frontier Foundation, a founding
partner of the Let's Encrypt project and the non-profit organization that
develops Certbot? We'd like to send you email about our work encrypting the web,
EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: n
Account registered.
Please enter in your domain name(s) (comma and/or space separated) (Enter 'c'
to cancel): m75q.lan
Requesting a certificate for m75q.lan
Performing the following challenges:
http-01 challenge for m75q.lan
Waiting for verification...
Cleaning up challenges
IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/m75q.lan/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/m75q.lan/privkey.pem
Your certificate will expire on 2021-05-11. To obtain a new or
tweaked version of this certificate in the future, simply run
certbot again. To non-interactively renew *all* of your
certificates, run "certbot renew"
- If you like Certbot, please consider supporting our work by:
Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le
And now let’s renew:
❯ sudo openssl x509 -in /etc/letsencrypt/live/m75q.lan/fullchain.pem -noout -text | grep After
Not After : May 11 11:16:01 2021 GMT
❯ sudo REQUESTS_CA_BUNDLE=$(step path)/certs/root_ca.crt \
certbot renew --server https://ca.lan:8443/acme/acme/directory
[...]
❯ sudo openssl x509 -in /etc/letsencrypt/live/m75q.lan/fullchain.pem -noout -text | grep After
Not After : May 11 11:17:04 2021 GMT
This is documented here. Note that the certificate ends up in /etc/letsencrypt/live/ and because root ran the command, you need root to get it out. Not the way it should be, but this is more a test for the ACME provider for step-ca.
Turns out that my マイナンバーカード is not the only thing which can do things like signing files and PIV is the official(?) standard for this. My old Yubikey 3 Neo can do that too thanks to the yubico-piv-tool.
And it’s basically the same as the マイナンバーカード except I have to set up everything myself.
As on my マイナンバーカード there’s 2 slots for 2 different keys and certificates:
Slot 9a for identification
Slot 9c for signing
Creating them is simple. I just show the ones for the signing slot 9c:
❯ yubico-piv-tool -s9c -AECCP256 -agenerate -o f2-9c.pub
❯ yubico-piv-tool -s9c -S'/CN=Harald Kubota/OU=Home/O=lan/' -averify -arequest -i f2-9c.pub -o f2-9c.csr
Enter PIN:
Successfully verified PIN.
Successfully generated a certificate request.
# I need a DNS Name. And 8670h is about 1 year.
❯ step ca sign --set=dnsNames='["test5.lan"]' --not-after=8760h f2-9c.csr f2-9c.crt
✔ Provisioner: myCA@home (JWK) [kid: IFXxmmZDCX76WMNbFfUoBOBZdubx0SG45Jsd0VGxaz1]
✔ Please enter the password to decrypt the provisioner key:
✔ CA: https://ca.lan:8443
✔ Certificate: f2-9c.crt
❯ yubico-piv-tool -s9c -aimport-certificate -i f2-9c.crt
Successfully imported a new certificate.
And here is how to sign a file and verify the signature:
For anyone outside Japan this is probably not of any interest. Please pass. Nothing to see here.
For me it was interesting: this is a smart card which can also use NFC, which makes it very interesting: How does it work? What data is inside? Can I look at it? Can other people look at it (without the PIN)? Why does it have 2 different PINs?
Things I learned in half a day since I got my MyNumber card:
It can do NFC too.
This app works to read data off the card. Including pictures, certificates and other stuff. Interesting.
On this site it explains the file system structure and other internals of the card. Very interesting.
That unrelated app works great to read my Suica/PASMO card. And both apps work and they figure out which card is for which app. Neat.
Here is a 5 year old article about how to use the data via its PKCS#11 API. And how to use this for ssh with OpenSC. I did that a while ago with a YubiKey. I prefer the YubiKey form factor a lot.
I got so many gadgets at home, but no Smart Card reader/writer. I should get this fixed so I can read the certificate with my PC. Makea paying tax via e-Tax much easier too.
Harald's Random Stuff 