Yubikeys and SSH

About a year ago I found out that the Yubikey Neo can be used as a SmartCard which can keep a secret key on-board. You can also use an actual SmartCard if you have one. But the setup procedure is quite involved and you need gpg.

Yubikey to the rescue! Or maybe OpenSSH in this case: As this explains, most Yubikeys, including the cheap blue ones which can only do U2F or FIDO2, can work with OpenSSH 8.2 to provide the private key without storing the secret key unencrypted on disk. Similar to using a SmartCard, but much easier.

More important is that those keys are supported by GitHub since May 2021 and GitLab 14.8+ since March 2022.

Create a key:

❯ ssh-keygen -t ecdsa-sk
Generating public/private ecdsa-sk key pair.
You may need to touch your authenticator to authorize key generation.
Enter PIN for authenticator: 
You may need to touch your authenticator (again) to authorize key generation.
Enter file in which to save the key (/home/harald/.ssh/id_ecdsa_sk): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/harald/.ssh/id_ecdsa_sk
Your public key has been saved in /home/harald/.ssh/id_ecdsa_sk.pub
The key fingerprint is:
SHA256:2xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx8 harald@m75q
The key's randomart image is:
+-[ECDSA-SK 256]--+
|            .ooo |
|      .*+ . .    |
❯ ls -la .ssh/id_ecdsa_sk*
-rw------- 1 harald users 626 Apr 11 20:35 .ssh/id_ecdsa_sk
-rw-r--r-- 1 harald users 224 Apr 11 20:35 .ssh/id_ecdsa_sk.pub

Once the public key part is added on the target system in its ~/.ssh/authorized_keys file, you can connect to it like this:

❯ ssh -i .ssh/id_ecdsa_sk t621.lan
Confirm user presence for key ECDSA-SK SHA256:2xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx8
Welcome to Ubuntu 21.10 (GNU/Linux 5.13.0-39-generic x86_64)

Note that the private key ~/.ssh/id_ecdsa_sk is encrypted by the Yubikey, so this is a complete 2-factor authentication here, plus it checks for the user presence. And the maybe best part: that works on old U2F-only keys as well as new FIDO2 security keys. Love it!

Some FIDO2 keys can store the private key on the key directly which is convenient, but unfortunately less secure. Unlike SmartCards which have a limit for unsuccessful attempts, Yubikeys miss that feature for the U2F/FIDO2 part.

In case of errors…

I found 2 potential problems:

  • ssh-keygen failed with a “Key enrollment failed: invalid format”. If you run ssh-keygen with -vvv, you’ll see a line “debug1: sk_probe: 0 device(s) detected”. What this means is that /dev/hidrawX either does not exist or has the wrong permissions. Default is “0600” and owner is “root:root”.
    • Quick fix: sudo chmod a+rw /dev/hidrawX
    • Better fix: edit /lib/udev/rules.d/60-fido-id.rules and add a ‘, MODE=”0666″‘ to the line which starts with SUBSYSTEM==”hidraw”, then do a sudo udevadm control –reload and when you plug in the Yubikey, /dev/hidrawX will have 0666 permissions.
      You might need to install libyubikey-udev.
    • The above problem is likely only an issue when you do not use a graphical UI. If you log in via a graphical UI, all input devices should be owned by the logged in user.
  • Your OpenSSH version is older than 8.2. Check with “ssh -V”

What about ed25519-sk instead of ecdsa-sk?

ed25519-sk is only supported by FIDO2 Yubikeys keys and possibly not even all of them do depending on the firmware version.

U2F on the CLI

U2F works well and easily via a web browser, but you can also use it directly on the command line. You “just” have to implement the USB protocol part of U2F, namely talk to /dev/hidrawX.

u2fcli did that and it worked on my R2S (ARMv8):

harald@r2s2:~/git$ git clone git@github.com:mdp/u2fcli.git
Cloning into 'u2fcli'...
remote: Enumerating objects: 57, done.
remote: Total 57 (delta 0), reused 0 (delta 0), pack-reused 57
Receiving objects: 100% (57/57), 19.26 KiB | 1.20 MiB/s, done.
Resolving deltas: 100% (21/21), done.
harald@r2s2:~/git$ cd u2fcli
harald@r2s2:~/git/u2fcli$ go mod init u2fcli
go: creating new go.mod: module u2fcli
go: to add module requirements and sums:
        go mod tidy
harald@r2s2:~/git/u2fcli$ go mod tidy
go: finding module for package github.com/flynn/u2f/u2ftoken
go: finding module for package github.com/flynn/hid
go: finding module for package github.com/mdp/u2fcli/cmd
go: finding module for package github.com/flynn/u2f/u2fhid
go: finding module for package github.com/spf13/cobra
go: found github.com/mdp/u2fcli/cmd in github.com/mdp/u2fcli v0.0.0-20180327171945-2b7ae3bbca08
go: found github.com/flynn/hid in github.com/flynn/hid v0.0.0-20190502022136-f1b9b6cc019a
go: found github.com/flynn/u2f/u2fhid in github.com/flynn/u2f v0.0.0-20180613185708-15554eb68e5d
go: found github.com/flynn/u2f/u2ftoken in github.com/flynn/u2f v0.0.0-20180613185708-15554eb68e5d
go: found github.com/spf13/cobra in github.com/spf13/cobra v1.2.1
harald@r2s2:~/git/u2fcli$ go build
harald@r2s2:~/git/u2fcli$ ls
cmd  go.mod  go.sum  LICENSE  main.go  README.md  u2fcli

Permissions for /dev/hidrawX needs to be given:

harald@r2s2:~/git/u2fcli$ sudo chmod a+rw /dev/hidraw0

And now a full cycle of register (once), sign+verify (log in):

harald@r2s2:~/git/u2fcli$ ./u2fcli reg --challenge MyComplexChallenge --appid https://test.com
Registering, press the button on your U2F device #1 [Yubico Security Key by Yubico]{
  "KeyHandle": "-374aUcG7iWqVc5rsX8jE_8yr1iS-EEDdt106-CAKec90Gg1VVK9dv5E_JmZRIyKVaas9vhLVHb7zbbJ6rNltg",
  "PublicKey": "BHBwVKLRYZZKZGaL96FQtzis8i01M2DMw4IQwuMIKbWa2dZJSC1GlXlYiWhycig4R3DdlipdR675o_e4QfpI-UU",

harald@r2s2:~/git/u2fcli$ ./u2fcli sig --appid https://test.com --challenge SomethingElse --keyhandle "-374aUcG7iWqVc5rsX8jE_8yr1iS-EEDdt106-CAKec90Gg1VVK9dv5E_JmZRIyKVaas9vhLVHb7zbbJ6rNltg"
Authenticating, press the button on your U2F device
  "Counter": 50,
  "Signature": "AQAAADIwRQIhALlZyMmormC2b9JCaOXYAdKq4wvpdKg4wMu68fLgXmclAiADDHbFxKrm5eYCoCvC-m1vEEegXzWHfwuPLpUh81qHoA"

harald@r2s2:~/git/u2fcli$ ./u2fcli ver --appid https://test.com --challenge SomethingElse --publickey "BHBwVKLRYZZKZGaL96FQtzis8i01M2DMw4IQwuMIKbWa2dZJSC1GlXlYiWhycig4R3DdlipdR675o_e4QfpI-UU" --signature "AQAAADIwRQIhALlZyMmormC2b9JCaOXYAdKq4wvpdKg4wMu68fLgXmclAiADDHbFxKrm5eYCoCvC-m1vEEegXzWHfwuPLpUh81qHoA"
Signature verified

Mikrotik RouterOS and TLS

Browsers nowadays complain (rightly) about HTTP being used instead of HTTPS. My Mikrotik routers still use HTTP. Time to fix this!

❯ step ca certificate --provisioner=myCA@home --san= \
--san=sxt17.lan --not-after=8760h \
sxt17.lan sxt17.cer sxt17.key \
--provisioner-password-file ~/.step/pass/provisioner_pass.txt

❯ scp sxt17.key sxt17.cer admin@sxt17.lan:
❯ ssh admin@sxt17.lan

/certificate import file-name=sxt17.cer name=sxt17.lan passphrase=""
/certificate import file-name=sxt17.key passphrase=""
/file remove sxt17.cer
/file remove sxt17.key
/ip service set www-ssl certificate=sxt17.lan disabled=no

Note that the certificate is valid for a year instead of the default of 24h. Renewing is simple:

❯ step ca renew --force ./sxt17.cer ./sxt17.key

❯ scp sxt17.key sxt17.cer admin@sxt17.lan:
❯ ssh admin@sxt17.lan

/certificate import file-name=sxt17.cer name=sxt17.lan passphrase=""
/certificate import file-name=sxt17.key passphrase=""
/file remove sxt17.cer
/file remove sxt17.key

If the REST API of Router OS would work, I could use those instead of using ssh, but they don’t work for me. Seems version 7 of RouterOS will work with the REST API but I am not adventurous enough to verify that.

step-ca and ACME

For my home Kubernetes installation I guess it’s time to enable TLS. Can’t use Let’s Encrypt for this as my internal network is not reachable and while I have workaround for this problem, I’d rather use my internal Certificate Authority via step-ca.

It’s actually simpler than I thought it is, mainly because the documentation I found first included options which were not explained at all. Turns out they are indeed fully optional… Thus on the CA server do:

step ca provisioner add acme --type ACME

This adds the ACME provisioner to the ~/.step/config/ca.json file:

  "type": "ACME",
  "name": "myacme",
  "forceCN": true,
  "claims": {
     "maxTLSCertDuration": "12h",
     "defaultTLSCertDuration": "2h"

The first 2 items were added by above command. The next were added by me. They are optional. Restart step-ca:

harald@r2s1:~$ sudo systemctl restart step-ca
harald@r2s1:~$ systemctl status step-ca
● step-ca.service - Step Certificates
     Loaded: loaded (/etc/systemd/system/step-ca.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2021-05-11 18:27:39 JST; 15s ago
   Main PID: 547880 (step-ca)
      Tasks: 8 (limit: 998)
     Memory: 10.5M
     CGroup: /system.slice/step-ca.service
             └─547880 /usr/local/bin/step-ca /home/harald/.step/config/ca.json --password-file /home/harald/.step/pass/key_pass.txt

To create a new certificate on a different machine which runs no HTTP server on port 80:

❯ sudo REQUESTS_CA_BUNDLE=$(step path)/certs/root_ca.crt \
    certbot certonly --standalone  \
    --server https://ca.lan:8443/acme/acme/directory
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Enter email address (used for urgent renewal and security notices)
 (Enter 'c' to cancel): my.mail@some.mail.server

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please read the Terms of Service at None. You must agree in order to register
with the ACME server. Do you agree?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: y

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing, once your first certificate is successfully issued, to
share your email address with the Electronic Frontier Foundation, a founding
partner of the Let's Encrypt project and the non-profit organization that
develops Certbot? We'd like to send you email about our work encrypting the web,
EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: n
Account registered.
Please enter in your domain name(s) (comma and/or space separated)  (Enter 'c'
to cancel): m75q.lan
Requesting a certificate for m75q.lan
Performing the following challenges:
http-01 challenge for m75q.lan
Waiting for verification...
Cleaning up challenges

 - Congratulations! Your certificate and chain have been saved at:
   Your key file has been saved at:
   Your certificate will expire on 2021-05-11. To obtain a new or
   tweaked version of this certificate in the future, simply run
   certbot again. To non-interactively renew *all* of your
   certificates, run "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

And now let’s renew:

❯ sudo openssl x509 -in /etc/letsencrypt/live/m75q.lan/fullchain.pem -noout -text | grep After
            Not After : May 11 11:16:01 2021 GMT
❯ sudo REQUESTS_CA_BUNDLE=$(step path)/certs/root_ca.crt \
    certbot renew --server https://ca.lan:8443/acme/acme/directory
❯ sudo openssl x509 -in /etc/letsencrypt/live/m75q.lan/fullchain.pem -noout -text | grep After
            Not After : May 11 11:17:04 2021 GMT

This is documented here. Note that the certificate ends up in /etc/letsencrypt/live/ and because root ran the command, you need root to get it out. Not the way it should be, but this is more a test for the ACME provider for step-ca.

Fun with PIV with my YubiKey 3 Neo

Turns out that my マイナンバーカード is not the only thing which can do things like signing files and PIV is the official(?) standard for this. My old Yubikey 3 Neo can do that too thanks to the yubico-piv-tool.

And it’s basically the same as the マイナンバーカード except I have to set up everything myself.

As on my マイナンバーカード there’s 2 slots for 2 different keys and certificates:

  • Slot 9a for identification
  • Slot 9c for signing

Creating them is simple. I just show the ones for the signing slot 9c:

❯ yubico-piv-tool -s9c -AECCP256 -agenerate -o f2-9c.pub
❯ yubico-piv-tool -s9c -S'/CN=Harald Kubota/OU=Home/O=lan/' -averify -arequest -i f2-9c.pub -o f2-9c.csr
Enter PIN: 
Successfully verified PIN.
Successfully generated a certificate request.
# I need a DNS Name. And 8670h is about 1 year.
❯ step ca sign --set=dnsNames='["test5.lan"]' --not-after=8760h f2-9c.csr f2-9c.crt
✔ Provisioner: myCA@home (JWK) [kid: IFXxmmZDCX76WMNbFfUoBOBZdubx0SG45Jsd0VGxaz1]
✔ Please enter the password to decrypt the provisioner key: 
✔ CA: https://ca.lan:8443
✔ Certificate: f2-9c.crt
❯ yubico-piv-tool -s9c -aimport-certificate -i f2-9c.crt
Successfully imported a new certificate.

And here is how to sign a file and verify the signature:

❯ yubico-piv-tool -averify-pin --sign -s9c -HSHA256 -AECCP256 -i test.txt -o test.signature
Enter PIN: 
Successfully verified PIN.
Signature successful!

❯ yubico-piv-tool -s9c -aread-certificate >f2-9c.crt
❯ openssl x509 -pubkey -in f2-9c.crt -noout > f2-9c.pub
❯ openssl dgst -sha256 -verify f2-9c.pub -signature test.signature test.txt
Verified OK

マイナンバーカード fun!

Got a Smart Card reader, so the fun can begin!

PDF Signing

This is most common, so it’s best documented.

For Windows you need:

Old non-Unicode software, not well tested on non-Japanese PCs
  • Optional: Import the root certificate as it’s documented in the English description.
  • PDF Signing software from here
  • Usage is simple enough. Worked on my first try.
  • To test, get Acrobat Reader DC. Don’t forget to un-click all unwanted extra-software.
  • In Acrobat, go to Edit/Preferences and click on Signatures/Verification’s More… button and enable Windows Integration:
  • Alternatively import the user signing CA certificate into Acrobat Reader so it trusts it.

Now the previously signed PDF should show up in Acrobat as signed:


Far more interesting (for me) is signing arbitrary files to proof that those are “mine”.

myna is a simple program for Windows, Linux and Mac to do all basic things the マイナンバーカード can do. Usage is simple (Windows here):

> myna.exe jpki cms sign -i INPUTFILE -o OUTPUTFILE
> myna.exe jpki cms verify FILE

Note that the both command needs the マイナンバーカード available. The OUTPUTFILE is in PKCS#7 format. Verification via openssl is possible:

$ openssl cms -verify -inform DER -in FILE -CAfile jpki.pem

The jpki.pem is the PEM encoded root certificate from your マイナンバーカード . But in return you don’t need the card at this point.

If you want to see who signed:

$ openssl pkcs7 -print_certs -inform DER -in test-signed.p7m -noout
subject=C = JP, L = Tokyo-to, L = Xxx-shi, CN = 

issuer=C = JP, O = JPKI, OU = JPKI for digital signature, OU = Japan Agency for Local Authority Information Systems

# Many more cert details:
$ openssl pkcs7 -print_certs -inform DER -in test-signed.p7m -noout -text

# Actual verify:
$ openssl cms -verify -inform DER -in test-signed.p7m -CAfile jpki.pem -cmsout -print
  contentType: pkcs7-signedData (1.2.840.113549.1.7.2)
    version: 1
        algorithm: sha1 (
        parameter: <ABSENT>
      eContentType: pkcs7-data (1.2.840.113549.1.7.1)
        0000 - 4d 6f 64 65 6c 73 50 61-74 68 3d 22 4d 6f 64   ModelsPath="Mod
        00b4 - 76 65 32 44 22 0a                              ve2D".
          version: 2
          serialNumber: 4xxxxxx5
            algorithm: sha256WithRSAEncryption (1.2.840.113549.1.1.11)
            parameter: NULL
          issuer: C=JP, O=JPKI, OU=JPKI for digital signature, OU=Japan Agency for Local Authority Information Systems
            notBefore: Feb 15 19:11:22 2021 GMT
            notAfter: Xxx xx 14:59:59 2025 GMT
          subject: C=JP, L=Tokyo-to, L=Xxx-shi, CN=2xxxxxxxxxxxxxxxxxxxxxxxxxxA

You should be able to recognize the signing serial number: that’s the user certificate on the マイナンバーカード.

Other Notes

There’s 4 certificates on the card:

  • user certificate
  • the CA’s public certificate which signed the user certificate
  • user signing certificate
  • the CA’s public certificate which signed the user signing certificate

asn1decode shows the internal structure of data, but a lot of data is hidden in “OCTET STRING” like this:

 1637:d=8  hl=3 l= 176 cons: SEQUENCE
 1640:d=9  hl=2 l=   3 prim: OBJECT            :X509v3 Authority Key Identifier
 1645:d=9  hl=3 l= 168 prim: OCTET STRING      [HEX DUMP]:3081A580144DE017DE4B7F473DCD867A62D38B134ACE83558AA18186A48183308180310B3009060355040613024A50310D300B060355040A0C044A504B4931233021060355040B0C1A4A504B4920666F72206469676974616C207369676E6174757265313D303B060355040B0C344A6170616E204167656E637920666F72204C6F63616C20417574686F7269747920496E666F726D6174696F6E2053797374656D7382040132C4AB
 1816:d=8  hl=2 l=  29 cons: SEQUENCE
 1818:d=9  hl=2 l=   3 prim: OBJECT            :X509v3 Subject Key Identifier

You can display that data in the binary blob like this:

harald@r2s1:~/t$ openssl asn1parse -inform DER -in test-signed.p7m -strparse 1645
    0:d=0  hl=3 l= 165 cons: SEQUENCE
    3:d=1  hl=2 l=  20 prim: cont [ 0 ]
   25:d=1  hl=3 l= 134 cons: cont [ 1 ]
   28:d=2  hl=3 l= 131 cons: cont [ 4 ]
   31:d=3  hl=3 l= 128 cons: SEQUENCE
   34:d=4  hl=2 l=  11 cons: SET
   36:d=5  hl=2 l=   9 cons: SEQUENCE
   38:d=6  hl=2 l=   3 prim: OBJECT            :countryName
   43:d=6  hl=2 l=   2 prim: PRINTABLESTRING   :JP
   47:d=4  hl=2 l=  13 cons: SET
   49:d=5  hl=2 l=  11 cons: SEQUENCE
   51:d=6  hl=2 l=   3 prim: OBJECT            :organizationName
   56:d=6  hl=2 l=   4 prim: UTF8STRING        :JPKI
   62:d=4  hl=2 l=  35 cons: SET
   64:d=5  hl=2 l=  33 cons: SEQUENCE
   66:d=6  hl=2 l=   3 prim: OBJECT            :organizationalUnitName
   71:d=6  hl=2 l=  26 prim: UTF8STRING        :JPKI for digital signature
   99:d=4  hl=2 l=  61 cons: SET
  101:d=5  hl=2 l=  59 cons: SEQUENCE
  103:d=6  hl=2 l=   3 prim: OBJECT            :organizationalUnitName
  108:d=6  hl=2 l=  52 prim: UTF8STRING        :Japan Agency for Local Authority Information Systems
  162:d=1  hl=2 l=   4 prim: cont [ 2 ]

Got my マイナンバーカード!

For anyone outside Japan this is probably not of any interest. Please pass. Nothing to see here.

For me it was interesting: this is a smart card which can also use NFC, which makes it very interesting: How does it work? What data is inside? Can I look at it? Can other people look at it (without the PIN)? Why does it have 2 different PINs?

Things I learned in half a day since I got my MyNumber card:

  • It can do NFC too.
  • This app works to read data off the card. Including pictures, certificates and other stuff. Interesting.
  • On this site it explains the file system structure and other internals of the card. Very interesting.
  • That unrelated app works great to read my Suica/PASMO card. And both apps work and they figure out which card is for which app. Neat.
  • Here is a 5 year old article about how to use the data via its PKCS#11 API. And how to use this for ssh with OpenSC. I did that a while ago with a YubiKey. I prefer the YubiKey form factor a lot.
  • I got so many gadgets at home, but no Smart Card reader/writer. I should get this fixed so I can read the certificate with my PC. Makea paying tax via e-Tax much easier too.

Comparing Sandboxing Tools

The motivation came from here:


When programming in Node.js, a huge problem is that “npm install” downloads libraries you did not specify. It downloaded all dependencies listed in package.json, but it also downloaded their dependencies and the dependencies of their dependencies etc., which is code you did not explicitly ask for. While you can point your direct dependencies to trustworthy sources, you have no control about anything further down the line. In short: this is a (known) security hazard. A recent example is here. Auditing code in npm helps, but the whole concept is a fundamental problem.

Dart and Deno are reducing the problem significantly since you have to name all dependencies, but it does not necessarily help you if that dependency itself is compromised.

The runtime of Deno as well as wasmtime use a sandbox-approach to mitigate that: you have to enable access explicitly to anything: A Deno program has very few permissions otherwise. From a security point of view, this is much better.

Node.js nor Python have no sandbox model and when loading libraries from the Internet, which both do a lot, do you always know what you get? So I’m looking for choices how to retrofit programs with potentially questionable code.

My requirements:

  • Possible to use ad-hoc: I want to run a program with somehow limited access (e.g.: no root and no ability to become root, network access only when I allowed it, no access to files it does not need access to)
  • Protect my files from programs which run as me and thus with my normal privileges (e.g. very few program would need access to my ssh keys)

Test case:

  • Run a Node.js program which wants to read ~/.test_me and access http://www.google.com. It should not be able to do either unless it’s enabled.

Here the simple Node.js program:

const fs=require('fs');
const fetch=require('node-fetch');

async function accessStuff() {
  try {
    let f=await fs.promises.readFile(`${process.env.HOME}/.test_me`);
    console.log(`File: ${f}`);
  } catch(e) {
    console.log(`Error while accessing .test_me: ${e}`);
  fetch('http://www.google.com', {
                method: 'get',
    .then(res => res.text())
    .then(body => console.log(body.split('\n').slice(0,1)))
    .catch(e => {
      console.error(`Error: ${e}`);

(async function () {
try {
  await accessStuff();
} catch(e) {
  console.error(`Error: ${e}`);

and a sample run without limitations:

❯ node index.js 
File: test 1

[...many more lines from .test.me...]
  `<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage"
[...some HTML code from www.google.com...]

When it comes to security and sandboxing, those choices came up after a quick check with Google:

System-wide Mandatory Access Control (MAC)

SELinux needs special policies/contexts set up for the whole system. While this is great, it’s something root does. I see no simple way to do ad-hoc configurations to run a single command with the “correct” permissions. Plus the policy files are not easy to read nor write.

AppArmor is similar. A bit easier to read policy files, but they are all root owned, so not suitable for ad-hoc commands.

Both have a point to secure the complete system with the user explicitly not allowed to change the policies. Their purpose it not to protect the user from hurting themselves.

Sandbox Tools

Docker or containers in general provide a good way of isolation from the rest of the system and via bind mounts you can allow access to files or directories easily, but you have to create a container image first, upload it to a container registry and download and run it (Update: turns out that this is not required and a locally created image can be executed without problems). While it has its use, creating containers is a significant overhead if it’s needed for every program you are suspicious about.

minijail from Google looks good:

Minijail […] provides an executable that can be used to launch and sandbox other programs, […]


Installing on Debian was straightforward (needs kernel-headers and libcap-dev). Running a command with a specific user-definable policy is possible:

 # minijail0 -S /usr/share/minijail0/$(uname -m)/cat.policy -- \\
             /bin/cat /proc/self/seccomp_filter
but the examples/ directory was a small shock to me: a single example, and not a well explained one.

❯ cat examples/cat.policy 
# In this directory, test with:
# make LIBDIR=.
# ./minijail0 -n -S examples/cat.policy -- /bin/cat /proc/self/status
# This policy only works on x86_64.

read: 1
write: 1
restart_syscall: 1
rt_sigreturn: 1
exit_group: 1

open: 1
openat: 1
close: 1
fstat: 1
# Enforce W^X.
mmap: arg2 in ~PROT_EXEC || arg2 in ~PROT_WRITE
fadvise64: 1

While there is a tool to record the uses system calls (via strace) to create a policy (similar to what SELinux’s audit2allow tool), that means running a potentially harmful program once without restrictions. Plus the policy file is not exactly easy to understand. And the documentation does not help.

This is a dead end for my purpose.

bubblewrap is used as security layer for FlatPack installations before it was spun out. Using it is very command-line-option intensive, but a wrapper script will handle this. A test run:

❯ cat bwrap.test 

bwrap \
 --dev /dev \
 --ro-bind /lib /lib \
 --ro-bind /usr/bin /usr/bin \
 --ro-bind /bin /bin \
 --ro-bind /etc/resolv.conf /etc/resolv.conf \
 --ro-bind $HOME/js $HOME/js \
 --ro-bind $HOME/.test_me $HOME/.test_me \
 --tmpfs /tmp \
 --unshare-all \
 --share-net \
~/js/node/bin/node index.js

❯ ./bwrap.test 
~/.test_me contains: test 1
  `<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage"
[...some more HTML code...]

Removing the “–share-net” and removing the “–ro-bind” for .test_me stops both from being accessible:

❯ ./bwrap.test 
Error while accessing .test_me: Error: ENOENT: no such file or directory, open '/home/harald/.test_me'
Error: FetchError: request to http://www.google.com/ failed, reason: getaddrinfo ENOTFOUND www.google.com

Note that you also need /etc/resolv.conf too to allow resolving DNS names. Also the order of “–unshare-all” and “–share-net” is important as the last one wins.

firejail is conceptually similar to bubblewrap, but beside having a large list of command line options, it also has configuration files in /etc/firejail/ and it also allows user-owned configurations (default in ~/.config/firejail):

❯ cat ~/.config/firejail/nodejs.profile
whitelist /home/harald/js
#whitelist /home/harald/.test_me
net none
include /usr/local/etc/firejail/whitelist-common.inc
include /usr/local/etc/firejail/default.profile

❯ firejail --profile=~/.config/firejail/nodejs.profile node index.js
Reading profile /home/harald/.config/firejail/nodejs.profile
Reading profile /usr/local/etc/firejail/whitelist-common.inc
Reading profile /usr/local/etc/firejail/default.profile
Reading profile /usr/local/etc/firejail/disable-common.inc
Reading profile /usr/local/etc/firejail/disable-passwdmgr.inc
Reading profile /usr/local/etc/firejail/disable-programs.inc
Parent pid 231521, child pid 231522
Warning: cleaning all supplementary groups
Warning: cleaning all supplementary groups
Warning: cleaning all supplementary groups
Warning: cleaning all supplementary groups
Warning: cleaning all supplementary groups
Child process initialized in 94.84 ms
Error while accessing .test_me: Error: ENOENT: no such file or directory, open '/home/harald/.test_me'
Error: FetchError: request to http://www.google.com/ failed, reason: getaddrinfo ENOTFOUND www.google.com

Parent is shutting down, bye...
❯ firejail --quiet --net=none node index.js
Error while accessing .test_me: Error: EACCES: permission denied, open '/home/harald/.test_me'
Error: FetchError: request to http://www.google.com/ failed, reason: getaddrinfo ENOTFOUND www.google.com

The last sample shows that you don’t need to create a separate profile but similar to bwrap you can use command line options for most settings.

Uncommenting the “whitelist /home/harald/.test_me” line allows access to that file. Commenting out the “net none” allows network access. Per default network access is granted, but you can change this in /etc/firejail/default.profile. Once disabled in a profile, it cannot be re-enabled though. (Update: A “–ignore=net” option will ignore the “net none” in a profile).

After above tests I found out you can skip the “–profile=~/.config/firejail/PROFILENAME” if PROFILENAME is the binary name plus “.profile” as firejail will pick this up automatically. Very neat!

❯ firejail node index.js 
Reading profile /home/harald/.config/firejail/node.profile
Reading profile /etc/firejail/whitelist-common.inc

And you can make it less verbose too and with sensible defaults you don’t even need to create any profiles. E.g. shell history files are inaccessible by default:

❯ firejail --quiet bash
$ cd
$ cat .bash_history 
cat: .bash_history: Permission denied
$ ls -la .bash_history
-r-------- 1 nobody nogroup 0 Dec 30 23:39 .bash_history

My Conclusion

SELinux and AppArmor are not something users can manage by themselves. Different scope than what I am looking for.

Using containers, especially when running as non-root works as long as you want to use containers anyway. Otherwise it’s a huge overhead: create container image, store it in a registry, and then run it. Any code changes would need a new container image to be created. Good for certain workload, especially those which will run as containers anyway later on. While I use containers extensively, a lot of programs I run are not a container.

bubblewrap works. It needs an extensive list of options to be useful. That’s not hard to put into a script. Since you have to add a lot of options and there’s no default options you can specify, it’s very explicit about permissions which makes it easier to debug since everything is configured when running your suspicious program. As the order of options is important, I can see this getting complicated quickly for non-trivial programs. Here is an example. Luckily most programs are trivial: few accesses are needed plus some capabilities like network access.

firejail got the spot between security and ease-of-use right in my opinion: sensible defaults (e.g. disabling at and crontab commands) with profiles for many programs. You can also have user-configurable profiles and they are not hard to create. The amount of extra work when using firejail is low: just adding “firejail” before the command helps a lot already out-of-the-box by hiding sensitive files and disabling miss-usable commands. Creating a specific profile makes this very configurable. And if you name the profiles like the binary you plan to use, it’s both simple to use while still being configurable.

Note that no solution is 100% secure. There’s always a trade-off between convenience and security. Unless you enforce it, if it’s inconvenient, it won’t be done.

PS: While testing I experienced firejail to not be able run programs which have capabilities set if you use “caps.drop all” which is included in the default profile. See bug report. Can’t say yet if it’s a bug or badly worded option or lack of documentation or just unexpected behavior.

HTTPS on Synology’s DSM

My NAS is a Synology DS212 and it can do https. But to make it use my own CA’s certificate, a bit extra work is needed:

Add my own root CA’s Certificate

# Copy to the default folder for CA Root Certs of DSM 
cp root_ca.crt /usr/share/ca-certificates/mozilla/myCA.crt

# Linking to the system folder
ln -s /usr/share/ca-certificates/mozilla/myCA.crt /etc/ssl/certs/myCA.pem 

# Create hashed link
cd /etc/ssl/certs
ln -s myCA.pem `openssl x509 -hash -noout -in myCA.pem`.0

cat myCA.pem >> /etc/ssl/certs/ca-certificates.crt

# Testing
openssl verify -CApath /etc/ssl/certs myCA.pem

Use our own TLS Certificate

Create certificate

step ca certificate ds.lan ds.crt ds.key --kty RSA --size 2048 --not-after=8760h

DSM → Control Panel → Security → Certificate → Add. Then Configure and use the new one as system default.

Now https://ds.lan:5001 will use the new certificate. Repeat in 1 year. Since the default maximum lifetime of certificates was 720h, I had to change this to 1 year (8760h) on the step CA server:

    "minTLSCertDuration": "5m", 
    "maxTLSCertDuration": "8760h",
    "defaultTLSCertDuration": "24h",

Creating TLS Certificates for Home Use – Part 2

Part 1 was technically correct, but turns out that it’s too manual to be used by me:

  • you have to do it only once a while (once a year, because certs might have a 1 year validity time)
  • you don’t do it if it’s a lot of extra manual work

So here is Part 2 because I found something easier: Step CLI and Step CA.

The main difference to the openssl method (which continues to work): this CA runs as a service. So on the client side, you just need once to connect and then you can get certificates from a single place.

Get the releases and install, either as Debian package or tar file.

Extra step on ARMv7 (and possible all 32 bit architectures): replace “badger” with “badgerv2” in .step/config/ca.json. Also add those “claims” section under “authority” key unless you like the defaults:

    "authority": {
        "claims": {
            "minTLSCertDuration": "5m",
            "maxTLSCertDuration": "168h",
            "defaultTLSCertDuration": "24h",
            "disableRenewal": false,
            "minHostSSHCertDuration": "5m",
            "maxHostSSHCertDuration": "168h",
            "minUserSSHCertDuration": "5m",
            "maxUserSSHCertDuration": "24h"
        "provisioners": [

Then run your CA by

harald@opz3:~$ step-ca .step/config/ca.json --password-file step-ca-pw.txt
 2020/11/19 18:57:55 Serving HTTPS on :8443 …

To set up the client side, install step-cli and then do:

step ca bootstrap --ca-url https://opz3.lan:8443 \ 
                    --fingerprint 8d9345ed9c8f84729fb82005cf36b9e595c7a40efe2db52bee114c2dbdabd63d

The fingerprint you got during ca initialization, or get the root certificate and run a fingerprint:

# on CA server
> step ca root root.crt
> step certificate fingerprint root.crt

Once done, back on the client, getting a certificate is simple:

❯ step ca certificate m75q.lan m75q.crt m75q.key --provisioner-password-file ./pass.txt

and renew like this (–force to overwrite without asking):

❯ step ca renew --force m75q.crt m75q.key
# Or automated and restarting nginx:
❯ step ca renew --daemon --exec "nginx -s reload" m75q.crt m75q.key

Proper docs for step CLI and step CA. There’s also nice examples for mTLS

Update: See https://github.com/smallstep/certificates/discussions/427 which gave me some hints how to decouple the key encryption passwords for the 3 different keys:

  1. the root CA key
  2. the intermediate CA key
  3. the provisioner key

To change the passphrases:

cd $(step path)
cd secret/
cp intermediate_ca_key intermediate_ca_key.original
openssl ec -in intermediate_ca_key.original | openssl ec -out intermediate_ca_key -aes256
read EC key 
read EC key 
Enter PEM pass phrase: <OLD PASSPHRASE>
writing EC key 
writing EC key 
Enter PEM pass phrase: <NEW_PASSPHRASE>
Verifying - Enter PEM pass phrase: <NEW_PASSPHRASE>

Now when you start your CA (via

step-ca .step/config/ca.json

it’ll ask for the passphrase of the intermediate certificate. Repeat for the root certificate. You can remove it to another place as it’s not used.

The JWK provisioner’s encryptedKey (see $(step path)/config/ca.json) still has the original passphrase from the initial setup. Let’s fix this:

❯ cd $(step path)/config
jq -r '.authority.provisioners[] | select(.type=="JWK") | .encryptedKey' ca.json | step crypto jwe decrypt > k.json
Please enter the password to decrypt the content encryption key: 

On the clients side you can now use the decrypted k.json like this:

❯ TOKEN=$(step ca token foo --provisioner "provisioner@name" --key k.json)
✔ Provisioner: provisioner@name (JWK) [kid: -6pq-22r6yaQg]
❯ step ca certificate foo foo.crt foo.key --token $TOKEN 

Update 2020-11-21: This is working so well, I made an Ansible Playbook to make (re-)installs much easier for me: https://github.com/haraldkubota/step-ca

Renewing certificates is quite simple too (more details here):

step ca renew --daemon --exec "nginx -s reload" internal.crt internal.key