osuv.de

hcloud compress volumes

I use the hetzner cloud for private use and since they introduce volumes, I use them for my production data and for my backups.
Basically I use two 10GB volumes.

  • backups
  • dockerdata

Over time the dockerdata will grow and grow. That means, when I resize the dockerdata volume, the backups volume must follow. To prevent this (and to save money), I switched the backups volume to use btrfs with ZSTD compression.

sudo mount -o discard,defaults,compress=zstd /dev/disk/by-id/scsi-0HC_Volume_1234567 /mnt/backups

With a default mkfs.btrfs the result is dissapointing

/dev/sdc        9.8G  7.4G  1.9G  80% /mnt/dockerdata
/dev/sdb         10G  6.0G  2.2G  74% /mnt/backups

The reason for the lousy result is, that btrfs isolates the data from the metadata. And for metadata it reserved 1GB on the 10GB volume.
To solve this issue, you must use the --mixed option while formatting.

Normally the data and metadata block groups are isolated. The mixed mode will remove the isolation and store both types in the same block group type.

The official recommendation is to use this option on devices < 1GB. The soft-recommendation is < 5GB. Somewhere else I've read < 16GB.
So the negative aspect might be a weak performance (yeah, rsync to my backups mountpoint results in a very high cpu load!).

The mixed mode may lead to degraded performance on larger filesystems, but is otherwise usable, even on multiple devices.

Since this is a small volume at the moment (10GB) and I give a shit about performance on my backup device, I simply start using it in mixed mode. This is the result:

/dev/sdc        9.8G  7.4G  1.9G  80% /mnt/dockerdata
/dev/sdb         10G  5.9G  4.2G  59% /mnt/backups

$ sudo compsize /mnt/backups/
Processed 50218 files, 58842 regular extents (58842 refs), 37152 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       80%      5.8G         7.1G         7.1G       
none       100%      1.4G         1.4G         1.4G       
zstd        76%      4.3G         5.7G         5.7G  

This is pretty awesome. So even when my dockerdata volume grows > 10GB, my backups volume can stay at 10GB.


summary 2019

hcloud compress volumes

I use the hetzner cloud for private use and since they introduce volumes, I use them for my production data and for my backups.
Basically I use two 10GB volumes.

  • backups
  • dockerdata

Over time the dockerdata will grow and grow. That means, when I resize the dockerdata volume, the backups volume must follow. To prevent this (and to save money), I switched the backups volume to use btrfs with ZSTD compression.

sudo mount -o discard,defaults,compress=zstd /dev/disk/by-id/scsi-0HC_Volume_1234567 /mnt/backups

With a default mkfs.btrfs the result is dissapointing

/dev/sdc        9.8G  7.4G  1.9G  80% /mnt/dockerdata
/dev/sdb         10G  6.0G  2.2G  74% /mnt/backups

The reason for the lousy result is, that btrfs isolates the data from the metadata. And for metadata it reserved 1GB on the 10GB volume.
To solve this issue, you must use the --mixed option while formatting.

Normally the data and metadata block groups are isolated. The mixed mode will remove the isolation and store both types in the same block group type.

The official recommendation is to use this option on devices < 1GB. The soft-recommendation is < 5GB. Somewhere else I've read < 16GB.
So the negative aspect might be a weak performance (yeah, rsync to my backups mountpoint results in a very high cpu load!).

The mixed mode may lead to degraded performance on larger filesystems, but is otherwise usable, even on multiple devices.

Since this is a small volume at the moment (10GB) and I give a shit about performance on my backup device, I simply start using it in mixed mode. This is the result:

/dev/sdc        9.8G  7.4G  1.9G  80% /mnt/dockerdata
/dev/sdb         10G  5.9G  4.2G  59% /mnt/backups

$ sudo compsize /mnt/backups/
Processed 50218 files, 58842 regular extents (58842 refs), 37152 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       80%      5.8G         7.1G         7.1G       
none       100%      1.4G         1.4G         1.4G       
zstd        76%      4.3G         5.7G         5.7G  

This is pretty awesome. So even when my dockerdata volume grows > 10GB, my backups volume can stay at 10GB.


Serverless Ansible Tasks on AWS

When you start scripting your infrastructure using ansible, you might come up with the idea to schedule some ansible tasks periodically.
Lambda seems a perfect place for it, because it can execute Python and Ansible is written in Python.

After a short while you'll notice that it is not that easy to invoke a playbook directly from python. So you end up with a quick'n'dirty subprocess call to invoke the playbook. It works fine - locally.

from subprocess import call
call(["ansible-playbook", playbook])

After packing everything for lambda and do a test invoke, all what lambda responses is:

ERROR! Unable to use multiprocessing, this is normally caused by lack of access to /dev/shm: [Errno 38] Function not implemented

That's modest and you can nothing do about it to fix this - and possibly AWS won't ever fix it too.

My workaround is to put everything into a Dockerfile and run this with AWS Fargate (serverless container execution).

FROM alpine:edge

RUN apk --update --no-cache add ansible python3
RUN pip3 install boto3

COPY some_infrastructure.yml /

CMD ansible-playbook some_infrastructure.yml

For me my playbook execution time is between 17 and 23 seconds. When it is triggered one time per day, the costs are: 0.00376464 USD.

> 23 * (0.25 * 0.00001406 + 0.5 * 0.00000353) * 31
0.00376464

That's near to nothing...and you can even invoke it from lambda to benefit from all triggers which are available.

summary 2018

Docker Pull Resolver Problem

Say you're running a docker swarm mode and your CI (e.g. gitlab runner) runs as a service in your swarm too.
Next to your ci service, you're running a registry inside the swarm too.

When your runner is entering the CD stage, it possibly wants to run docker push and might ended up like this.

docker push registry:5000/some_shit
Using default tag: latest
Error response from daemon: Get http://registry:5000/v2/: dial tcp: lookup registry on 10.50.0.2:53: no such host

Because you're clever, you attached the runner service and start debugging like this

# system configuration
cat /etc/resolv.conf 
search eu-central-1.compute.internal
nameserver 127.0.0.11
options timeout:2 attempts:5 ndots:0

# resolves correctly
dig registry

; <<>> DiG 9.11.2 <<>> enc-registry
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33867
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;registry.			IN	A
;; ANSWER SECTION:
registry.		600	IN	A	10.0.1.5
;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Wed Jan 24 10:48:31 UTC 2018
;; MSG SIZE  rcvd: 58

# other programms behave corretly too
curl http://registry:5000/v2/_catalog
{"repositories":["some_shit"]}

And now you don't understand the world anymore.

The puzzle solution is how you've started your CI service.

docker create service \
...
--mount type=bind,source=/var/run/docker.sock,destination=/var/run/docker.sock \
...

So it is using the docker daemon from your host system, which is not part of the swarm network and does not have the resolver information of any node in the swarm.

The solution: publish (-p 5000:5000) the ports of your registry and use localhost:5000 for pull and push in your CI job.


Docker Swarm UFW iptables

When you're running docker swarm mode, you're facing the problem that iptable rules of ufw has no affect on published ports.
This is right and a corrected behaviour and also well documentated.

There are sevaral blog entries, github projects, stackoverflow answers and so one, which are fixing this issue by disabling iptable manipulation by docker and routes everything by there own. I bet that most people who blindly copy/paste those hacks, have no clue what they're doing.

It's so damn simple

1st. simple only expose ports you want to serve public :)
Seriously. Example: You need to make a dump from your MariaDB, which runs in the docker network db.
Just run the dump from another docker container which is attached to the db network, has the backup destination mounted as a volume, start xtrabackup and you're done.

When you're not able to fulfill the first rule, just follow the 2nd rule.

2nd. simple add a iptables rule to the DOCKER-USER chain.
This chain is evaluated before docker set the iptables. Sadly, ufw is not able to apply rules in different chains. But don't worry, you can do it KISS with ansible

- name: drop public fluentd port
    become: yes
    iptables:
        action: insert
        chain: DOCKER-USER
        protocol: tcp
        destination_port: 24224
        jump: DROP
        ip_version: "ipv4"
        comment: drop fluentd port

... and you're done.

disadvantages

Yes, every solution in IT - every - has advantages and disadvantages.
When the server is rebooting, the added rules in the DOCKER-USER chain are gone!

The best idea is to use a simple systemd unit file, which invokes your ansible playbook to re-apply your addition iptables rules for the DOCKER-USER chain on reboot.

[Unit]
Description=Firewall configuration script
After=docker.service
Requires=docker.service

[Service]
Type=simple
ExecStart=/usr/local/bin/ansible-playbook /path/to/playbook/iptable.yml
Restart=on-failure
RestartSec=2

[Install]
WantedBy=basic.target

And UFW?

3rd rule: Just use ufw for everything which is not deployed via docker.


Docker Swarm Load Balancer

One of the key feature from Docker Swarm Mode is load balancing.

Demonstration

Precondition, you're running this on a manager node (when running this on your machine, simple run docker swarm init)

Given this simple Dockerfile

FROM alpine:3.6

RUN echo "red" > /index.html

EXPOSE 80
ENTRYPOINT busybox httpd -p 80 -h / && tail -f /dev/null

and build it as demo: docker build -t demo .

Now we create a new Docker overlay network named dev for this demonstration.

docker network create -d overlay --attachable dev

It's important to make it attachable, so we can demonstrate all features from docker swarm load balancing.

  • round robin load balancing
  • resolving names

We can start now our demo service

docker service create \
--network dev \
--name demo \
demo

and change the "red" in index.html to "blue" by attaching the service.

docker exec -ti \
$(docker ps --filter "Name=demo" --format {{.Names}}) \
sed -i 's/red/blue/' /index.html

It's time to scale our demo service

docker service scale demo=2

When both services are running (docker service ls), we can start our container to testing the DNSRR feature.

docker run -ti --network dev alpine:3.6

We need to install curl

/ # apk --update add curl

And finaly we can request for http://demo

/ # curl http://demo
blue
/ # curl http://demo
red
/ # curl http://demo
blue
/ # curl http://demo
red

As you see, one time the swarm resolver 127.0.0.11 requested our first demo service where we changed the index.html from "red" to "blue".
The second request resolves for the 2nd demo service which contains the original image we've build at the beginning with "red" in index.html.

It's such simple to scale (load balance) services with Docker Swarm Mode ;)