Michael Hampton asked:
Which is really messing with my plan to back up this machine…
I have a server which is a KVM hypervisor to several virtual machines. One of these is running Docker. It has its Docker volumes on /dev/vdb, which is set up as an LVM PV, on which Docker uses its direct-lvm driver to store Docker container data. This virtual disk is an LVM LV on the host’s local disk.
Both host and guest run Fedora 21.
The host’s view of this volume is (only the relevant volume is shown):
[root@host ~]# lvs LV VG Attr LSize docker2.example.com-volumes vm-volumes -wi-ao---- 40.00g [root@host ~]# dmsetup ls --tree vm--volumes-docker2.example.com--volumes (253:10) └─ (9:125)
The guest’s view of this volume is (again, only the relevant volume is shown):
[root@docker2 ~]# pvs PV VG Fmt Attr PSize PFree /dev/vdb docker-volumes lvm2 a-- 40.00g 0
With all the other LVM volumes on the host, I can take a snapshot with
lvcreate --snapshot, backup the snapshot and then
lvremove it with no issue. But with this particular volume, I can’t
lvremove it because it is in use:
[root@host ~]# lvremove /dev/vm-volumes/snap-docker2.example.com-volumes Logical volume vm-volumes/snap-docker2.example.com-volumes is used by another device.
Eventually I figured out that device-mapper on the host had somehow figured out that this logical volume snapshot contained an LVM PV, and then proceeded to map the logical volumes within the snapshot to the host (only the relevant volumes are shown):
[root@host ~]# dmsetup ls --tree vm--volumes-docker2.example.com--volumes (253:10) └─vm--volumes-docker2.example.com--volumes-real (253:14) └─ (9:125) docker--volumes-docker--data (253:18) └─vm--volumes-snap--docker2.example.com--volumes (253:16) ├─vm--volumes-snap--docker2.example.com--volumes-cow (253:15) │ └─ (9:125) └─vm--volumes-docker2.example.com--volumes-real (253:14) └─ (9:125) docker--volumes-docker--meta (253:17) └─vm--volumes-snap--docker2.example.com--volumes (253:16) ├─vm--volumes-snap--docker2.example.com--volumes-cow (253:15) │ └─ (9:125) └─vm--volumes-docker2.example.com--volumes-real (253:14) └─ (9:125)
These correspond exactly to the logical volumes inside the VM:
[root@docker2 ~]# lvs LV VG Attr LSize docker-data docker-volumes -wi-ao---- 39.95g docker-meta docker-volumes -wi-ao---- 44.00m
Notably, it doesn’t try to do this to the LVM LV when the system is booting, but only when I take a snapshot.
What is going on here? I really don’t want device-mapper inspecting the contents of LVM snapshots to see if there’s anything within them it can unhelpfully map for me. Can I suppress this behavior? Or do I need to create the snapshot via some other method?
Sometimes the relevant documentation is hidden away in configuration files rather than in, say, the documentation. So it seems with LVM.
By default LVM will automatically attempt to activate volumes on any physical devices which get connected to the system after boot, so long as all of the PVs are present, and lvmetad and udev (or more recently systemd) are running. When the LVM snapshot gets created, a udev event gets fired off, and since the snapshot contains a PV, lvmetad automatically runs
pvscan, and so forth.
By looking at
/etc/lvm/backup/docker-volumes I was able to determine that lvmetad had explicitly run
pvscan on the snapshot by using the device major and minor numbers, which bypassed LVM filters that would normally prevent this. The file contained:
description = "Created *after* executing 'pvscan --cache --activate ay 253:13'"
This behavior can be controlled by setting the
/etc/lvm/lvm.conf. It allows you to set which volume groups, volumes, or tags are allowed to be activated automatically.
So, I simply set the filter to contain both of the volume groups for the host; anything else won’t match the filter and does not get automatically activated.
auto_activation_volume_list = [ "mandragora", "vm-volumes" ]
The guest’s LVM volumes are no longer appearing on the host, and finally, my backups are running…
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.