Modifying libvirt firewall rules between virtual networks

Joshua Kugler asked:

On my my VM host, I have two libvirt virtual networks:

  • virbr0: 192.168.122.1/255.255.255.0
  • virbr1: 192.168.130.1/255.255.255.0

I have a VM behind each of those virtual networks. Machines can see the host, and see the Internet. However, when I try to connect from one network to the other, I get “connection refused” even though I am connecting to a port I know is open.

I have disabled UFW (host is Ubuntu) and still fails. Nothing is logged, so I’m not sure which rule is blocking it. My FORWARD block (I assume created by virtlib) is:

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             192.168.122.0/24     ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.122.0/24     anywhere            
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
ACCEPT     all  --  anywhere             192.168.130.0/24     ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.130.0/24     anywhere            
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

If I clear all the firewall rules, then the VMs on different virtual networks can talk to each other, but of course they can’t talk off the VM hosts.

I found if I got rid of these rules:

-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -o virbr1 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr1 -j REJECT --reject-with icmp-port-unreachable

The VMs could then talk to each other. That’s great, but how do I instruct libvirt to modify how it creates the forward rules so it doesn’t prohibit traffic between the virtual networks?

Thanks for any tips, pointers, or URLs!

My answer:


This is happening because you’ve configured your virtual networks as “nat” networks.

Such a network is set up so that it can only access:

  • Other virtual machines on the same virtual network
  • The host
  • The external network (usually the Internet)

In particular, access to any other virtual networks on the host is blocked, as you’ve discovered.


To solve this problem requires two steps:

  1. Reconfigure each virtual network as a “routed” network, without NAT. In this case, libvirtd will not attempt to isolate the virtual networks from each other, but it also will not perform any NAT.

    You will need to do this by editing the XML (with virsh net-edit networkname; the virt-manager GUI cannot make this change):

      <forward mode='nat'/>
    

    should be changed to:

      <forward mode='route'/>
    

    This change takes effect when you shutdown all VMs using the network, stop the network (virsh net-stop networkname), restart the network (virsh net-start networkname), and restart all VMs using the network.

  2. You will then also have to insert your own masquerading rules, if you want the virtual machines to access the Internet.

    For example, in the nat table section of /etc/ufw/before.rules:

    -A POSTROUTING -s 192.168.122.0/24 -o eth0 -j MASQUERADE
    -A POSTROUTING -s 192.168.130.0/24 -o eth0 -j MASQUERADE
    

Alternately, you can forget all that, and create a new virtual network which is isolated, and give each VM a second virtual NIC which is connected to the isolated network. The VMs can then communicate via this network.


View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.