Bsrgvty

This is the first time I am configuring DRBD(drbd-utils 8.9.10-2) on a Ubuntu 18.04 with systemd-networkd (using netplan).

Generally the setup seems successful. I have one resource that is being synced over two hosts via a dedicated interface. The NIC of node1 and node2 are directly connected (no Switch). Another NIC is used for the hearbeat resource and ressources like webserver.

Now to the part that does not work:

When I unplug the cable of the dedicated drdb network connection, the nodes fall into the StandAlone mode instead of WFConnection. The logs show, that it tries to get into WFConnection but fails and thereafter falls back into StandAlone:

Apr 17 09:08:01 gts-test-node2 systemd-networkd[284]: ens192: Flags change: -UP -LOWER_UP -RUNNING
Apr 17 09:08:01 gts-test-node2 systemd-networkd[284]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1/link/_33 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=21 reply_cookie=0 signature=sasvas error-name=n/a error-message=n/a
Apr 17 09:08:01 gts-test-node2 systemd-networkd[284]: LLDP: Stopping LLDP client
Apr 17 09:08:01 gts-test-node2 systemd-networkd[284]: ens192: Stopped LLDP.
Apr 17 09:08:01 gts-test-node2 systemd-networkd[284]: ens192: Lost carrier
Apr 17 09:08:01 gts-test-node2 systemd-networkd[284]: ens192: State is configured, dropping config
Apr 17 09:08:01 gts-test-node2 systemd-networkd[284]: ens192: Removing address: 192.168.0.2/24 (valid forever)
Apr 17 09:08:01 gts-test-node2 systemd-timesyncd[388]: Network configuration changed, trying to establish connection.
Apr 17 09:08:01 gts-test-node2 systemd-timesyncd[388]: Synchronized to time server 91.189.94.4:123 (ntp.ubuntu.com).
Apr 17 09:08:02 gts-test-node2 corosync[480]: Apr 17 09:08:02 warning [TOTEM ] Incrementing problem counter for seqid 7082 iface 192.168.0.2 to [1 of 10]
Apr 17 09:08:02 gts-test-node2 corosync[480]: [TOTEM ] Incrementing problem counter for seqid 7082 iface 192.168.0.2 to [1 of 10]
Apr 17 09:08:02 gts-test-node2 corosync[480]: Apr 17 09:08:02 warning [TOTEM ] Incrementing problem counter for seqid 7084 iface 192.168.0.2 to [2 of 10]
Apr 17 09:08:02 gts-test-node2 corosync[480]: [TOTEM ] Incrementing problem counter for seqid 7084 iface 192.168.0.2 to [2 of 10]
Apr 17 09:08:02 gts-test-node2 kernel: drbd storage1: PingAck did not arrive in time.
Apr 17 09:08:02 gts-test-node2 kernel: drbd storage1: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Apr 17 09:08:02 gts-test-node2 kernel: drbd storage1: ack_receiver terminated
Apr 17 09:08:02 gts-test-node2 kernel: drbd storage1: Terminating drbd_a_storage1
Apr 17 09:08:03 gts-test-node2 kernel: drbd storage1: Connection closed
Apr 17 09:08:03 gts-test-node2 kernel: drbd storage1: conn( NetworkFailure -> Unconnected )
Apr 17 09:08:03 gts-test-node2 kernel: drbd storage1: receiver terminated
Apr 17 09:08:03 gts-test-node2 kernel: drbd storage1: Restarting receiver thread
Apr 17 09:08:03 gts-test-node2 kernel: drbd storage1: receiver (re)started
Apr 17 09:08:03 gts-test-node2 kernel: drbd storage1: conn( Unconnected -> WFConnection )
Apr 17 09:08:03 gts-test-node2 kernel: drbd storage1: bind before listen failed, err = -99
Apr 17 09:08:03 gts-test-node2 kernel: drbd storage1: conn( WFConnection -> Disconnecting )
Apr 17 09:08:03 gts-test-node2 kernel: drbd storage1: Connection closed
Apr 17 09:08:03 gts-test-node2 kernel: drbd storage1: conn( Disconnecting -> StandAlone )
Apr 17 09:08:03 gts-test-node2 kernel: drbd storage1: State change failed: Need a connection to start verify or resync
Apr 17 09:08:03 gts-test-node2 kernel: drbd storage1: mask = 0x1f0 val = 0x80
Apr 17 09:08:03 gts-test-node2 kernel: drbd storage1: old_conn:StandAlone wanted_conn:WFConnection
Apr 17 09:08:03 gts-test-node2 kernel: drbd storage1: receiver terminated
Apr 17 09:08:03 gts-test-node2 kernel: drbd storage1: Terminating drbd_r_storage1

This error is new to me and never happened with Ubuntu 16.04, where ifupdown was used rather than systemd-networkd:

bind before listen failed, err = -99

If I systemctl stop systemd-networkd before unplugging the cable, the behavior is correct --> it remains in WFConnection but I want to be able to use systemd-networkd and not reconfigure everything to go back to "the old way".

The disconnecting of the cable has been simulated in two different ways with the same result:

ip link set ens190 down

or simply disconnecting the interface physically.

Does anyone have an idea what process of systemd-networkd (or possibly networkd-dispatcher or other?) causes this misbehavior? I have not been able to find anything on the internet related to this topic.

Any help is highly appreciated.

I have been experiencing this issue too. Using networkd-dispatch to log the interface state I saw that when there is no-carrier the interface is stripped of its IP configuration causing drbd to make all resources standalone. This systemd behaviour seems to unhelpfully invert all sorts of established ideas about what should happen in this case.

https://github.com/systemd/systemd/commit/a9cc0189aa69a5fa1bcbacbdc69740fa3e5353db has been added but this is not included in the bionic systemd package. The option would probably also need to be exposed through netplan so the generated units set the desired behaviour.

Possible workarounds:

• Assign the drbd IP to a network bridge and add your replication interface to the bridge


• Use networkd-dispatcher to drbdadm connect all at a suitable stage

(I think the systemd behaviour is plain wrong. When it is possible to make applications bind to specific addresses I don't want that address to disappear because the peer network device is restarted, cable unplugged etc.)