Prevent concurrent instances of dhcp6c

[bensmithurst]

After an unexpected reboot due to a power cut earlier I noticed some dhcp6c errors in the log about XID mismatch, however IPv6 was working fine. On closer inspection it turns out there are 2 dhcp6c processes running, both started at the same time by rtsold.

That script does check for an existing pid file and if found, sends SIGHUP to an existing instance, however it seems as if there is a race condition where if the two runs of rtsold_script.sh are close enough, two instances of dhcp6c are started.

Avoiding this seems to be as simple as wrapping dhcp6c inside a flock command.

Log snippet showing 2 instances being started:

<13>1 2025-03-21T10:48:53+00:00 HOSTNAME dhcp6c 16941 - [meta sequenceId="280"] RTSOLD script - Starting dhcp6c daemon
<29>1 2025-03-21T10:48:53+00:00 HOSTNAME dhcp6c 17028 - [meta sequenceId="281"] extracted an existing DUID from /var/db/dhcp6c_duid: ****
<29>1 2025-03-21T10:48:53+00:00 HOSTNAME dhcp6c 17028 - [meta sequenceId="282"] <3>[interface] (9)
<29>1 2025-03-21T10:48:53+00:00 HOSTNAME dhcp6c 17028 - [meta sequenceId="283"] <5>[igb0] (4)
<13>1 2025-03-21T10:48:53+00:00 HOSTNAME dhcp6c 18091 - [meta sequenceId="284"] RTSOLD script - Starting dhcp6c daemon
<29>1 2025-03-21T10:48:53+00:00 HOSTNAME dhcp6c 17028 - [meta sequenceId="285"] <3>begin of closure [{] (1)

This unsurprisingly leads to interleaving of requests, and the wrong instance reading a response to the other instance's requests, leading to the mismatches:

<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 18518 - [meta sequenceId="484"] Sending Solicit
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 18518 - [meta sequenceId="485"] a new XID (f5b51) is generated
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 18518 - [meta sequenceId="486"] set client ID (len 14)
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 18518 - [meta sequenceId="487"] set elapsed time (len 2)
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 18518 - [meta sequenceId="488"] set option request (len 4)
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 18518 - [meta sequenceId="489"] set IA_PD prefix
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 18518 - [meta sequenceId="490"] set IA_PD
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 18518 - [meta sequenceId="491"] send solicit to ff02::1:2%igb0
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 18518 - [meta sequenceId="492"] reset a timer on igb0, state=SOLICIT, timeo=0, retrans=1052
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="493"] receive advertise from fe80::****%igb0 on igb0
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="494"] get DHCP option server ID, len 10
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="495"]   DUID: ****
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="496"] get DHCP option client ID, len 14
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="497"]   DUID: ****
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="498"] get DHCP option IA_PD, len 41
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="499"]   IA_PD: ID=0, T1=1800, T2=2880
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="500"] get DHCP option IA_PD prefix, len 25
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="501"]   IA_PD prefix: 2a0e:***::/56 pltime=3600 vltime=3600
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="502"] get DHCP option DNS, len 32
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="503"] XID mismatch
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="504"] Sending Solicit
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="505"] a new XID (ba5ecf) is generated
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="506"] set client ID (len 14)
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="507"] set elapsed time (len 2)
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="508"] set option request (len 4)
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="509"] set IA_PD prefix
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="510"] set IA_PD
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="511"] send solicit to ff02::1:2%igb0
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="512"] reset a timer on igb0, state=SOLICIT, timeo=0, retrans=1020
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="513"] receive advertise from fe80::****%igb0 on igb0
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="514"] get DHCP option server ID, len 10
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="515"]   DUID: ****
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="516"] get DHCP option client ID, len 14
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="517"]   DUID: ****
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="518"] get DHCP option IA_PD, len 41
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="519"]   IA_PD: ID=0, T1=1800, T2=2880
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="520"] get DHCP option IA_PD prefix, len 25
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="521"]   IA_PD prefix: 2a0e:***::/56 pltime=3600 vltime=3600
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="522"] get DHCP option DNS, len 32
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="523"] server ID: ****, pref=-1
<29>1 2025-03-21T10:48:54+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="524"] reset timer for igb0 to 0.995808
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 18518 - [meta sequenceId="525"] Sending Solicit
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 18518 - [meta sequenceId="526"] set client ID (len 14)
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 18518 - [meta sequenceId="527"] set elapsed time (len 2)
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 18518 - [meta sequenceId="528"] set option request (len 4)
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 18518 - [meta sequenceId="529"] set IA_PD prefix
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 18518 - [meta sequenceId="530"] set IA_PD
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 18518 - [meta sequenceId="531"] send solicit to ff02::1:2%igb0
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 18518 - [meta sequenceId="532"] reset a timer on igb0, state=SOLICIT, timeo=1, retrans=2008
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="533"] receive advertise from fe80::****%igb0 on igb0
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="534"] get DHCP option server ID, len 10
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="535"]   DUID: ****
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="536"] get DHCP option client ID, len 14
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="537"]   DUID: ****
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="538"] get DHCP option IA_PD, len 41
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="539"]   IA_PD: ID=0, T1=1800, T2=2880
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="540"] get DHCP option IA_PD prefix, len 25
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="541"]   IA_PD prefix: 2a0e:***::/56 pltime=3600 vltime=3600
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="542"] get DHCP option DNS, len 32
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="543"] XID mismatch
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="544"] picked a server (ID: ****)
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="545"] Sending Request
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="546"] a new XID (68923e) is generated
<29>1 2025-03-21T10:48:55+00:00 HOSTNAME dhcp6c 19076 - [meta sequenceId="547"] set client ID (len 14)

On my test instance I haven't seen the 2 instances happen for real but I have been able to simulate it to test this fix using

/var/etc/rtsold_script.sh em0 & /var/etc/rtsold_script.sh em0 &

which leads to the same XID mismatch error so seems to be representative of what can happen for real. When rebooting my test instance I have seen logs showing the double instance protection is kicking in though.

<13>1 2025-03-21T20:00:01+00:00 HOSTNAME dhcp6c 6143 - [meta sequenceId="265"] RTSOLD script - Starting dhcp6c daemon
<13>1 2025-03-21T20:00:01+00:00 HOSTNAME dhcp6c 9650 - [meta sequenceId="334"] RTSOLD script - Starting dhcp6c daemon
<13>1 2025-03-21T20:00:01+00:00 HOSTNAME dhcp6c 15501 - [meta sequenceId="335"] RTSOLD script - Failed to start dhcp6c daemon

After this I do see a single dhcp6c running as expected and IPv6 connectivity comes up as normal.

I'm not sure how/why the 2 instances of dhcp6c happened and haven't dug into the rtsold source to see exactly how it runs the wrapper script, but this seems to me to be a useful protection against two instances running.

[fichtner]

Sounds like #8431 but also that something in 25.1 caused this as we haven’t had this for a long time. The surface timing is odd, either due to FreeBSD 14.2 or the rtsold no delay option 7fe558c

[bensmithurst]

I had a quick look and I couldn't see any changes in rtsold upstream.

The two execs here in interface_dhcpv6_configure could be a problem, perhaps? Starting rtsold which will likely run rtsold_script.sh almost immediately and then immediately also running rtsold_script.sh directly.