Makefile: Update mtime of $(TARGET_DIR)/usr

Message ID 20180416153339.23002-1-chris.lesiak@licor.com
State Superseded
Headers show
Series
  • Makefile: Update mtime of $(TARGET_DIR)/usr
Related show

Commit Message

Chris Lesiak April 16, 2018, 3:33 p.m.
For the systemd-update-done.service to function, updates to /usr
must be followed by an update of the modification time of /usr.
If updates were always created with a clean build, the mtime of
/usr would automatically be newer than in the old build; but
especially during development, that may not always be the case.
So touch $(TARGET_DIR)/usr as the last step of target-finalize.

For more details, see:
http://0pointer.de/public/systemd-man/systemd-update-done.service.html

Signed-off-by: Chris Lesiak <chris.lesiak@licor.com>
---
 Makefile | 2 ++
 1 file changed, 2 insertions(+)

Comments

Yann E. MORIN April 16, 2018, 8:18 p.m. | #1
Chris, All,

On 2018-04-16 10:33 -0500, Chris Lesiak spake thusly:
> For the systemd-update-done.service to function, updates to /usr
> must be followed by an update of the modification time of /usr.
> If updates were always created with a clean build, the mtime of
> /usr would automatically be newer than in the old build; but
> especially during development, that may not always be the case.
> So touch $(TARGET_DIR)/usr as the last step of target-finalize.
> 
> For more details, see:
> http://0pointer.de/public/systemd-man/systemd-update-done.service.html

I'm not sure I folowed corretly.. So let me try to rephrase with my own
understanding of the issue.

If /usr it not mtime-newer than /var and /etc, then the laters will not
be correctly populated with vendor-provided resources (aka factory
settings). This is especially critical after a build, so that the
first-boot condition get correctly detected, so that populating /var
and /etc gets triggered.

So, we touch /usr so that the first-boot condition is correctly
detected.

Subsequent updates to /usr at runtime are not covered by this trick,
obviously.

Right?

Regards,
Yann E. MORIN.

> Signed-off-by: Chris Lesiak <chris.lesiak@licor.com>
> ---
>  Makefile | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index 957ba6634a..12a53840c0 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -761,6 +761,8 @@ endif
>  		$(call MESSAGE,"Executing post-build script $(s)"); \
>  		$(EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
>  
> +	touch $(TARGET_DIR)/usr
> +
>  .PHONY: target-post-image
>  target-post-image: $(TARGETS_ROOTFS) target-finalize
>  	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
> -- 
> 2.14.3
> 
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
Chris Lesiak April 16, 2018, 9:27 p.m. | #2
On 04/16/2018 03:18 PM, Yann E. MORIN wrote:

> Chris, All,
>
> On 2018-04-16 10:33 -0500, Chris Lesiak spake thusly:
>> For the systemd-update-done.service to function, updates to /usr
>> must be followed by an update of the modification time of /usr.
>> If updates were always created with a clean build, the mtime of
>> /usr would automatically be newer than in the old build; but
>> especially during development, that may not always be the case.
>> So touch $(TARGET_DIR)/usr as the last step of target-finalize.
>>
>> For more details, see:
>> http://0pointer.de/public/systemd-man/systemd-update-done.service.html
> I'm not sure I folowed corretly.. So let me try to rephrase with my own
> understanding of the issue.
>
> If /usr it not mtime-newer than /var and /etc, then the laters will not
> be correctly populated with vendor-provided resources (aka factory
> settings).

That is correct except in detail.  The mtime of /usr is compared to the 
mtime of /etc/.updated and /var/.updated, not the /etc and /var 
directories themselves.

The systemd-update-done.service transfers the mtime of /usr to 
/etc/.updated and /var/.updated.  Other services that want to run when 
the system has been updated will arrange themselves to run before 
systemd-update-done.service and condition themselves on 
ConditionNeedsUpdate=/etc and/or ConditionNeedsUpdate=/var.

> This is especially critical after a build, so that the
> first-boot condition get correctly detected, so that populating /var
> and /etc gets triggered.
>
> So, we touch /usr so that the first-boot condition is correctly
> detected.

It isn't the ConditionFirstBoot I'm concerned with here, but 
ConditionNeedsUpdate.

ConditionFirstBoot will be true only if /etc is not populated.  If 
/etc/machine-id exists, then /etc is considered to be populated.

> Subsequent updates to /usr at runtime are not covered by this trick,
> obviously.
>
> Right?

Maybe you have some data to refute this, but without a package manager, 
I suspect that most buildroot users do updates of a device by replacing 
the entire contents of /usr and maybe even /.  In my particular case I 
use fwup to replace /usr with a ping/pong scheme alternating between two 
partitions.

Updating using rsync -t (and without -O) would also benefit from 
touching /usr at build time.

Other update mechanisms might need to touch /usr at runtime.

>
> Regards,
> Yann E. MORIN.
>
>> Signed-off-by: Chris Lesiak <chris.lesiak@licor.com>
>> ---
>>   Makefile | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/Makefile b/Makefile
>> index 957ba6634a..12a53840c0 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -761,6 +761,8 @@ endif
>>                $(call MESSAGE,"Executing post-build script $(s)"); \
>>                $(EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
>>
>> +     touch $(TARGET_DIR)/usr
>> +
>>   .PHONY: target-post-image
>>   target-post-image: $(TARGETS_ROOTFS) target-finalize
>>        @$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
>> --
>> 2.14.3
>>
>> _______________________________________________
>> buildroot mailing list
>> buildroot@busybox.net
>> http://lists.busybox.net/mailman/listinfo/buildroot
> --
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'
>
Yann E. MORIN April 22, 2018, 9:37 p.m. | #3
Chris, All,

On 2018-04-16 16:27 -0500, Chris Lesiak spake thusly:
> On 04/16/2018 03:18 PM, Yann E. MORIN wrote:
> >On 2018-04-16 10:33 -0500, Chris Lesiak spake thusly:
> >>For the systemd-update-done.service to function, updates to /usr
> >>must be followed by an update of the modification time of /usr.
> >>If updates were always created with a clean build, the mtime of
> >>/usr would automatically be newer than in the old build; but
> >>especially during development, that may not always be the case.
> >>So touch $(TARGET_DIR)/usr as the last step of target-finalize.
> >>
> >>For more details, see:
> >>http://0pointer.de/public/systemd-man/systemd-update-done.service.html
> >I'm not sure I folowed corretly.. So let me try to rephrase with my own
> >understanding of the issue.
> >
> >If /usr it not mtime-newer than /var and /etc, then the laters will not
> >be correctly populated with vendor-provided resources (aka factory
> >settings).
> 
> That is correct except in detail.  The mtime of /usr is compared to the
> mtime of /etc/.updated and /var/.updated, not the /etc and /var directories
> themselves.
> 
> The systemd-update-done.service transfers the mtime of /usr to /etc/.updated
> and /var/.updated.  Other services that want to run when the system has been
> updated will arrange themselves to run before systemd-update-done.service
> and condition themselves on ConditionNeedsUpdate=/etc and/or
> ConditionNeedsUpdate=/var.

OK, thanks.

> >This is especially critical after a build, so that the
> >first-boot condition get correctly detected, so that populating /var
> >and /etc gets triggered.
> >
> >So, we touch /usr so that the first-boot condition is correctly
> >detected.
> 
> It isn't the ConditionFirstBoot I'm concerned with here, but
> ConditionNeedsUpdate.
> 
> ConditionFirstBoot will be true only if /etc is not populated.  If
> /etc/machine-id exists, then /etc is considered to be populated.

ACK.

> >Subsequent updates to /usr at runtime are not covered by this trick,
> >obviously.
> >
> >Right?
> 
> Maybe you have some data to refute this, but without a package manager, I
> suspect that most buildroot users do updates of a device by replacing the
> entire contents of /usr and maybe even /.  In my particular case I use fwup
> to replace /usr with a ping/pong scheme alternating between two partitions.

I even think that most use a read-only filesystem, so they update by
just dumping the new image onto the raw device (either with a dual-bank,
or a main vs. rescue sytems).

> Updating using rsync -t (and without -O) would also benefit from touching
> /usr at build time.
> 
> Other update mechanisms might need to touch /usr at runtime.

OK, so I eventually understood what it is good for. In fact, it is not
usefull for the first system one builds, but for the next versions.

The mtime of /usr of (say) version 2 will be more recent than the mtime
of /etc/.updated and/or /var/.updated of a system running version 1.

Then, when the /usr version 2 is dumped into a /usr of the running
system, the /etc/.updated and/or /var/.updated will be older than /usr,
which would kick the update mechanism.

Correct?

If so, I see a very big issue with this mechanism. For example:

    date    what
    0       build version 1
    1       version is flahes onto a device, boxed and shipped
    2       build version 2
    3       the device with version 1 is received, unboxed and booted
    4       that device downloads version 2 and updates to it

As you can see, the /etc/.updated and/or /var/.updated will dated '3'
but the mtime of /usr version 2 will be 2 (because touched at build
time).

Thus, when the system eventually updates to version 2, the mtime does
not trigger an update.

So, if you want to be able to correctly trigger the update mchanism, I
think the touch should be done by your extractor at runtime, right after
extracting the content of /usr.

Unless I missed something (very likely), in which case I'm ready to be
lectured. ;-)

Regards,
Yann E. MORIN.

> >Regards,
> >Yann E. MORIN.
> >
> >>Signed-off-by: Chris Lesiak <chris.lesiak@licor.com>
> >>---
> >>  Makefile | 2 ++
> >>  1 file changed, 2 insertions(+)
> >>
> >>diff --git a/Makefile b/Makefile
> >>index 957ba6634a..12a53840c0 100644
> >>--- a/Makefile
> >>+++ b/Makefile
> >>@@ -761,6 +761,8 @@ endif
> >>               $(call MESSAGE,"Executing post-build script $(s)"); \
> >>               $(EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
> >>
> >>+     touch $(TARGET_DIR)/usr
> >>+
> >>  .PHONY: target-post-image
> >>  target-post-image: $(TARGETS_ROOTFS) target-finalize
> >>       @$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
> >>--
> >>2.14.3
> >>
> >>_______________________________________________
> >>buildroot mailing list
> >>buildroot@busybox.net
> >>http://lists.busybox.net/mailman/listinfo/buildroot
> >--
> >.-----------------.--------------------.------------------.--------------------.
> >|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> >| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> >| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> >| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> >'------------------------------^-------^------------------^--------------------'
> >
> 
> -- 
> Chris Lesiak
> Principal Design Engineer, Software
> LI-COR Biosciences
> chris.lesiak@licor.com
> 
> Any opinions expressed are those of the author and
> do not necessarily represent those of his employer.
>
Chris Lesiak April 24, 2018, 6:56 p.m. | #4
On 04/22/2018 04:37 PM, Yann E. MORIN wrote:

> Chris, All,
>
> On 2018-04-16 16:27 -0500, Chris Lesiak spake thusly:
>> On 04/16/2018 03:18 PM, Yann E. MORIN wrote:
>>> On 2018-04-16 10:33 -0500, Chris Lesiak spake thusly:
>>>> For the systemd-update-done.service to function, updates to /usr
>>>> must be followed by an update of the modification time of /usr.
>>>> If updates were always created with a clean build, the mtime of
>>>> /usr would automatically be newer than in the old build; but
>>>> especially during development, that may not always be the case.
>>>> So touch $(TARGET_DIR)/usr as the last step of target-finalize.
>>>>
>>>> For more details, see:
>>>> http://0pointer.de/public/systemd-man/systemd-update-done.service.html
>>> I'm not sure I folowed corretly.. So let me try to rephrase with my own
>>> understanding of the issue.
>>>
>>> If /usr it not mtime-newer than /var and /etc, then the laters will not
>>> be correctly populated with vendor-provided resources (aka factory
>>> settings).
>> That is correct except in detail.  The mtime of /usr is compared to the
>> mtime of /etc/.updated and /var/.updated, not the /etc and /var directories
>> themselves.
>>
>> The systemd-update-done.service transfers the mtime of /usr to /etc/.updated
>> and /var/.updated.  Other services that want to run when the system has been
>> updated will arrange themselves to run before systemd-update-done.service
>> and condition themselves on ConditionNeedsUpdate=/etc and/or
>> ConditionNeedsUpdate=/var.
> OK, thanks.
>
>>> This is especially critical after a build, so that the
>>> first-boot condition get correctly detected, so that populating /var
>>> and /etc gets triggered.
>>>
>>> So, we touch /usr so that the first-boot condition is correctly
>>> detected.
>> It isn't the ConditionFirstBoot I'm concerned with here, but
>> ConditionNeedsUpdate.
>>
>> ConditionFirstBoot will be true only if /etc is not populated.  If
>> /etc/machine-id exists, then /etc is considered to be populated.
> ACK.
>
>>> Subsequent updates to /usr at runtime are not covered by this trick,
>>> obviously.
>>>
>>> Right?
>> Maybe you have some data to refute this, but without a package manager, I
>> suspect that most buildroot users do updates of a device by replacing the
>> entire contents of /usr and maybe even /.  In my particular case I use fwup
>> to replace /usr with a ping/pong scheme alternating between two partitions.
> I even think that most use a read-only filesystem, so they update by
> just dumping the new image onto the raw device (either with a dual-bank,
> or a main vs. rescue sytems).
>
>> Updating using rsync -t (and without -O) would also benefit from touching
>> /usr at build time.
>>
>> Other update mechanisms might need to touch /usr at runtime.
> OK, so I eventually understood what it is good for. In fact, it is not
> usefull for the first system one builds, but for the next versions.
>
> The mtime of /usr of (say) version 2 will be more recent than the mtime
> of /etc/.updated and/or /var/.updated of a system running version 1.
>
> Then, when the /usr version 2 is dumped into a /usr of the running
> system, the /etc/.updated and/or /var/.updated will be older than /usr,
> which would kick the update mechanism.
>
> Correct?

Yes.

>
> If so, I see a very big issue with this mechanism. For example:
>
>      date    what
>      0       build version 1
>      1       version is flahes onto a device, boxed and shipped
>      2       build version 2
>      3       the device with version 1 is received, unboxed and booted
>      4       that device downloads version 2 and updates to it
>
> As you can see, the /etc/.updated and/or /var/.updated will dated '3'
> but the mtime of /usr version 2 will be 2 (because touched at build
> time).

Not quite.

If systemd-update-done if working correctly, then on date 3, 
/etc/.updated and /var/.updated will get created with mtime 0, not mtime 
3.  Then on date 4, /etc/.updated and /var/.updated will get updated 
(because 0 is less than 2) to mtime 2.

When /etc/.updated and /var/.updated are originally created and then 
again when they are updated, their mtime will match the mtime of /usr, 
not the current time when systemd-update-done runs.

>
> Thus, when the system eventually updates to version 2, the mtime does
> not trigger an update.
>
> So, if you want to be able to correctly trigger the update mchanism, I
> think the touch should be done by your extractor at runtime, right after
> extracting the content of /usr.
>
> Unless I missed something (very likely), in which case I'm ready to be
> lectured. ;-)

Thanks for reviewing this and asking questions.  It caused me to dig a 
little deeper and find a systemd-update-done regression in v234 that 
will be fixed in v239.  See: https://github.com/systemd/systemd/issues/8806

Regards,
Chris Lesiak

>
> Regards,
> Yann E. MORIN.
>
>>> Regards,
>>> Yann E. MORIN.
>>>
>>>> Signed-off-by: Chris Lesiak <chris.lesiak@licor.com>
>>>> ---
>>>>   Makefile | 2 ++
>>>>   1 file changed, 2 insertions(+)
>>>>
>>>> diff --git a/Makefile b/Makefile
>>>> index 957ba6634a..12a53840c0 100644
>>>> --- a/Makefile
>>>> +++ b/Makefile
>>>> @@ -761,6 +761,8 @@ endif
>>>>                $(call MESSAGE,"Executing post-build script $(s)"); \
>>>>                $(EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
>>>>
>>>> +     touch $(TARGET_DIR)/usr
>>>> +
>>>>   .PHONY: target-post-image
>>>>   target-post-image: $(TARGETS_ROOTFS) target-finalize
>>>>        @$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
>>>> --
>>>> 2.14.3
>>>>
>>>> _______________________________________________
>>>> buildroot mailing list
>>>> buildroot@busybox.net
>>>> http://lists.busybox.net/mailman/listinfo/buildroot
>>> --
>>> .-----------------.--------------------.------------------.--------------------.
>>> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
>>> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
>>> | +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
>>> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
>>> '------------------------------^-------^------------------^--------------------'
>>>
>> --
>> Chris Lesiak
>> Principal Design Engineer, Software
>> LI-COR Biosciences
>> chris.lesiak@licor.com
>>
>> Any opinions expressed are those of the author and
>> do not necessarily represent those of his employer.
>>
> --
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'
>
Yann E. MORIN April 28, 2018, 8 p.m. | #5
Chris, All,

On 2018-04-24 13:56 -0500, Chris Lesiak spake thusly:
> >     date    what
> >     0       build version 1
> >     1       version is flahes onto a device, boxed and shipped
> >     2       build version 2
> >     3       the device with version 1 is received, unboxed and booted
> >     4       that device downloads version 2 and updates to it
> >
> >As you can see, the /etc/.updated and/or /var/.updated will dated '3'
> >but the mtime of /usr version 2 will be 2 (because touched at build
> >time).
> 
> Not quite.
> 
> If systemd-update-done if working correctly, then on date 3, /etc/.updated
> and /var/.updated will get created with mtime 0, not mtime 3.  Then on date
> 4, /etc/.updated and /var/.updated will get updated (because 0 is less than
> 2) to mtime 2.
> 
> When /etc/.updated and /var/.updated are originally created and then again
> when they are updated, their mtime will match the mtime of /usr, not the
> current time when systemd-update-done runs.

OK, makes sense. Thanks for the clraification.

So, this touch is eventually mostly usefull for a read-only rootfs, in
fact (for a read-only rootfs, the touch could be done by the update
mechanis).

Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

Patch

diff --git a/Makefile b/Makefile
index 957ba6634a..12a53840c0 100644
--- a/Makefile
+++ b/Makefile
@@ -761,6 +761,8 @@  endif
 		$(call MESSAGE,"Executing post-build script $(s)"); \
 		$(EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
 
+	touch $(TARGET_DIR)/usr
+
 .PHONY: target-post-image
 target-post-image: $(TARGETS_ROOTFS) target-finalize
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \