[00/11] Overwritten file detection, improvements to file listing logic

Hello,

The initial starting point for this series was that I wanted to detect
when a package overwrites files installed by another package, which is
something that is not compatible with per-package directory support. I
indeed have a complex build configuration which I'm converting to use
TLP, and this build configuration does lots of overwrites, which I
want to detect and fix.

While doing this overwrite detection, I realized that our existing
logic to detect which files are installed by each package is not
reliable: if a target package installs something in $(HOST_DIR), those
files installed in $(HOST_DIR) are not tracked. Getting this list of
files correct is also needed to properly support the
-reinstall,-rebuild,-reconfigure targets in a per-package
configuration (which this series does not address yet).

So this series goes on like this (I'm only highlighting the main
commits, not all preparation commits and details)

 - We create a new "install" step in PATCH 3. It is executed after all
   installation steps have executed. As of PATCH 3, it does nothing
   except creating a stamp file, but it will allow us to do some logic
   once a package is finished installing. Indeed, so far we had no
   place where we could run some logic after all installation steps of
   a package have completed.

 - In PATCH 5, we rework the pkg_size logic that detects files
   installed by each package to rely on the new "install"
   step. Instead of capturing the list of files "before" at the
   beginning of install-target, install-staging, install-images,
   install-host, and the list of files "after" at the end of these
   steps, we capture the list of files "before" at the beginning of
   the configure step, and the list of files "after" during the new
   "install" step.

 - In PATCH 8, we introduce the logic to detect files overwritten by
   other packages, using a simple md5sum hash.

 - In PATCH 9-11, we add test cases for the "check binary
   architecture" functionality and the "overwritten files detection"
   functionality.

This has received some basic testing so far, I'm definitely interested
in getting feedback from people, simply by testing if it works for
them.

All these changes are also available on the Git branch at:

  https://github.com/tpetazzoni/buildroot/tree/detect-overwrite

Best regards,

Thomas Petazzoni

Thomas Petazzoni (11):
  package/pkg-generic.mk: use $(PKG)_NAME in step_pkg_size_after
  package/pkg-generic.mk: drop useless $(1) argument in
    step_pkg_size_{before,after}
  package/pkg-generic.mk: introduce final 'install' step
  package/pkg-generic.mk: create directories upfront in the configure
    step
  package/pkg-generic.mk: rework pkg_size logic with the "installed"
    step
  package/pkg-generic.mk: exclude the staging sub-directory
  package/pkg-generic.mk: move pkg_size_{before,after} and
    check_bin_arch functions
  package/pkg-generic.mk: detect files overwritten in TARGET_DIR and
    HOST_DIR
  support/testing/infra: add log_file_path() function
  support/testing/tests: add test for check_bin_arch
  support/testing/tests: add test for file overwrite detection

 .gitlab-ci.yml                                |   3 +
 package/pkg-generic.mk                        | 146 ++++++++++--------
 package/pkg-utils.mk                          |   1 -
 support/testing/infra/__init__.py             |  13 +-
 .../br2-external/detect-bad-arch/Config.in    |   1 +
 .../detect-bad-arch/external.desc             |   1 +
 .../br2-external/detect-bad-arch/external.mk  |   1 +
 .../package/detect-bad-arch/Config.in         |   4 +
 .../detect-bad-arch/detect-bad-arch.mk        |  15 ++
 .../br2-external/detect-overwrite/Config.in   |   1 +
 .../detect-overwrite/external.desc            |   1 +
 .../br2-external/detect-overwrite/external.mk |   1 +
 .../package/detect-overwrite/Config.in        |   5 +
 .../detect-overwrite/detect-overwrite.mk      |  19 +++
 support/testing/tests/core/test_bad_arch.py   |  19 +++
 .../testing/tests/core/test_file_overwrite.py |  47 ++++++
 16 files changed, 213 insertions(+), 65 deletions(-)
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.desc
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.desc
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk
 create mode 100644 support/testing/tests/core/test_bad_arch.py
 create mode 100644 support/testing/tests/core/test_file_overwrite.py

Thomas, All,

On 2020-04-30 11:52 +0200, Thomas Petazzoni spake thusly:
[--SNIP--]
> Thomas Petazzoni (11):
>   package/pkg-generic.mk: use $(PKG)_NAME in step_pkg_size_after
>   package/pkg-generic.mk: drop useless $(1) argument in
>     step_pkg_size_{before,after}
>   package/pkg-generic.mk: introduce final 'install' step
>   package/pkg-generic.mk: create directories upfront in the configure
>     step
>   package/pkg-generic.mk: rework pkg_size logic with the "installed"
>     step
>   package/pkg-generic.mk: exclude the staging sub-directory
>   package/pkg-generic.mk: move pkg_size_{before,after} and
>     check_bin_arch functions

Patches 1..7 applied to master, thanks.

I take back my comments on patch 5, you were right. As a consequence, I
also take back my comments on patch 7.

I take back my suggestion on patch 6: it does not work perfectly, and
the output changes slightly, so scritps that interpert the content would
have to be changed

>   package/pkg-generic.mk: detect files overwritten in TARGET_DIR and
>     HOST_DIR

For that one, I still have open questions about it, so I did not apply.

Thanks! :-)

Regards,
Yann E. MORIN.

>   support/testing/infra: add log_file_path() function
>   support/testing/tests: add test for check_bin_arch
>   support/testing/tests: add test for file overwrite detection
> 
>  .gitlab-ci.yml                                |   3 +
>  package/pkg-generic.mk                        | 146 ++++++++++--------
>  package/pkg-utils.mk                          |   1 -
>  support/testing/infra/__init__.py             |  13 +-
>  .../br2-external/detect-bad-arch/Config.in    |   1 +
>  .../detect-bad-arch/external.desc             |   1 +
>  .../br2-external/detect-bad-arch/external.mk  |   1 +
>  .../package/detect-bad-arch/Config.in         |   4 +
>  .../detect-bad-arch/detect-bad-arch.mk        |  15 ++
>  .../br2-external/detect-overwrite/Config.in   |   1 +
>  .../detect-overwrite/external.desc            |   1 +
>  .../br2-external/detect-overwrite/external.mk |   1 +
>  .../package/detect-overwrite/Config.in        |   5 +
>  .../detect-overwrite/detect-overwrite.mk      |  19 +++
>  support/testing/tests/core/test_bad_arch.py   |  19 +++
>  .../testing/tests/core/test_file_overwrite.py |  47 ++++++
>  16 files changed, 213 insertions(+), 65 deletions(-)
>  create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/Config.in
>  create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.desc
>  create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.mk
>  create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in
>  create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk
>  create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/Config.in
>  create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.desc
>  create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.mk
>  create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in
>  create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk
>  create mode 100644 support/testing/tests/core/test_bad_arch.py
>  create mode 100644 support/testing/tests/core/test_file_overwrite.py
> 
> -- 
> 2.25.4
>