[v4,13/18] python2: generate reproducible .pyc

.pyc files contain modification time of .py source. In order to make
build reproducible, we fix modification time of all .py before to
compile .pyc files.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: Jérôme Pouiller <jezz@sysmic.org>
---

Notes:
    v3:
      - Force compilation instead of removing .pyc

 package/python/python.mk | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

Hello,

On Wed, 23 Nov 2016 13:58:52 +0100, Jérôme Pouiller wrote:
> .pyc files contain modification time of .py source. In order to make
> build reproducible, we fix modification time of all .py before to
> compile .pyc files.

"before to compile" -> "before compiling".

> diff --git a/package/python/python.mk b/package/python/python.mk
> index cc65376..c17b267 100644
> --- a/package/python/python.mk
> +++ b/package/python/python.mk
> @@ -226,10 +226,18 @@ PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/
>  $(eval $(autotools-package))
>  $(eval $(host-autotools-package))
>  
> +ifeq ($(BR2_REPRODUCIBLE),y)
> +define PYTHON_FIX_TIME
> +find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \
> +		xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
> +endef
> +PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_FIX_TIME
> +endif
> +
>  define PYTHON_CREATE_PYC_FILES
>  	PYTHONPATH="$(PYTHON_PATH)" \

It would make more sense to just do:

	$(PYTHON_FIX_TIME)

here, rather than registering it as a PYTHON_TARGET_FINALIZE_HOOKS.
Also, maybe it should be named PYTHON_SET_PY_FILES_TIME.

>  	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
> -		support/scripts/pycompile.py \
> +		support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \

Why do we need to force?

Thomas

On 23-11-16 23:05, Thomas Petazzoni wrote:
> Hello,
> 
> On Wed, 23 Nov 2016 13:58:52 +0100, Jérôme Pouiller wrote:
>> .pyc files contain modification time of .py source. In order to make
>> build reproducible, we fix modification time of all .py before to
>> compile .pyc files.
> 
> "before to compile" -> "before compiling".
> 
>> diff --git a/package/python/python.mk b/package/python/python.mk
>> index cc65376..c17b267 100644
>> --- a/package/python/python.mk
>> +++ b/package/python/python.mk
>> @@ -226,10 +226,18 @@ PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/
>>  $(eval $(autotools-package))
>>  $(eval $(host-autotools-package))
>>  
>> +ifeq ($(BR2_REPRODUCIBLE),y)
>> +define PYTHON_FIX_TIME
>> +find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \
>> +		xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
>> +endef
>> +PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_FIX_TIME
>> +endif
>> +
>>  define PYTHON_CREATE_PYC_FILES
>>  	PYTHONPATH="$(PYTHON_PATH)" \
> 
> It would make more sense to just do:
> 
> 	$(PYTHON_FIX_TIME)
> 
> here, rather than registering it as a PYTHON_TARGET_FINALIZE_HOOKS.

 No. It iterates over all python files in $(TARGET_DIR), not just the ones
installed by this package, so it really should be a TARGET_FINALIZE_HOOK.


 Regards,
 Arnout



> Also, maybe it should be named PYTHON_SET_PY_FILES_TIME.
> 
>>  	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
>> -		support/scripts/pycompile.py \
>> +		support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
> 
> Why do we need to force?
> 
> Thomas
>

Hello Thomas,

On Thursday 24 November 2016 20:06:55 Arnout Vandecappelle wrote:
[...]
> >>  	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
> >> -		support/scripts/pycompile.py \
> >> +		support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
> > 
> > Why do we need to force?
Pycompile rely on .py modification times to know if a file need to be
recompiled. Since reproducible builds tweak modification time, it is
safer to force recompilation of all source files.

(I will add this explanation to commit log)