[ovs-dev,v2,4/4] appveyor: Build with OpenSSL 3.0.

OpenSSL 1.0.2u is long deprecated and not available for download.
So, our CI never actually downloads it and uses whatever is in the
OpenSSL-Win64 folder provided by AppVeyor.  Luckily, it happens to
be OpenSSL 1.0.2u today.

The oldest supported version of OpenSSL upstream today is 3.0.
And it is an LTS version.  3.1 and 3.2 are not LTS.

Use OpenSSL 3.0 for testing instead.

This commit does a few things to achieve that:

1. Removes the folder provided by AppVeyor.  This way we will fail
   the build if something goes wrong instead of silently using
   OpenSSL version provided by AppVeyor.

2. Obtains the JSON description of available releases and downloads
   the latest minor version of OpenSSL 3.0 64-bit.  With this approach
   we should not need to update the download link that frequently.
   New minor releases will be picked up automatically.  They should
   not have any breaking changes, so should be fine to use in CI.
   OpenSSL 3.0 is supported until at least Sep 2026.

   The JSON file is an official file referenced on the:
        https://slproweb.com/products/Win32OpenSSL.html
   So, it should be safe to use.

3. Executes the downloaded installer with 'Start-Process -Wait' to
   properly wait for installation to finish instead of just sleeping
   for 30 seconds.

4. Caches the downloaded installer, so we're not downloading 300 MB
   on each CI run as that is not nice to do.  We know the hash of the
   latest version, so we will re-download only when the binary changes,
   i.e. on a new minor release.

   For the cache to work we need to introduce the 'install' phase,
   because caches are populated after 'init', but before 'install'.
   Alternatively, we could have just renamed 'init' to 'install',
   but I think it's a little nicer to have separate phases, and we
   can also move 'windows-prepare.sh' to the install phase.

   Cache is also invalidated whenever appveyor.yml changes.

Acked-by: Simon Horman <horms@ovn.org>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
---
 appveyor.yml | 52 ++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 42 insertions(+), 10 deletions(-)

Lgtm.  Thank you so much Ilya

Acked-by: Alin-Gabriel Serdean <aserdean@ovn.org>

> 
> On 1 Mar 2024, at 22:10, Ilya Maximets <i.maximets@ovn.org> wrote:
> 
> OpenSSL 1.0.2u is long deprecated and not available for download.
> So, our CI never actually downloads it and uses whatever is in the
> OpenSSL-Win64 folder provided by AppVeyor.  Luckily, it happens to
> be OpenSSL 1.0.2u today.
> 
> The oldest supported version of OpenSSL upstream today is 3.0.
> And it is an LTS version.  3.1 and 3.2 are not LTS.
> 
> Use OpenSSL 3.0 for testing instead.
> 
> This commit does a few things to achieve that:
> 
> 1. Removes the folder provided by AppVeyor.  This way we will fail
>   the build if something goes wrong instead of silently using
>   OpenSSL version provided by AppVeyor.
> 
> 2. Obtains the JSON description of available releases and downloads
>   the latest minor version of OpenSSL 3.0 64-bit.  With this approach
>   we should not need to update the download link that frequently.
>   New minor releases will be picked up automatically.  They should
>   not have any breaking changes, so should be fine to use in CI.
>   OpenSSL 3.0 is supported until at least Sep 2026.
> 
>   The JSON file is an official file referenced on the:
>        https://slproweb.com/products/Win32OpenSSL.html
>   So, it should be safe to use.
> 
> 3. Executes the downloaded installer with 'Start-Process -Wait' to
>   properly wait for installation to finish instead of just sleeping
>   for 30 seconds.
> 
> 4. Caches the downloaded installer, so we're not downloading 300 MB
>   on each CI run as that is not nice to do.  We know the hash of the
>   latest version, so we will re-download only when the binary changes,
>   i.e. on a new minor release.
> 
>   For the cache to work we need to introduce the 'install' phase,
>   because caches are populated after 'init', but before 'install'.
>   Alternatively, we could have just renamed 'init' to 'install',
>   but I think it's a little nicer to have separate phases, and we
>   can also move 'windows-prepare.sh' to the install phase.
> 
>   Cache is also invalidated whenever appveyor.yml changes.
> 
> Acked-by: Simon Horman <horms@ovn.org>
> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
> ---
> appveyor.yml | 52 ++++++++++++++++++++++++++++++++++++++++++----------
> 1 file changed, 42 insertions(+), 10 deletions(-)
> 
> diff --git a/appveyor.yml b/appveyor.yml
> index 373f01a43..29cc44d6c 100644
> --- a/appveyor.yml
> +++ b/appveyor.yml
> @@ -8,28 +8,60 @@ configuration:
>   - Release
> clone_folder: C:\openvswitch_compile
> shallow_clone: true
> +
> init:
> - ps: $env:PATH ="C:\Python312-x64;"+$env:PATH
> - ps: New-Item -Type HardLink -Path  "C:\Python312-x64\python3.exe"
>                               -Value "C:\Python312-x64\python.exe"
> +
> +cache:
> +- C:\ovs-build-downloads -> appveyor.yml
> +
> +install:
> - ps: |
> -    mkdir C:\ovs-build-downloads
> +    Remove-Item -Recurse -Force -Path C:/OpenSSL-Win64
> +    New-Item -ItemType Directory -Force -Path C:\ovs-build-downloads
> +
> +    # Find and download the latest stable OpenSSl 3.0.
> +    $URL = "https://raw.githubusercontent.com/slproweb/opensslhashes/master/win32_openssl_hashes.json"
> +    $webData = (Invoke-WebRequest -Uri $URL).content | ConvertFrom-Json
> +    $source = ($webData.files.PSObject.Properties | Where-Object {
> +        $_.Value.basever   -match "3.0.*" -and
> +        $_.Value.bits      -eq    "64"    -and
> +        $_.Value.arch      -eq    "INTEL" -and
> +        $_.Value.installer -eq    "exe"   -and
> +        -not $_.Value.light
> +    } | Select-Object Value).PSObject.Properties.Value
> +
> +    Write-Host "Latest OpenSSL 3.0:" ($source | Format-List | Out-String)
> +
> +    $destination = "C:\ovs-build-downloads\Win64OpenSSL.exe"
> +    if (Test-Path $destination) {
> +        $fileHash = (Get-FileHash $destination -Algorithm SHA256).Hash.ToLower()
> +        if ($fileHash -ne $source.sha256) {
> +            Write-Host "Cache miss:" $fileHash "!=" $source.sha256
> +            Remove-Item -Path $destination
> +        }
> +    }
> 
> -    $source = "https://slproweb.com/download/Win64OpenSSL-1_0_2u.exe"
> -    $destination = "C:\ovs-build-downloads\Win64OpenSSL-1_0_2u.exe"
> -    Invoke-WebRequest $source -OutFile $destination
> +    if (Test-Path $destination) {
> +        Write-Host "Using cached:" $destination
> +    } else {
> +        Write-Host "Downloading:" $source.url
> +        Invoke-WebRequest $source.url -OutFile $destination
> +    }
> +
> +    Write-Host "Installing:" $destination
> +    Start-Process -FilePath $destination `
> +        -ArgumentList "/silent /verysilent /sp- /suppressmsgboxes" -Wait
> 
> -    cd C:\ovs-build-downloads
> -    .\Win64OpenSSL-1_0_2u.exe /silent /verysilent /sp- /suppressmsgboxes
> -    Start-Sleep -s 30
> -    cd C:\openvswitch_compile
> - ps: git clone -q https://git.code.sf.net/p/pthreads4w/code c:\pthreads4w-code
> - ps: python3 -m pip install pypiwin32 --disable-pip-version-check
> -
> -build_script:
> - '"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat"'
> - ps: C:\msys64\msys2_shell.cmd -here -defterm -no-start -use-full-path -c
>         ".ci/windows-prepare.sh 2>&1"
> +
> +build_script:
> - ps: C:\msys64\msys2_shell.cmd -here -defterm -no-start -use-full-path -c
>         ".ci/windows-build.sh $env:CONFIGURATION 2>&1"
> - ps: cp C:\PTHREADS-BUILT\bin\pthreadVC3.dll C:\openvswitch\usr\bin
> --
> 2.43.0
>

On 3/1/24 22:29, Alin Serdean wrote:
> Lgtm.  Thank you so much Ilya
> 
> Acked-by: Alin-Gabriel Serdean <aserdean@ovn.org>

Thanks, Alin and Simon!

I applied the set now.

Also backported patches 2 and 3 to branch-3.3, since they are bug fixes
and 3.3 is planned to be our next LTS.  We may backport further, but
I'm a little hesitant to do so since we don't have CI for stable branches
and no-one complained so far.  Let me know if you think we should backport
further, I can do that.

Best regards, Ilya Maximets.

> 
>>
>> On 1 Mar 2024, at 22:10, Ilya Maximets <i.maximets@ovn.org> wrote:
>>
>> OpenSSL 1.0.2u is long deprecated and not available for download.
>> So, our CI never actually downloads it and uses whatever is in the
>> OpenSSL-Win64 folder provided by AppVeyor.  Luckily, it happens to
>> be OpenSSL 1.0.2u today.
>>
>> The oldest supported version of OpenSSL upstream today is 3.0.
>> And it is an LTS version.  3.1 and 3.2 are not LTS.
>>
>> Use OpenSSL 3.0 for testing instead.
>>
>> This commit does a few things to achieve that:
>>
>> 1. Removes the folder provided by AppVeyor.  This way we will fail
>>   the build if something goes wrong instead of silently using
>>   OpenSSL version provided by AppVeyor.
>>
>> 2. Obtains the JSON description of available releases and downloads
>>   the latest minor version of OpenSSL 3.0 64-bit.  With this approach
>>   we should not need to update the download link that frequently.
>>   New minor releases will be picked up automatically.  They should
>>   not have any breaking changes, so should be fine to use in CI.
>>   OpenSSL 3.0 is supported until at least Sep 2026.
>>
>>   The JSON file is an official file referenced on the:
>>        https://slproweb.com/products/Win32OpenSSL.html
>>   So, it should be safe to use.
>>
>> 3. Executes the downloaded installer with 'Start-Process -Wait' to
>>   properly wait for installation to finish instead of just sleeping
>>   for 30 seconds.
>>
>> 4. Caches the downloaded installer, so we're not downloading 300 MB
>>   on each CI run as that is not nice to do.  We know the hash of the
>>   latest version, so we will re-download only when the binary changes,
>>   i.e. on a new minor release.
>>
>>   For the cache to work we need to introduce the 'install' phase,
>>   because caches are populated after 'init', but before 'install'.
>>   Alternatively, we could have just renamed 'init' to 'install',
>>   but I think it's a little nicer to have separate phases, and we
>>   can also move 'windows-prepare.sh' to the install phase.
>>
>>   Cache is also invalidated whenever appveyor.yml changes.
>>
>> Acked-by: Simon Horman <horms@ovn.org>
>> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
>> ---
>> appveyor.yml | 52 ++++++++++++++++++++++++++++++++++++++++++----------
>> 1 file changed, 42 insertions(+), 10 deletions(-)
>>
>> diff --git a/appveyor.yml b/appveyor.yml
>> index 373f01a43..29cc44d6c 100644
>> --- a/appveyor.yml
>> +++ b/appveyor.yml
>> @@ -8,28 +8,60 @@ configuration:
>>   - Release
>> clone_folder: C:\openvswitch_compile
>> shallow_clone: true
>> +
>> init:
>> - ps: $env:PATH ="C:\Python312-x64;"+$env:PATH
>> - ps: New-Item -Type HardLink -Path  "C:\Python312-x64\python3.exe"
>>                               -Value "C:\Python312-x64\python.exe"
>> +
>> +cache:
>> +- C:\ovs-build-downloads -> appveyor.yml
>> +
>> +install:
>> - ps: |
>> -    mkdir C:\ovs-build-downloads
>> +    Remove-Item -Recurse -Force -Path C:/OpenSSL-Win64
>> +    New-Item -ItemType Directory -Force -Path C:\ovs-build-downloads
>> +
>> +    # Find and download the latest stable OpenSSl 3.0.
>> +    $URL = "https://raw.githubusercontent.com/slproweb/opensslhashes/master/win32_openssl_hashes.json"
>> +    $webData = (Invoke-WebRequest -Uri $URL).content | ConvertFrom-Json
>> +    $source = ($webData.files.PSObject.Properties | Where-Object {
>> +        $_.Value.basever   -match "3.0.*" -and
>> +        $_.Value.bits      -eq    "64"    -and
>> +        $_.Value.arch      -eq    "INTEL" -and
>> +        $_.Value.installer -eq    "exe"   -and
>> +        -not $_.Value.light
>> +    } | Select-Object Value).PSObject.Properties.Value
>> +
>> +    Write-Host "Latest OpenSSL 3.0:" ($source | Format-List | Out-String)
>> +
>> +    $destination = "C:\ovs-build-downloads\Win64OpenSSL.exe"
>> +    if (Test-Path $destination) {
>> +        $fileHash = (Get-FileHash $destination -Algorithm SHA256).Hash.ToLower()
>> +        if ($fileHash -ne $source.sha256) {
>> +            Write-Host "Cache miss:" $fileHash "!=" $source.sha256
>> +            Remove-Item -Path $destination
>> +        }
>> +    }
>>
>> -    $source = "https://slproweb.com/download/Win64OpenSSL-1_0_2u.exe"
>> -    $destination = "C:\ovs-build-downloads\Win64OpenSSL-1_0_2u.exe"
>> -    Invoke-WebRequest $source -OutFile $destination
>> +    if (Test-Path $destination) {
>> +        Write-Host "Using cached:" $destination
>> +    } else {
>> +        Write-Host "Downloading:" $source.url
>> +        Invoke-WebRequest $source.url -OutFile $destination
>> +    }
>> +
>> +    Write-Host "Installing:" $destination
>> +    Start-Process -FilePath $destination `
>> +        -ArgumentList "/silent /verysilent /sp- /suppressmsgboxes" -Wait
>>
>> -    cd C:\ovs-build-downloads
>> -    .\Win64OpenSSL-1_0_2u.exe /silent /verysilent /sp- /suppressmsgboxes
>> -    Start-Sleep -s 30
>> -    cd C:\openvswitch_compile
>> - ps: git clone -q https://git.code.sf.net/p/pthreads4w/code c:\pthreads4w-code
>> - ps: python3 -m pip install pypiwin32 --disable-pip-version-check
>> -
>> -build_script:
>> - '"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat"'
>> - ps: C:\msys64\msys2_shell.cmd -here -defterm -no-start -use-full-path -c
>>         ".ci/windows-prepare.sh 2>&1"
>> +
>> +build_script:
>> - ps: C:\msys64\msys2_shell.cmd -here -defterm -no-start -use-full-path -c
>>         ".ci/windows-build.sh $env:CONFIGURATION 2>&1"
>> - ps: cp C:\PTHREADS-BUILT\bin\pthreadVC3.dll C:\openvswitch\usr\bin
>> --
>> 2.43.0
>>

That’s understandable, I think we should be fine for now and if someone asks for a backport we can address it then.

Thank you again for all the work you put into this!

Alin

> 
> On 4 Mar 2024, at 23:44, Ilya Maximets <i.maximets@ovn.org> wrote:
> 
> On 3/1/24 22:29, Alin Serdean wrote:
>> Lgtm.  Thank you so much Ilya
>> 
>> Acked-by: Alin-Gabriel Serdean <aserdean@ovn.org>
> 
> Thanks, Alin and Simon!
> 
> I applied the set now.
> 
> Also backported patches 2 and 3 to branch-3.3, since they are bug fixes
> and 3.3 is planned to be our next LTS.  We may backport further, but
> I'm a little hesitant to do so since we don't have CI for stable branches
> and no-one complained so far.  Let me know if you think we should backport
> further, I can do that.
> 
> Best regards, Ilya Maximets.
> 
>> 
>>> 
>>>> On 1 Mar 2024, at 22:10, Ilya Maximets <i.maximets@ovn.org> wrote:
>>> 
>>> OpenSSL 1.0.2u is long deprecated and not available for download.
>>> So, our CI never actually downloads it and uses whatever is in the
>>> OpenSSL-Win64 folder provided by AppVeyor.  Luckily, it happens to
>>> be OpenSSL 1.0.2u today.
>>> 
>>> The oldest supported version of OpenSSL upstream today is 3.0.
>>> And it is an LTS version.  3.1 and 3.2 are not LTS.
>>> 
>>> Use OpenSSL 3.0 for testing instead.
>>> 
>>> This commit does a few things to achieve that:
>>> 
>>> 1. Removes the folder provided by AppVeyor.  This way we will fail
>>>  the build if something goes wrong instead of silently using
>>>  OpenSSL version provided by AppVeyor.
>>> 
>>> 2. Obtains the JSON description of available releases and downloads
>>>  the latest minor version of OpenSSL 3.0 64-bit.  With this approach
>>>  we should not need to update the download link that frequently.
>>>  New minor releases will be picked up automatically.  They should
>>>  not have any breaking changes, so should be fine to use in CI.
>>>  OpenSSL 3.0 is supported until at least Sep 2026.
>>> 
>>>  The JSON file is an official file referenced on the:
>>>       https://slproweb.com/products/Win32OpenSSL.html
>>>  So, it should be safe to use.
>>> 
>>> 3. Executes the downloaded installer with 'Start-Process -Wait' to
>>>  properly wait for installation to finish instead of just sleeping
>>>  for 30 seconds.
>>> 
>>> 4. Caches the downloaded installer, so we're not downloading 300 MB
>>>  on each CI run as that is not nice to do.  We know the hash of the
>>>  latest version, so we will re-download only when the binary changes,
>>>  i.e. on a new minor release.
>>> 
>>>  For the cache to work we need to introduce the 'install' phase,
>>>  because caches are populated after 'init', but before 'install'.
>>>  Alternatively, we could have just renamed 'init' to 'install',
>>>  but I think it's a little nicer to have separate phases, and we
>>>  can also move 'windows-prepare.sh' to the install phase.
>>> 
>>>  Cache is also invalidated whenever appveyor.yml changes.
>>> 
>>> Acked-by: Simon Horman <horms@ovn.org>
>>> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
>>> ---
>>> appveyor.yml | 52 ++++++++++++++++++++++++++++++++++++++++++----------
>>> 1 file changed, 42 insertions(+), 10 deletions(-)
>>> 
>>> diff --git a/appveyor.yml b/appveyor.yml
>>> index 373f01a43..29cc44d6c 100644
>>> --- a/appveyor.yml
>>> +++ b/appveyor.yml
>>> @@ -8,28 +8,60 @@ configuration:
>>>  - Release
>>> clone_folder: C:\openvswitch_compile
>>> shallow_clone: true
>>> +
>>> init:
>>> - ps: $env:PATH ="C:\Python312-x64;"+$env:PATH
>>> - ps: New-Item -Type HardLink -Path  "C:\Python312-x64\python3.exe"
>>>                              -Value "C:\Python312-x64\python.exe"
>>> +
>>> +cache:
>>> +- C:\ovs-build-downloads -> appveyor.yml
>>> +
>>> +install:
>>> - ps: |
>>> -    mkdir C:\ovs-build-downloads
>>> +    Remove-Item -Recurse -Force -Path C:/OpenSSL-Win64
>>> +    New-Item -ItemType Directory -Force -Path C:\ovs-build-downloads
>>> +
>>> +    # Find and download the latest stable OpenSSl 3.0.
>>> +    $URL = "https://raw.githubusercontent.com/slproweb/opensslhashes/master/win32_openssl_hashes.json"
>>> +    $webData = (Invoke-WebRequest -Uri $URL).content | ConvertFrom-Json
>>> +    $source = ($webData.files.PSObject.Properties | Where-Object {
>>> +        $_.Value.basever   -match "3.0.*" -and
>>> +        $_.Value.bits      -eq    "64"    -and
>>> +        $_.Value.arch      -eq    "INTEL" -and
>>> +        $_.Value.installer -eq    "exe"   -and
>>> +        -not $_.Value.light
>>> +    } | Select-Object Value).PSObject.Properties.Value
>>> +
>>> +    Write-Host "Latest OpenSSL 3.0:" ($source | Format-List | Out-String)
>>> +
>>> +    $destination = "C:\ovs-build-downloads\Win64OpenSSL.exe"
>>> +    if (Test-Path $destination) {
>>> +        $fileHash = (Get-FileHash $destination -Algorithm SHA256).Hash.ToLower()
>>> +        if ($fileHash -ne $source.sha256) {
>>> +            Write-Host "Cache miss:" $fileHash "!=" $source.sha256
>>> +            Remove-Item -Path $destination
>>> +        }
>>> +    }
>>> 
>>> -    $source = "https://slproweb.com/download/Win64OpenSSL-1_0_2u.exe"
>>> -    $destination = "C:\ovs-build-downloads\Win64OpenSSL-1_0_2u.exe"
>>> -    Invoke-WebRequest $source -OutFile $destination
>>> +    if (Test-Path $destination) {
>>> +        Write-Host "Using cached:" $destination
>>> +    } else {
>>> +        Write-Host "Downloading:" $source.url
>>> +        Invoke-WebRequest $source.url -OutFile $destination
>>> +    }
>>> +
>>> +    Write-Host "Installing:" $destination
>>> +    Start-Process -FilePath $destination `
>>> +        -ArgumentList "/silent /verysilent /sp- /suppressmsgboxes" -Wait
>>> 
>>> -    cd C:\ovs-build-downloads
>>> -    .\Win64OpenSSL-1_0_2u.exe /silent /verysilent /sp- /suppressmsgboxes
>>> -    Start-Sleep -s 30
>>> -    cd C:\openvswitch_compile
>>> - ps: git clone -q https://git.code.sf.net/p/pthreads4w/code c:\pthreads4w-code
>>> - ps: python3 -m pip install pypiwin32 --disable-pip-version-check
>>> -
>>> -build_script:
>>> - '"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat"'
>>> - ps: C:\msys64\msys2_shell.cmd -here -defterm -no-start -use-full-path -c
>>>        ".ci/windows-prepare.sh 2>&1"
>>> +
>>> +build_script:
>>> - ps: C:\msys64\msys2_shell.cmd -here -defterm -no-start -use-full-path -c
>>>        ".ci/windows-build.sh $env:CONFIGURATION 2>&1"
>>> - ps: cp C:\PTHREADS-BUILT\bin\pthreadVC3.dll C:\openvswitch\usr\bin
>>> --
>>> 2.43.0
>>> 
>