| Message ID | 20230109163017.1260911-1-yann.morin.1998@free.fr |
|---|---|
| State | Accepted |
| Headers | show |
| Series | support/download: fix cargo vendoring | expand |
All, On 2023-01-09 17:30 +0100, Yann E. MORIN spake thusly: > Commit de4cf253752d (package/{rust, rust-bin}: bump to version 1.66.0) > forgot, despite the big comment above the version strings, to confirm > that the vendoring was still working. > > Previously, we were adding the vendoring equivalence manually, but in > commit 04154a651729 (support/download/cargo-post-process: cargo output > for vendor config), we switched to using the output of "vargo vendor" > (on stdout) to support caxses were the vendoring equivalence would be > more complex (e.g. when using crates not hosted on crates.io). > > With rust until and including 1.65.0, "cargo vendor" would output (for > crates.io crates) the same output as our manual fixups, except it was > preceded by an empty line. So, to avoid recompting all our hashes, we > added a tweak to strip away the leading empty line in 04154a651729. > > But rust 1.66.0 includes [0] which changes the output (on stdout) of > "cargo vendor", where the first empty line is no longer emitted. > > This means that our tweak for rust 1.65.0 now strips out an important > part of the cargo vendor output, which renders the archives invalid, and > thus generates different archives, which fail to validate against our > hashes. > > Fix this by doing what the comment in the post-process helper states, > and just keep the whole output of "cargo vendor", byt just removing the > "tail --lines=+2". Since that comment is no longer meaningful, we drop > it too. > > Now, all our 6 cargo-based packages can be vendored again, without > changing our hashes, but most importantly, with valid archives. Of course, I forgot to account for the 5 python packages that have rust code; they too are fixed with this change. Regards, Yann E. MORIN. > Still, we keep the comment above the versions strings, in the hope that > a future bumper will notice and be more careful at validating the > vendoring. > > [0] https://github.com/rust-lang/cargo/pull/11273 > > Fixes: > http://autobuild.buildroot.org/results/bea/beac7674bbc9fd2f8777b5861f65afee9c485753/ (bat) > http://autobuild.buildroot.org/results/d1e/d1ec1ebbde115628a4b8b9099544347242a97c1c/ (dust) > http://autobuild.buildroot.org/results/f96/f968be895be9ca98b314fdd688ef8d3bdf4e5dfb/ (hyerfine) > http://autobuild.buildroot.org/results/a0c/a0cdb6cc9493f5248d98f98b13da854e12adc2be/ (ripgrep) > ... and so many others... > > Reported-by: Romain Naour <romain.naour@gmail.com> > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> > Cc: James Hilliard <james.hilliard1@gmail.com> > Cc: Simon Richter <simon.richter@ptwdosimetry.com> > --- > support/download/cargo-post-process | 16 +--------------- > 1 file changed, 1 insertion(+), 15 deletions(-) > > diff --git a/support/download/cargo-post-process b/support/download/cargo-post-process > index 186e9eb69b..21a6be8dbe 100755 > --- a/support/download/cargo-post-process > +++ b/support/download/cargo-post-process > @@ -25,25 +25,11 @@ post_process_unpack "${base_name}" "${output}" > pushd "${base_name}" > /dev/null > > # Create the local .cargo/config with vendor info > -# > -# The first line of the output to stdout is empty. > -# So skip it to have the file start with the vendoring > -# configuration (`tail --lines=+2`). > -# > -# NOTE: > -# There is a patch for cargo to remove the first empty line: > -# See: https://github.com/rust-lang/cargo/pull/11273 > -# > -# The patch already landed in +nightly and will end up > -# in +stable soon. > -# > -# -> When updating rust/cargo, the call to `tail` must be removed. > -# > mkdir -p .cargo/ > cargo vendor \ > --manifest-path ${BR_CARGO_MANIFEST_PATH-Cargo.toml} \ > --locked VENDOR \ > - | tail --lines=+2 | tee .cargo/config > + | tee .cargo/config > > popd > /dev/null > > -- > 2.25.1 >
On Mon, Jan 9, 2023 at 9:30 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Commit de4cf253752d (package/{rust, rust-bin}: bump to version 1.66.0) > forgot, despite the big comment above the version strings, to confirm > that the vendoring was still working. > > Previously, we were adding the vendoring equivalence manually, but in > commit 04154a651729 (support/download/cargo-post-process: cargo output > for vendor config), we switched to using the output of "vargo vendor" > (on stdout) to support caxses were the vendoring equivalence would be > more complex (e.g. when using crates not hosted on crates.io). > > With rust until and including 1.65.0, "cargo vendor" would output (for > crates.io crates) the same output as our manual fixups, except it was > preceded by an empty line. So, to avoid recompting all our hashes, we > added a tweak to strip away the leading empty line in 04154a651729. > > But rust 1.66.0 includes [0] which changes the output (on stdout) of > "cargo vendor", where the first empty line is no longer emitted. > > This means that our tweak for rust 1.65.0 now strips out an important > part of the cargo vendor output, which renders the archives invalid, and > thus generates different archives, which fail to validate against our > hashes. > > Fix this by doing what the comment in the post-process helper states, > and just keep the whole output of "cargo vendor", byt just removing the > "tail --lines=+2". Since that comment is no longer meaningful, we drop > it too. > > Now, all our 6 cargo-based packages can be vendored again, without > changing our hashes, but most importantly, with valid archives. > > Still, we keep the comment above the versions strings, in the hope that > a future bumper will notice and be more careful at validating the > vendoring. > > [0] https://github.com/rust-lang/cargo/pull/11273 > > Fixes: > http://autobuild.buildroot.org/results/bea/beac7674bbc9fd2f8777b5861f65afee9c485753/ (bat) > http://autobuild.buildroot.org/results/d1e/d1ec1ebbde115628a4b8b9099544347242a97c1c/ (dust) > http://autobuild.buildroot.org/results/f96/f968be895be9ca98b314fdd688ef8d3bdf4e5dfb/ (hyerfine) > http://autobuild.buildroot.org/results/a0c/a0cdb6cc9493f5248d98f98b13da854e12adc2be/ (ripgrep) > ... and so many others... > > Reported-by: Romain Naour <romain.naour@gmail.com> > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> > Cc: James Hilliard <james.hilliard1@gmail.com> Reviewed-by: James Hilliard <james.hilliard1@gmail.com> > Cc: Simon Richter <simon.richter@ptwdosimetry.com> > --- > support/download/cargo-post-process | 16 +--------------- > 1 file changed, 1 insertion(+), 15 deletions(-) > > diff --git a/support/download/cargo-post-process b/support/download/cargo-post-process > index 186e9eb69b..21a6be8dbe 100755 > --- a/support/download/cargo-post-process > +++ b/support/download/cargo-post-process > @@ -25,25 +25,11 @@ post_process_unpack "${base_name}" "${output}" > pushd "${base_name}" > /dev/null > > # Create the local .cargo/config with vendor info > -# > -# The first line of the output to stdout is empty. > -# So skip it to have the file start with the vendoring > -# configuration (`tail --lines=+2`). > -# > -# NOTE: > -# There is a patch for cargo to remove the first empty line: > -# See: https://github.com/rust-lang/cargo/pull/11273 > -# > -# The patch already landed in +nightly and will end up > -# in +stable soon. > -# > -# -> When updating rust/cargo, the call to `tail` must be removed. > -# > mkdir -p .cargo/ > cargo vendor \ > --manifest-path ${BR_CARGO_MANIFEST_PATH-Cargo.toml} \ > --locked VENDOR \ > - | tail --lines=+2 | tee .cargo/config > + | tee .cargo/config > > popd > /dev/null > > -- > 2.25.1 >
Hello Yann, All, Le 09/01/2023 à 17:30, Yann E. MORIN a écrit : > Commit de4cf253752d (package/{rust, rust-bin}: bump to version 1.66.0) > forgot, despite the big comment above the version strings, to confirm > that the vendoring was still working. > > Previously, we were adding the vendoring equivalence manually, but in > commit 04154a651729 (support/download/cargo-post-process: cargo output > for vendor config), we switched to using the output of "vargo vendor" vargo/cargo > (on stdout) to support caxses were the vendoring equivalence would be caxses/cases > more complex (e.g. when using crates not hosted on crates.io). > > With rust until and including 1.65.0, "cargo vendor" would output (for > crates.io crates) the same output as our manual fixups, except it was > preceded by an empty line. So, to avoid recompting all our hashes, we > added a tweak to strip away the leading empty line in 04154a651729. > > But rust 1.66.0 includes [0] which changes the output (on stdout) of > "cargo vendor", where the first empty line is no longer emitted. > > This means that our tweak for rust 1.65.0 now strips out an important > part of the cargo vendor output, which renders the archives invalid, and > thus generates different archives, which fail to validate against our > hashes. > > Fix this by doing what the comment in the post-process helper states, > and just keep the whole output of "cargo vendor", byt just removing the byt/by > "tail --lines=+2". Since that comment is no longer meaningful, we drop > it too. > > Now, all our 6 cargo-based packages can be vendored again, without > changing our hashes, but most importantly, with valid archives. > > Still, we keep the comment above the versions strings, in the hope that > a future bumper will notice and be more careful at validating the > vendoring. > > [0] https://github.com/rust-lang/cargo/pull/11273 We have some runtime test for gitlab/github download wrapper, do we want a test for cargo vendor too? Reviewed-by: Romain Naour <romain.naour@smile.fr> Tested-by: Romain Naour <romain.naour@smile.fr> Best regards, Romain > > Fixes: > http://autobuild.buildroot.org/results/bea/beac7674bbc9fd2f8777b5861f65afee9c485753/ (bat) > http://autobuild.buildroot.org/results/d1e/d1ec1ebbde115628a4b8b9099544347242a97c1c/ (dust) > http://autobuild.buildroot.org/results/f96/f968be895be9ca98b314fdd688ef8d3bdf4e5dfb/ (hyerfine) > http://autobuild.buildroot.org/results/a0c/a0cdb6cc9493f5248d98f98b13da854e12adc2be/ (ripgrep) > ... and so many others... > > Reported-by: Romain Naour <romain.naour@gmail.com> > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> > Cc: James Hilliard <james.hilliard1@gmail.com> > Cc: Simon Richter <simon.richter@ptwdosimetry.com> > --- > support/download/cargo-post-process | 16 +--------------- > 1 file changed, 1 insertion(+), 15 deletions(-) > > diff --git a/support/download/cargo-post-process b/support/download/cargo-post-process > index 186e9eb69b..21a6be8dbe 100755 > --- a/support/download/cargo-post-process > +++ b/support/download/cargo-post-process > @@ -25,25 +25,11 @@ post_process_unpack "${base_name}" "${output}" > pushd "${base_name}" > /dev/null > > # Create the local .cargo/config with vendor info > -# > -# The first line of the output to stdout is empty. > -# So skip it to have the file start with the vendoring > -# configuration (`tail --lines=+2`). > -# > -# NOTE: > -# There is a patch for cargo to remove the first empty line: > -# See: https://github.com/rust-lang/cargo/pull/11273 > -# > -# The patch already landed in +nightly and will end up > -# in +stable soon. > -# > -# -> When updating rust/cargo, the call to `tail` must be removed. > -# > mkdir -p .cargo/ > cargo vendor \ > --manifest-path ${BR_CARGO_MANIFEST_PATH-Cargo.toml} \ > --locked VENDOR \ > - | tail --lines=+2 | tee .cargo/config > + | tee .cargo/config > > popd > /dev/null >
All, On 2023-01-09 17:30 +0100, Yann E. MORIN spake thusly: > Commit de4cf253752d (package/{rust, rust-bin}: bump to version 1.66.0) > forgot, despite the big comment above the version strings, to confirm > that the vendoring was still working. > > Previously, we were adding the vendoring equivalence manually, but in > commit 04154a651729 (support/download/cargo-post-process: cargo output > for vendor config), we switched to using the output of "vargo vendor" > (on stdout) to support caxses were the vendoring equivalence would be > more complex (e.g. when using crates not hosted on crates.io). > > With rust until and including 1.65.0, "cargo vendor" would output (for > crates.io crates) the same output as our manual fixups, except it was > preceded by an empty line. So, to avoid recompting all our hashes, we > added a tweak to strip away the leading empty line in 04154a651729. > > But rust 1.66.0 includes [0] which changes the output (on stdout) of > "cargo vendor", where the first empty line is no longer emitted. > > This means that our tweak for rust 1.65.0 now strips out an important > part of the cargo vendor output, which renders the archives invalid, and > thus generates different archives, which fail to validate against our > hashes. > > Fix this by doing what the comment in the post-process helper states, > and just keep the whole output of "cargo vendor", byt just removing the > "tail --lines=+2". Since that comment is no longer meaningful, we drop > it too. > > Now, all our 6 cargo-based packages can be vendored again, without > changing our hashes, but most importantly, with valid archives. > > Still, we keep the comment above the versions strings, in the hope that > a future bumper will notice and be more careful at validating the > vendoring. > > [0] https://github.com/rust-lang/cargo/pull/11273 > > Fixes: > http://autobuild.buildroot.org/results/bea/beac7674bbc9fd2f8777b5861f65afee9c485753/ (bat) > http://autobuild.buildroot.org/results/d1e/d1ec1ebbde115628a4b8b9099544347242a97c1c/ (dust) > http://autobuild.buildroot.org/results/f96/f968be895be9ca98b314fdd688ef8d3bdf4e5dfb/ (hyerfine) > http://autobuild.buildroot.org/results/a0c/a0cdb6cc9493f5248d98f98b13da854e12adc2be/ (ripgrep) > ... and so many others... > > Reported-by: Romain Naour <romain.naour@gmail.com> > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> > Cc: James Hilliard <james.hilliard1@gmail.com> > Cc: Simon Richter <simon.richter@ptwdosimetry.com> Applied to master, with all the typoes noticed by Romain fixed, thanks. Regards, Yann E. MORIN. > --- > support/download/cargo-post-process | 16 +--------------- > 1 file changed, 1 insertion(+), 15 deletions(-) > > diff --git a/support/download/cargo-post-process b/support/download/cargo-post-process > index 186e9eb69b..21a6be8dbe 100755 > --- a/support/download/cargo-post-process > +++ b/support/download/cargo-post-process > @@ -25,25 +25,11 @@ post_process_unpack "${base_name}" "${output}" > pushd "${base_name}" > /dev/null > > # Create the local .cargo/config with vendor info > -# > -# The first line of the output to stdout is empty. > -# So skip it to have the file start with the vendoring > -# configuration (`tail --lines=+2`). > -# > -# NOTE: > -# There is a patch for cargo to remove the first empty line: > -# See: https://github.com/rust-lang/cargo/pull/11273 > -# > -# The patch already landed in +nightly and will end up > -# in +stable soon. > -# > -# -> When updating rust/cargo, the call to `tail` must be removed. > -# > mkdir -p .cargo/ > cargo vendor \ > --manifest-path ${BR_CARGO_MANIFEST_PATH-Cargo.toml} \ > --locked VENDOR \ > - | tail --lines=+2 | tee .cargo/config > + | tee .cargo/config > > popd > /dev/null > > -- > 2.25.1 >
Romain, All, On 2023-01-10 13:49 +0100, Romain Naour spake thusly: > Le 09/01/2023 à 17:30, Yann E. MORIN a écrit : [--SNIP--] > > Still, we keep the comment above the versions strings, in the hope that > > a future bumper will notice and be more careful at validating the > > vendoring. > We have some runtime test for gitlab/github download wrapper, do we want a test > for cargo vendor too? Yes, that would be nice, indeed. > Reviewed-by: Romain Naour <romain.naour@smile.fr> > Tested-by: Romain Naour <romain.naour@smile.fr> Typoes fixed when applying, thanks! ;-) Regards, Yann E. MORIN. > Best regards, > Romain > > > > > Fixes: > > http://autobuild.buildroot.org/results/bea/beac7674bbc9fd2f8777b5861f65afee9c485753/ (bat) > > http://autobuild.buildroot.org/results/d1e/d1ec1ebbde115628a4b8b9099544347242a97c1c/ (dust) > > http://autobuild.buildroot.org/results/f96/f968be895be9ca98b314fdd688ef8d3bdf4e5dfb/ (hyerfine) > > http://autobuild.buildroot.org/results/a0c/a0cdb6cc9493f5248d98f98b13da854e12adc2be/ (ripgrep) > > ... and so many others... > > > > Reported-by: Romain Naour <romain.naour@gmail.com> > > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> > > Cc: James Hilliard <james.hilliard1@gmail.com> > > Cc: Simon Richter <simon.richter@ptwdosimetry.com> > > --- > > support/download/cargo-post-process | 16 +--------------- > > 1 file changed, 1 insertion(+), 15 deletions(-) > > > > diff --git a/support/download/cargo-post-process b/support/download/cargo-post-process > > index 186e9eb69b..21a6be8dbe 100755 > > --- a/support/download/cargo-post-process > > +++ b/support/download/cargo-post-process > > @@ -25,25 +25,11 @@ post_process_unpack "${base_name}" "${output}" > > pushd "${base_name}" > /dev/null > > > > # Create the local .cargo/config with vendor info > > -# > > -# The first line of the output to stdout is empty. > > -# So skip it to have the file start with the vendoring > > -# configuration (`tail --lines=+2`). > > -# > > -# NOTE: > > -# There is a patch for cargo to remove the first empty line: > > -# See: https://github.com/rust-lang/cargo/pull/11273 > > -# > > -# The patch already landed in +nightly and will end up > > -# in +stable soon. > > -# > > -# -> When updating rust/cargo, the call to `tail` must be removed. > > -# > > mkdir -p .cargo/ > > cargo vendor \ > > --manifest-path ${BR_CARGO_MANIFEST_PATH-Cargo.toml} \ > > --locked VENDOR \ > > - | tail --lines=+2 | tee .cargo/config > > + | tee .cargo/config > > > > popd > /dev/null > > > > _______________________________________________ > buildroot mailing list > buildroot@buildroot.org > https://lists.buildroot.org/mailman/listinfo/buildroot
diff --git a/support/download/cargo-post-process b/support/download/cargo-post-process index 186e9eb69b..21a6be8dbe 100755 --- a/support/download/cargo-post-process +++ b/support/download/cargo-post-process @@ -25,25 +25,11 @@ post_process_unpack "${base_name}" "${output}" pushd "${base_name}" > /dev/null # Create the local .cargo/config with vendor info -# -# The first line of the output to stdout is empty. -# So skip it to have the file start with the vendoring -# configuration (`tail --lines=+2`). -# -# NOTE: -# There is a patch for cargo to remove the first empty line: -# See: https://github.com/rust-lang/cargo/pull/11273 -# -# The patch already landed in +nightly and will end up -# in +stable soon. -# -# -> When updating rust/cargo, the call to `tail` must be removed. -# mkdir -p .cargo/ cargo vendor \ --manifest-path ${BR_CARGO_MANIFEST_PATH-Cargo.toml} \ --locked VENDOR \ - | tail --lines=+2 | tee .cargo/config + | tee .cargo/config popd > /dev/null
Commit de4cf253752d (package/{rust, rust-bin}: bump to version 1.66.0) forgot, despite the big comment above the version strings, to confirm that the vendoring was still working. Previously, we were adding the vendoring equivalence manually, but in commit 04154a651729 (support/download/cargo-post-process: cargo output for vendor config), we switched to using the output of "vargo vendor" (on stdout) to support caxses were the vendoring equivalence would be more complex (e.g. when using crates not hosted on crates.io). With rust until and including 1.65.0, "cargo vendor" would output (for crates.io crates) the same output as our manual fixups, except it was preceded by an empty line. So, to avoid recompting all our hashes, we added a tweak to strip away the leading empty line in 04154a651729. But rust 1.66.0 includes [0] which changes the output (on stdout) of "cargo vendor", where the first empty line is no longer emitted. This means that our tweak for rust 1.65.0 now strips out an important part of the cargo vendor output, which renders the archives invalid, and thus generates different archives, which fail to validate against our hashes. Fix this by doing what the comment in the post-process helper states, and just keep the whole output of "cargo vendor", byt just removing the "tail --lines=+2". Since that comment is no longer meaningful, we drop it too. Now, all our 6 cargo-based packages can be vendored again, without changing our hashes, but most importantly, with valid archives. Still, we keep the comment above the versions strings, in the hope that a future bumper will notice and be more careful at validating the vendoring. [0] https://github.com/rust-lang/cargo/pull/11273 Fixes: http://autobuild.buildroot.org/results/bea/beac7674bbc9fd2f8777b5861f65afee9c485753/ (bat) http://autobuild.buildroot.org/results/d1e/d1ec1ebbde115628a4b8b9099544347242a97c1c/ (dust) http://autobuild.buildroot.org/results/f96/f968be895be9ca98b314fdd688ef8d3bdf4e5dfb/ (hyerfine) http://autobuild.buildroot.org/results/a0c/a0cdb6cc9493f5248d98f98b13da854e12adc2be/ (ripgrep) ... and so many others... Reported-by: Romain Naour <romain.naour@gmail.com> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: James Hilliard <james.hilliard1@gmail.com> Cc: Simon Richter <simon.richter@ptwdosimetry.com> --- support/download/cargo-post-process | 16 +--------------- 1 file changed, 1 insertion(+), 15 deletions(-)