| Message ID | 20230908064507.14596-1-jasowang@redhat.com |
|---|---|
| State | New |
| Headers | show |
Hi Ilya and Jason,
There is a CI failure related to a missing Debian libxdp-dev package:
https://gitlab.com/qemu-project/qemu/-/jobs/5046139967
I think the issue is that the debian-amd64 container image that QEMU
uses for testing is based on Debian 11 ("bullseye" aka "oldstable")
and libxdp is not available on that release:
https://packages.debian.org/search?keywords=libxdp&searchon=names&suite=oldstable§ion=all
If we need to support Debian 11 CI then either XDP could be disabled
for that distro or libxdp could be compiled from source.
I have CCed Daniel Berrangé, because I think he knows how lcitool and
QEMU's minimum distro requirements work. Maybe he can suggest a way
forward.
Thanks,
Stefan
On 9/8/23 13:19, Stefan Hajnoczi wrote: > Hi Ilya and Jason, > There is a CI failure related to a missing Debian libxdp-dev package: > https://gitlab.com/qemu-project/qemu/-/jobs/5046139967 > > I think the issue is that the debian-amd64 container image that QEMU > uses for testing is based on Debian 11 ("bullseye" aka "oldstable") > and libxdp is not available on that release: > https://packages.debian.org/search?keywords=libxdp&searchon=names&suite=oldstable§ion=all Hmm. Sorry about that. > > If we need to support Debian 11 CI then either XDP could be disabled > for that distro or libxdp could be compiled from source. I'd suggest we just remove the attempt to install the package for now, building libxdp from sources may be a little painful to maintain. Can be re-added later once distributions with libxdp 1.4+ will be more widely available, i.e. when fedora dockerfile will be updated to 39, for example. That should be soon-ish, right? > > I have CCed Daniel Berrangé, because I think he knows how lcitool and > QEMU's minimum distro requirements work. Maybe he can suggest a way > forward. > > Thanks, > Stefan
On Fri, Sep 08, 2023 at 01:34:54PM +0200, Ilya Maximets wrote: > On 9/8/23 13:19, Stefan Hajnoczi wrote: > > Hi Ilya and Jason, > > There is a CI failure related to a missing Debian libxdp-dev package: > > https://gitlab.com/qemu-project/qemu/-/jobs/5046139967 > > > > I think the issue is that the debian-amd64 container image that QEMU > > uses for testing is based on Debian 11 ("bullseye" aka "oldstable") > > and libxdp is not available on that release: > > https://packages.debian.org/search?keywords=libxdp&searchon=names&suite=oldstable§ion=all > > Hmm. Sorry about that. > > > > > If we need to support Debian 11 CI then either XDP could be disabled > > for that distro or libxdp could be compiled from source. > > I'd suggest we just remove the attempt to install the package for now, > building libxdp from sources may be a little painful to maintain. > > Can be re-added later once distributions with libxdp 1.4+ will be more > widely available, i.e. when fedora dockerfile will be updated to 39, > for example. That should be soon-ish, right? If you follow the process in docs/devel/testing.rst for adding libxdp in libvirt-ci, then lcitool will "do the right thing" when we move the auto-generated dockerfiles to new distro versions. With regards, Daniel
On 9/8/23 13:49, Daniel P. Berrangé wrote: > On Fri, Sep 08, 2023 at 01:34:54PM +0200, Ilya Maximets wrote: >> On 9/8/23 13:19, Stefan Hajnoczi wrote: >>> Hi Ilya and Jason, >>> There is a CI failure related to a missing Debian libxdp-dev package: >>> https://gitlab.com/qemu-project/qemu/-/jobs/5046139967 >>> >>> I think the issue is that the debian-amd64 container image that QEMU >>> uses for testing is based on Debian 11 ("bullseye" aka "oldstable") >>> and libxdp is not available on that release: >>> https://packages.debian.org/search?keywords=libxdp&searchon=names&suite=oldstable§ion=all >> >> Hmm. Sorry about that. >> >>> >>> If we need to support Debian 11 CI then either XDP could be disabled >>> for that distro or libxdp could be compiled from source. >> >> I'd suggest we just remove the attempt to install the package for now, >> building libxdp from sources may be a little painful to maintain. >> >> Can be re-added later once distributions with libxdp 1.4+ will be more >> widely available, i.e. when fedora dockerfile will be updated to 39, >> for example. That should be soon-ish, right? > > If you follow the process in docs/devel/testing.rst for adding > libxdp in libvirt-ci, then lcitool will "do the right thing" > when we move the auto-generated dockerfiles to new distro versions. Thanks! I'll prepare changes for libvirt-ci. In the meantime, none of the currently tested images will have a required version of libxdp anyway, so I'm suggesting to just drop this one dockerfile modification from the patch. What do you think? Best regards, Ilya Maximets.
On Fri, Sep 08, 2023 at 02:00:47PM +0200, Ilya Maximets wrote: > On 9/8/23 13:49, Daniel P. Berrangé wrote: > > On Fri, Sep 08, 2023 at 01:34:54PM +0200, Ilya Maximets wrote: > >> On 9/8/23 13:19, Stefan Hajnoczi wrote: > >>> Hi Ilya and Jason, > >>> There is a CI failure related to a missing Debian libxdp-dev package: > >>> https://gitlab.com/qemu-project/qemu/-/jobs/5046139967 > >>> > >>> I think the issue is that the debian-amd64 container image that QEMU > >>> uses for testing is based on Debian 11 ("bullseye" aka "oldstable") > >>> and libxdp is not available on that release: > >>> https://packages.debian.org/search?keywords=libxdp&searchon=names&suite=oldstable§ion=all > >> > >> Hmm. Sorry about that. > >> > >>> > >>> If we need to support Debian 11 CI then either XDP could be disabled > >>> for that distro or libxdp could be compiled from source. > >> > >> I'd suggest we just remove the attempt to install the package for now, > >> building libxdp from sources may be a little painful to maintain. > >> > >> Can be re-added later once distributions with libxdp 1.4+ will be more > >> widely available, i.e. when fedora dockerfile will be updated to 39, > >> for example. That should be soon-ish, right? > > > > If you follow the process in docs/devel/testing.rst for adding > > libxdp in libvirt-ci, then lcitool will "do the right thing" > > when we move the auto-generated dockerfiles to new distro versions. > > Thanks! I'll prepare changes for libvirt-ci. > > In the meantime, none of the currently tested images will have a required > version of libxdp anyway, so I'm suggesting to just drop this one dockerfile > modification from the patch. What do you think? Sure, if none of the distros have it, then lcitool won't emit the dockerfile changes until we update the inherited distro version. So it is sufficient to just update libvirt-ci.git with the mappings.yml info for libxdp, and add 'libxdp' to the tests/lcitool/projects/qemu.yml file in qemu.git. It will then 'just work' when someone updates the distro versions later. With regards, Daniel
On 9/8/23 14:15, Daniel P. Berrangé wrote: > On Fri, Sep 08, 2023 at 02:00:47PM +0200, Ilya Maximets wrote: >> On 9/8/23 13:49, Daniel P. Berrangé wrote: >>> On Fri, Sep 08, 2023 at 01:34:54PM +0200, Ilya Maximets wrote: >>>> On 9/8/23 13:19, Stefan Hajnoczi wrote: >>>>> Hi Ilya and Jason, >>>>> There is a CI failure related to a missing Debian libxdp-dev package: >>>>> https://gitlab.com/qemu-project/qemu/-/jobs/5046139967 >>>>> >>>>> I think the issue is that the debian-amd64 container image that QEMU >>>>> uses for testing is based on Debian 11 ("bullseye" aka "oldstable") >>>>> and libxdp is not available on that release: >>>>> https://packages.debian.org/search?keywords=libxdp&searchon=names&suite=oldstable§ion=all >>>> >>>> Hmm. Sorry about that. >>>> >>>>> >>>>> If we need to support Debian 11 CI then either XDP could be disabled >>>>> for that distro or libxdp could be compiled from source. >>>> >>>> I'd suggest we just remove the attempt to install the package for now, >>>> building libxdp from sources may be a little painful to maintain. >>>> >>>> Can be re-added later once distributions with libxdp 1.4+ will be more >>>> widely available, i.e. when fedora dockerfile will be updated to 39, >>>> for example. That should be soon-ish, right? >>> >>> If you follow the process in docs/devel/testing.rst for adding >>> libxdp in libvirt-ci, then lcitool will "do the right thing" >>> when we move the auto-generated dockerfiles to new distro versions. >> >> Thanks! I'll prepare changes for libvirt-ci. >> >> In the meantime, none of the currently tested images will have a required >> version of libxdp anyway, so I'm suggesting to just drop this one dockerfile >> modification from the patch. What do you think? > > Sure, if none of the distros have it, then lcitool won't emit the > dockerfile changes until we update the inherited distro version. > So it is sufficient to just update libvirt-ci.git with the mappings.yml > info for libxdp, and add 'libxdp' to the tests/lcitool/projects/qemu.yml > file in qemu.git. It will then 'just work' when someone updates the > distro versions later. I posted an MR for libvirt-ci adding libxdp: https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/429 Please, take a look. The docs say that CI will try to build containers with the MR changes, but I don't think anything except sanity checks is actually tested on MR. Sorry if I missed something, never used GitLab pipelines before. Note that with this update we will be installing older version of libxdp in many containers, even though they will not be used by QEMU, unless they are newer than 1.4.0. tests/lcitool/projects/qemu.yml in qemu.git cannot be updated without updating a submodule after the MR merge. Best regards, Ilya Maximets.
On Fri, Sep 08, 2023 at 04:06:35PM +0200, Ilya Maximets wrote: > On 9/8/23 14:15, Daniel P. Berrangé wrote: > > On Fri, Sep 08, 2023 at 02:00:47PM +0200, Ilya Maximets wrote: > >> On 9/8/23 13:49, Daniel P. Berrangé wrote: > >>> On Fri, Sep 08, 2023 at 01:34:54PM +0200, Ilya Maximets wrote: > >>>> On 9/8/23 13:19, Stefan Hajnoczi wrote: > >>>>> Hi Ilya and Jason, > >>>>> There is a CI failure related to a missing Debian libxdp-dev package: > >>>>> https://gitlab.com/qemu-project/qemu/-/jobs/5046139967 > >>>>> > >>>>> I think the issue is that the debian-amd64 container image that QEMU > >>>>> uses for testing is based on Debian 11 ("bullseye" aka "oldstable") > >>>>> and libxdp is not available on that release: > >>>>> https://packages.debian.org/search?keywords=libxdp&searchon=names&suite=oldstable§ion=all > >>>> > >>>> Hmm. Sorry about that. > >>>> > >>>>> > >>>>> If we need to support Debian 11 CI then either XDP could be disabled > >>>>> for that distro or libxdp could be compiled from source. > >>>> > >>>> I'd suggest we just remove the attempt to install the package for now, > >>>> building libxdp from sources may be a little painful to maintain. > >>>> > >>>> Can be re-added later once distributions with libxdp 1.4+ will be more > >>>> widely available, i.e. when fedora dockerfile will be updated to 39, > >>>> for example. That should be soon-ish, right? > >>> > >>> If you follow the process in docs/devel/testing.rst for adding > >>> libxdp in libvirt-ci, then lcitool will "do the right thing" > >>> when we move the auto-generated dockerfiles to new distro versions. > >> > >> Thanks! I'll prepare changes for libvirt-ci. > >> > >> In the meantime, none of the currently tested images will have a required > >> version of libxdp anyway, so I'm suggesting to just drop this one dockerfile > >> modification from the patch. What do you think? > > > > Sure, if none of the distros have it, then lcitool won't emit the > > dockerfile changes until we update the inherited distro version. > > So it is sufficient to just update libvirt-ci.git with the mappings.yml > > info for libxdp, and add 'libxdp' to the tests/lcitool/projects/qemu.yml > > file in qemu.git. It will then 'just work' when someone updates the > > distro versions later. > > I posted an MR for libvirt-ci adding libxdp: > https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/429 > > Please, take a look. > > The docs say that CI will try to build containers with the MR changes, > but I don't think anything except sanity checks is actually tested on MR. > Sorry if I missed something, never used GitLab pipelines before. No, that's our fault - we've broken the CI and your change alerted me to that fact :-) > Note that with this update we will be installing older version of libxdp > in many containers, even though they will not be used by QEMU, unless > they are newer than 1.4.0. No problem, as it means QEMU CI will demonstrate the the meson.build change is ignoring the outdatd libxdp. > tests/lcitool/projects/qemu.yml in qemu.git cannot be updated without > updating a submodule after the MR merge. Yep. With regards, Daniel
On 9/8/23 16:15, Daniel P. Berrangé wrote: > On Fri, Sep 08, 2023 at 04:06:35PM +0200, Ilya Maximets wrote: >> On 9/8/23 14:15, Daniel P. Berrangé wrote: >>> On Fri, Sep 08, 2023 at 02:00:47PM +0200, Ilya Maximets wrote: >>>> On 9/8/23 13:49, Daniel P. Berrangé wrote: >>>>> On Fri, Sep 08, 2023 at 01:34:54PM +0200, Ilya Maximets wrote: >>>>>> On 9/8/23 13:19, Stefan Hajnoczi wrote: >>>>>>> Hi Ilya and Jason, >>>>>>> There is a CI failure related to a missing Debian libxdp-dev package: >>>>>>> https://gitlab.com/qemu-project/qemu/-/jobs/5046139967 >>>>>>> >>>>>>> I think the issue is that the debian-amd64 container image that QEMU >>>>>>> uses for testing is based on Debian 11 ("bullseye" aka "oldstable") >>>>>>> and libxdp is not available on that release: >>>>>>> https://packages.debian.org/search?keywords=libxdp&searchon=names&suite=oldstable§ion=all >>>>>> >>>>>> Hmm. Sorry about that. >>>>>> >>>>>>> >>>>>>> If we need to support Debian 11 CI then either XDP could be disabled >>>>>>> for that distro or libxdp could be compiled from source. >>>>>> >>>>>> I'd suggest we just remove the attempt to install the package for now, >>>>>> building libxdp from sources may be a little painful to maintain. >>>>>> >>>>>> Can be re-added later once distributions with libxdp 1.4+ will be more >>>>>> widely available, i.e. when fedora dockerfile will be updated to 39, >>>>>> for example. That should be soon-ish, right? >>>>> >>>>> If you follow the process in docs/devel/testing.rst for adding >>>>> libxdp in libvirt-ci, then lcitool will "do the right thing" >>>>> when we move the auto-generated dockerfiles to new distro versions. >>>> >>>> Thanks! I'll prepare changes for libvirt-ci. >>>> >>>> In the meantime, none of the currently tested images will have a required >>>> version of libxdp anyway, so I'm suggesting to just drop this one dockerfile >>>> modification from the patch. What do you think? >>> >>> Sure, if none of the distros have it, then lcitool won't emit the >>> dockerfile changes until we update the inherited distro version. >>> So it is sufficient to just update libvirt-ci.git with the mappings.yml >>> info for libxdp, and add 'libxdp' to the tests/lcitool/projects/qemu.yml >>> file in qemu.git. It will then 'just work' when someone updates the >>> distro versions later. >> >> I posted an MR for libvirt-ci adding libxdp: >> https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/429 >> >> Please, take a look. >> >> The docs say that CI will try to build containers with the MR changes, >> but I don't think anything except sanity checks is actually tested on MR. >> Sorry if I missed something, never used GitLab pipelines before. > > No, that's our fault - we've broken the CI and your change alerted > me to that fact :-) > >> Note that with this update we will be installing older version of libxdp >> in many containers, even though they will not be used by QEMU, unless >> they are newer than 1.4.0. > > No problem, as it means QEMU CI will demonstrate the the meson.build > change is ignoring the outdatd libxdp. > >> tests/lcitool/projects/qemu.yml in qemu.git cannot be updated without >> updating a submodule after the MR merge. > > Yep. Since all the required changes went into libvirt-ci project, I posted an updated patch set named: '[PATCH v4 0/2] net: add initial support for AF_XDP network backend' Please, take a look. This should fix the CI issues, though I'm not sure how to run QEMU gitlab pipelines myself, so I didn't actually test all the images. Sent as a patch set because the libvirt-ci submodule bump brings in a few unrelated changes. So, I split that into a separate patch. Best regards, Ilya Maximets.
On Wed, Sep 13, 2023 at 08:46:42PM +0200, Ilya Maximets wrote: > On 9/8/23 16:15, Daniel P. Berrangé wrote: > > On Fri, Sep 08, 2023 at 04:06:35PM +0200, Ilya Maximets wrote: > >> On 9/8/23 14:15, Daniel P. Berrangé wrote: > >>> On Fri, Sep 08, 2023 at 02:00:47PM +0200, Ilya Maximets wrote: > >>>> On 9/8/23 13:49, Daniel P. Berrangé wrote: > >>>>> On Fri, Sep 08, 2023 at 01:34:54PM +0200, Ilya Maximets wrote: > >>>>>> On 9/8/23 13:19, Stefan Hajnoczi wrote: > >>>>>>> Hi Ilya and Jason, > >>>>>>> There is a CI failure related to a missing Debian libxdp-dev package: > >>>>>>> https://gitlab.com/qemu-project/qemu/-/jobs/5046139967 > >>>>>>> > >>>>>>> I think the issue is that the debian-amd64 container image that QEMU > >>>>>>> uses for testing is based on Debian 11 ("bullseye" aka "oldstable") > >>>>>>> and libxdp is not available on that release: > >>>>>>> https://packages.debian.org/search?keywords=libxdp&searchon=names&suite=oldstable§ion=all > >>>>>> > >>>>>> Hmm. Sorry about that. > >>>>>> > >>>>>>> > >>>>>>> If we need to support Debian 11 CI then either XDP could be disabled > >>>>>>> for that distro or libxdp could be compiled from source. > >>>>>> > >>>>>> I'd suggest we just remove the attempt to install the package for now, > >>>>>> building libxdp from sources may be a little painful to maintain. > >>>>>> > >>>>>> Can be re-added later once distributions with libxdp 1.4+ will be more > >>>>>> widely available, i.e. when fedora dockerfile will be updated to 39, > >>>>>> for example. That should be soon-ish, right? > >>>>> > >>>>> If you follow the process in docs/devel/testing.rst for adding > >>>>> libxdp in libvirt-ci, then lcitool will "do the right thing" > >>>>> when we move the auto-generated dockerfiles to new distro versions. > >>>> > >>>> Thanks! I'll prepare changes for libvirt-ci. > >>>> > >>>> In the meantime, none of the currently tested images will have a required > >>>> version of libxdp anyway, so I'm suggesting to just drop this one dockerfile > >>>> modification from the patch. What do you think? > >>> > >>> Sure, if none of the distros have it, then lcitool won't emit the > >>> dockerfile changes until we update the inherited distro version. > >>> So it is sufficient to just update libvirt-ci.git with the mappings.yml > >>> info for libxdp, and add 'libxdp' to the tests/lcitool/projects/qemu.yml > >>> file in qemu.git. It will then 'just work' when someone updates the > >>> distro versions later. > >> > >> I posted an MR for libvirt-ci adding libxdp: > >> https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/429 > >> > >> Please, take a look. > >> > >> The docs say that CI will try to build containers with the MR changes, > >> but I don't think anything except sanity checks is actually tested on MR. > >> Sorry if I missed something, never used GitLab pipelines before. > > > > No, that's our fault - we've broken the CI and your change alerted > > me to that fact :-) > > > >> Note that with this update we will be installing older version of libxdp > >> in many containers, even though they will not be used by QEMU, unless > >> they are newer than 1.4.0. > > > > No problem, as it means QEMU CI will demonstrate the the meson.build > > change is ignoring the outdatd libxdp. > > > >> tests/lcitool/projects/qemu.yml in qemu.git cannot be updated without > >> updating a submodule after the MR merge. > > > > Yep. > > Since all the required changes went into libvirt-ci project, I posted an > updated patch set named: > > '[PATCH v4 0/2] net: add initial support for AF_XDP network backend' > > Please, take a look. > > This should fix the CI issues, though I'm not sure how to run QEMU gitlab > pipelines myself, so I didn't actually test all the images. git push gitlab -o ci.variable=QEMU_CI=2 will create pipeline and immediately run all jobs. Replace 'gitlab' with the name of the git remote pointing to your gitlab fork of QEMU. Using QEMU_CI=1 will create pipeline, but let you manually start individual jobs from the web UI. For further details see docs/devel/ci-jobs.rst.inc > > Sent as a patch set because the libvirt-ci submodule bump brings in a few > unrelated changes. So, I split that into a separate patch. Yep, that's perfect thanks. With regards, Daniel
On 9/14/23 10:13, Daniel P. Berrangé wrote: > On Wed, Sep 13, 2023 at 08:46:42PM +0200, Ilya Maximets wrote: >> On 9/8/23 16:15, Daniel P. Berrangé wrote: >>> On Fri, Sep 08, 2023 at 04:06:35PM +0200, Ilya Maximets wrote: >>>> On 9/8/23 14:15, Daniel P. Berrangé wrote: >>>>> On Fri, Sep 08, 2023 at 02:00:47PM +0200, Ilya Maximets wrote: >>>>>> On 9/8/23 13:49, Daniel P. Berrangé wrote: >>>>>>> On Fri, Sep 08, 2023 at 01:34:54PM +0200, Ilya Maximets wrote: >>>>>>>> On 9/8/23 13:19, Stefan Hajnoczi wrote: >>>>>>>>> Hi Ilya and Jason, >>>>>>>>> There is a CI failure related to a missing Debian libxdp-dev package: >>>>>>>>> https://gitlab.com/qemu-project/qemu/-/jobs/5046139967 >>>>>>>>> >>>>>>>>> I think the issue is that the debian-amd64 container image that QEMU >>>>>>>>> uses for testing is based on Debian 11 ("bullseye" aka "oldstable") >>>>>>>>> and libxdp is not available on that release: >>>>>>>>> https://packages.debian.org/search?keywords=libxdp&searchon=names&suite=oldstable§ion=all >>>>>>>> >>>>>>>> Hmm. Sorry about that. >>>>>>>> >>>>>>>>> >>>>>>>>> If we need to support Debian 11 CI then either XDP could be disabled >>>>>>>>> for that distro or libxdp could be compiled from source. >>>>>>>> >>>>>>>> I'd suggest we just remove the attempt to install the package for now, >>>>>>>> building libxdp from sources may be a little painful to maintain. >>>>>>>> >>>>>>>> Can be re-added later once distributions with libxdp 1.4+ will be more >>>>>>>> widely available, i.e. when fedora dockerfile will be updated to 39, >>>>>>>> for example. That should be soon-ish, right? >>>>>>> >>>>>>> If you follow the process in docs/devel/testing.rst for adding >>>>>>> libxdp in libvirt-ci, then lcitool will "do the right thing" >>>>>>> when we move the auto-generated dockerfiles to new distro versions. >>>>>> >>>>>> Thanks! I'll prepare changes for libvirt-ci. >>>>>> >>>>>> In the meantime, none of the currently tested images will have a required >>>>>> version of libxdp anyway, so I'm suggesting to just drop this one dockerfile >>>>>> modification from the patch. What do you think? >>>>> >>>>> Sure, if none of the distros have it, then lcitool won't emit the >>>>> dockerfile changes until we update the inherited distro version. >>>>> So it is sufficient to just update libvirt-ci.git with the mappings.yml >>>>> info for libxdp, and add 'libxdp' to the tests/lcitool/projects/qemu.yml >>>>> file in qemu.git. It will then 'just work' when someone updates the >>>>> distro versions later. >>>> >>>> I posted an MR for libvirt-ci adding libxdp: >>>> https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/429 >>>> >>>> Please, take a look. >>>> >>>> The docs say that CI will try to build containers with the MR changes, >>>> but I don't think anything except sanity checks is actually tested on MR. >>>> Sorry if I missed something, never used GitLab pipelines before. >>> >>> No, that's our fault - we've broken the CI and your change alerted >>> me to that fact :-) >>> >>>> Note that with this update we will be installing older version of libxdp >>>> in many containers, even though they will not be used by QEMU, unless >>>> they are newer than 1.4.0. >>> >>> No problem, as it means QEMU CI will demonstrate the the meson.build >>> change is ignoring the outdatd libxdp. >>> >>>> tests/lcitool/projects/qemu.yml in qemu.git cannot be updated without >>>> updating a submodule after the MR merge. >>> >>> Yep. >> >> Since all the required changes went into libvirt-ci project, I posted an >> updated patch set named: >> >> '[PATCH v4 0/2] net: add initial support for AF_XDP network backend' >> >> Please, take a look. >> >> This should fix the CI issues, though I'm not sure how to run QEMU gitlab >> pipelines myself, so I didn't actually test all the images. > > git push gitlab -o ci.variable=QEMU_CI=2 > > will create pipeline and immediately run all jobs. Thanks! That worked. Though I wasn't able to test much anyway as this thing burned through all my free compute credits less than half way through the pipeline. :D So, AFAIU, it's not something an occasional contributor like me can use, unless they are spending their own money. > > Replace 'gitlab' with the name of the git remote pointing to your > gitlab fork of QEMU. > > Using QEMU_CI=1 will create pipeline, but let you manually start > individual jobs from the web UI. > > For further details see docs/devel/ci-jobs.rst.inc > > >> >> Sent as a patch set because the libvirt-ci submodule bump brings in a few >> unrelated changes. So, I split that into a separate patch. > > Yep, that's perfect thanks. > > With regards, > Daniel
On Mon, Sep 18, 2023 at 09:36:10PM +0200, Ilya Maximets wrote: > On 9/14/23 10:13, Daniel P. Berrangé wrote: > > On Wed, Sep 13, 2023 at 08:46:42PM +0200, Ilya Maximets wrote: > >> On 9/8/23 16:15, Daniel P. Berrangé wrote: > >>> On Fri, Sep 08, 2023 at 04:06:35PM +0200, Ilya Maximets wrote: > >>>> On 9/8/23 14:15, Daniel P. Berrangé wrote: > >>>>> On Fri, Sep 08, 2023 at 02:00:47PM +0200, Ilya Maximets wrote: > >>>>>> On 9/8/23 13:49, Daniel P. Berrangé wrote: > >>>>>>> On Fri, Sep 08, 2023 at 01:34:54PM +0200, Ilya Maximets wrote: > >>>>>>>> On 9/8/23 13:19, Stefan Hajnoczi wrote: > >>>>>>>>> Hi Ilya and Jason, > >>>>>>>>> There is a CI failure related to a missing Debian libxdp-dev package: > >>>>>>>>> https://gitlab.com/qemu-project/qemu/-/jobs/5046139967 > >>>>>>>>> > >>>>>>>>> I think the issue is that the debian-amd64 container image that QEMU > >>>>>>>>> uses for testing is based on Debian 11 ("bullseye" aka "oldstable") > >>>>>>>>> and libxdp is not available on that release: > >>>>>>>>> https://packages.debian.org/search?keywords=libxdp&searchon=names&suite=oldstable§ion=all > >>>>>>>> > >>>>>>>> Hmm. Sorry about that. > >>>>>>>> > >>>>>>>>> > >>>>>>>>> If we need to support Debian 11 CI then either XDP could be disabled > >>>>>>>>> for that distro or libxdp could be compiled from source. > >>>>>>>> > >>>>>>>> I'd suggest we just remove the attempt to install the package for now, > >>>>>>>> building libxdp from sources may be a little painful to maintain. > >>>>>>>> > >>>>>>>> Can be re-added later once distributions with libxdp 1.4+ will be more > >>>>>>>> widely available, i.e. when fedora dockerfile will be updated to 39, > >>>>>>>> for example. That should be soon-ish, right? > >>>>>>> > >>>>>>> If you follow the process in docs/devel/testing.rst for adding > >>>>>>> libxdp in libvirt-ci, then lcitool will "do the right thing" > >>>>>>> when we move the auto-generated dockerfiles to new distro versions. > >>>>>> > >>>>>> Thanks! I'll prepare changes for libvirt-ci. > >>>>>> > >>>>>> In the meantime, none of the currently tested images will have a required > >>>>>> version of libxdp anyway, so I'm suggesting to just drop this one dockerfile > >>>>>> modification from the patch. What do you think? > >>>>> > >>>>> Sure, if none of the distros have it, then lcitool won't emit the > >>>>> dockerfile changes until we update the inherited distro version. > >>>>> So it is sufficient to just update libvirt-ci.git with the mappings.yml > >>>>> info for libxdp, and add 'libxdp' to the tests/lcitool/projects/qemu.yml > >>>>> file in qemu.git. It will then 'just work' when someone updates the > >>>>> distro versions later. > >>>> > >>>> I posted an MR for libvirt-ci adding libxdp: > >>>> https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/429 > >>>> > >>>> Please, take a look. > >>>> > >>>> The docs say that CI will try to build containers with the MR changes, > >>>> but I don't think anything except sanity checks is actually tested on MR. > >>>> Sorry if I missed something, never used GitLab pipelines before. > >>> > >>> No, that's our fault - we've broken the CI and your change alerted > >>> me to that fact :-) > >>> > >>>> Note that with this update we will be installing older version of libxdp > >>>> in many containers, even though they will not be used by QEMU, unless > >>>> they are newer than 1.4.0. > >>> > >>> No problem, as it means QEMU CI will demonstrate the the meson.build > >>> change is ignoring the outdatd libxdp. > >>> > >>>> tests/lcitool/projects/qemu.yml in qemu.git cannot be updated without > >>>> updating a submodule after the MR merge. > >>> > >>> Yep. > >> > >> Since all the required changes went into libvirt-ci project, I posted an > >> updated patch set named: > >> > >> '[PATCH v4 0/2] net: add initial support for AF_XDP network backend' > >> > >> Please, take a look. > >> > >> This should fix the CI issues, though I'm not sure how to run QEMU gitlab > >> pipelines myself, so I didn't actually test all the images. > > > > git push gitlab -o ci.variable=QEMU_CI=2 > > > > will create pipeline and immediately run all jobs. > > Thanks! That worked. Though I wasn't able to test much anyway as > this thing burned through all my free compute credits less than > half way through the pipeline. :D > > So, AFAIU, it's not something an occasional contributor like me can > use, unless they are spending their own money. That is not the expected behaviour. If your repo is a fork of https://gitlab.com/qemu-project/qemu it should benefit from a *massive* x125 reduction on CI costs. The critical thing is that it *MUST* have been created with the 'Fork' button on qemu-project/qemu. If that's not the case then you will burn CI credits at a cost of 1 minute == 1 credit, instead of 1 minute == 0.008 credits. Check this by going to the top page of your repo, and looking for a box a little above the file list, that says "Forked from QEMU / QEMU" If that is not the case, then you'll have to rename your existing repo to get it out of the way, and then use the 'Fork' button to create a new copy that is tracked as a fork. With most accounts getting 400 CI minutes per month, an averge QEMU CI run should consume about 7 CI minutes. NB, CI credits reset on the 1st of each month With regards, Daniel
On 9/19/23 10:40, Daniel P. Berrangé wrote: > On Mon, Sep 18, 2023 at 09:36:10PM +0200, Ilya Maximets wrote: >> On 9/14/23 10:13, Daniel P. Berrangé wrote: >>> On Wed, Sep 13, 2023 at 08:46:42PM +0200, Ilya Maximets wrote: >>>> On 9/8/23 16:15, Daniel P. Berrangé wrote: >>>>> On Fri, Sep 08, 2023 at 04:06:35PM +0200, Ilya Maximets wrote: >>>>>> On 9/8/23 14:15, Daniel P. Berrangé wrote: >>>>>>> On Fri, Sep 08, 2023 at 02:00:47PM +0200, Ilya Maximets wrote: >>>>>>>> On 9/8/23 13:49, Daniel P. Berrangé wrote: >>>>>>>>> On Fri, Sep 08, 2023 at 01:34:54PM +0200, Ilya Maximets wrote: >>>>>>>>>> On 9/8/23 13:19, Stefan Hajnoczi wrote: >>>>>>>>>>> Hi Ilya and Jason, >>>>>>>>>>> There is a CI failure related to a missing Debian libxdp-dev package: >>>>>>>>>>> https://gitlab.com/qemu-project/qemu/-/jobs/5046139967 >>>>>>>>>>> >>>>>>>>>>> I think the issue is that the debian-amd64 container image that QEMU >>>>>>>>>>> uses for testing is based on Debian 11 ("bullseye" aka "oldstable") >>>>>>>>>>> and libxdp is not available on that release: >>>>>>>>>>> https://packages.debian.org/search?keywords=libxdp&searchon=names&suite=oldstable§ion=all >>>>>>>>>> >>>>>>>>>> Hmm. Sorry about that. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> If we need to support Debian 11 CI then either XDP could be disabled >>>>>>>>>>> for that distro or libxdp could be compiled from source. >>>>>>>>>> >>>>>>>>>> I'd suggest we just remove the attempt to install the package for now, >>>>>>>>>> building libxdp from sources may be a little painful to maintain. >>>>>>>>>> >>>>>>>>>> Can be re-added later once distributions with libxdp 1.4+ will be more >>>>>>>>>> widely available, i.e. when fedora dockerfile will be updated to 39, >>>>>>>>>> for example. That should be soon-ish, right? >>>>>>>>> >>>>>>>>> If you follow the process in docs/devel/testing.rst for adding >>>>>>>>> libxdp in libvirt-ci, then lcitool will "do the right thing" >>>>>>>>> when we move the auto-generated dockerfiles to new distro versions. >>>>>>>> >>>>>>>> Thanks! I'll prepare changes for libvirt-ci. >>>>>>>> >>>>>>>> In the meantime, none of the currently tested images will have a required >>>>>>>> version of libxdp anyway, so I'm suggesting to just drop this one dockerfile >>>>>>>> modification from the patch. What do you think? >>>>>>> >>>>>>> Sure, if none of the distros have it, then lcitool won't emit the >>>>>>> dockerfile changes until we update the inherited distro version. >>>>>>> So it is sufficient to just update libvirt-ci.git with the mappings.yml >>>>>>> info for libxdp, and add 'libxdp' to the tests/lcitool/projects/qemu.yml >>>>>>> file in qemu.git. It will then 'just work' when someone updates the >>>>>>> distro versions later. >>>>>> >>>>>> I posted an MR for libvirt-ci adding libxdp: >>>>>> https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/429 >>>>>> >>>>>> Please, take a look. >>>>>> >>>>>> The docs say that CI will try to build containers with the MR changes, >>>>>> but I don't think anything except sanity checks is actually tested on MR. >>>>>> Sorry if I missed something, never used GitLab pipelines before. >>>>> >>>>> No, that's our fault - we've broken the CI and your change alerted >>>>> me to that fact :-) >>>>> >>>>>> Note that with this update we will be installing older version of libxdp >>>>>> in many containers, even though they will not be used by QEMU, unless >>>>>> they are newer than 1.4.0. >>>>> >>>>> No problem, as it means QEMU CI will demonstrate the the meson.build >>>>> change is ignoring the outdatd libxdp. >>>>> >>>>>> tests/lcitool/projects/qemu.yml in qemu.git cannot be updated without >>>>>> updating a submodule after the MR merge. >>>>> >>>>> Yep. >>>> >>>> Since all the required changes went into libvirt-ci project, I posted an >>>> updated patch set named: >>>> >>>> '[PATCH v4 0/2] net: add initial support for AF_XDP network backend' >>>> >>>> Please, take a look. >>>> >>>> This should fix the CI issues, though I'm not sure how to run QEMU gitlab >>>> pipelines myself, so I didn't actually test all the images. >>> >>> git push gitlab -o ci.variable=QEMU_CI=2 >>> >>> will create pipeline and immediately run all jobs. >> >> Thanks! That worked. Though I wasn't able to test much anyway as >> this thing burned through all my free compute credits less than >> half way through the pipeline. :D >> >> So, AFAIU, it's not something an occasional contributor like me can >> use, unless they are spending their own money. > > That is not the expected behaviour. > > If your repo is a fork of https://gitlab.com/qemu-project/qemu it > should benefit from a *massive* x125 reduction on CI costs. > > The critical thing is that it *MUST* have been created with the > 'Fork' button on qemu-project/qemu. Yeah, it might be that the problem is caused by me accidentally forking the gitlab.com/qemu/qemu repo instead of qemu-project. It is fairly confusing that qemu/qemu is not the main repository of QEMU project. It seems to be some sort of automated account and it closely follows updates of the main repository. It also has a better name, and it is *not a fork* of the qemu-project. There practically no indication that qemu/qemu is not a main repository, except for an icon and a lower star/fork count. It's easy to fork the wrong one. Do you folks have control over this account? Could you maybe add a description that it is not the official QEMU repository and add a link to qemu-project? Right now qemu-project/qemu states that it is a "QEMU main repository", but qemu/qemu doesn't say anything. > If that's not the case then > you will burn CI credits at a cost of 1 minute == 1 credit, > instead of 1 minute == 0.008 credits. Check this by going to > the top page of your repo, and looking for a box a little above > the file list, that says > > "Forked from QEMU / QEMU" > > If that is not the case, then you'll have to rename your existing > repo to get it out of the way, and then use the 'Fork' button to > create a new copy that is tracked as a fork. > > With most accounts getting 400 CI minutes per month, an averge > QEMU CI run should consume about 7 CI minutes. > > NB, CI credits reset on the 1st of each month I deleted my fork (there wasn't anything useful there) and re-forked the correct one. Will try again next month. For now "No more CI minutes available. You have used 788 out of 400 of your shared Runners pipeline minutes." :D Best regards, Ilya Maximets.
On Tue, Sep 19, 2023 at 11:39:31AM +0200, Ilya Maximets wrote: > On 9/19/23 10:40, Daniel P. Berrangé wrote: > > On Mon, Sep 18, 2023 at 09:36:10PM +0200, Ilya Maximets wrote: > >> Thanks! That worked. Though I wasn't able to test much anyway as > >> this thing burned through all my free compute credits less than > >> half way through the pipeline. :D > >> > >> So, AFAIU, it's not something an occasional contributor like me can > >> use, unless they are spending their own money. > > > > That is not the expected behaviour. > > > > If your repo is a fork of https://gitlab.com/qemu-project/qemu it > > should benefit from a *massive* x125 reduction on CI costs. > > > > The critical thing is that it *MUST* have been created with the > > 'Fork' button on qemu-project/qemu. > > Yeah, it might be that the problem is caused by me accidentally > forking the gitlab.com/qemu/qemu repo instead of qemu-project. > > It is fairly confusing that qemu/qemu is not the main repository > of QEMU project. It seems to be some sort of automated account > and it closely follows updates of the main repository. It also > has a better name, and it is *not a fork* of the qemu-project. > There practically no indication that qemu/qemu is not a main > repository, except for an icon and a lower star/fork count. > It's easy to fork the wrong one. > > Do you folks have control over this account? Could you maybe add > a description that it is not the official QEMU repository and add > a link to qemu-project? Right now qemu-project/qemu states that > it is a "QEMU main repository", but qemu/qemu doesn't say anything. The https://gitlab.com/qemu account is a user profile who has been squatting on that name for a while. I vaguely recall we tried to claim it with gitlab but because it regularly pushes code it isn't considered inactive and thus there's nothing we can do about it :-( With regards, Daniel
The following changes since commit 03a3a62fbd0aa5227e978eef3c67d3978aec9e5f: Merge tag 'for-upstream' of https://gitlab.com/bonzini/qemu into staging (2023-09-07 10:29:06 -0400) are available in the git repository at: https://github.com/jasowang/qemu.git tags/net-pull-request for you to fetch changes up to 049cfda145e96b2605cdf9739f1bcf9ebf3a83e1: ebpf: Updated eBPF program and skeleton. (2023-09-08 14:33:46 +0800) ---------------------------------------------------------------- ---------------------------------------------------------------- Andrew Melnychenko (7): tap: Add USO support to tap device. virtio-net: Add USO flags to vhost support. ebpf: Added eBPF map update through mmap. ebpf: Added eBPF initialization by fds. virtio-net: Added property to load eBPF RSS with fds. qmp: Added new command to retrieve eBPF blob. ebpf: Updated eBPF program and skeleton. Ilya Maximets (1): net: add initial support for AF_XDP network backend Tomasz Dzieciol (7): igb: remove TCP ACK detection igb: rename E1000E_RingInfo_st igb: RX descriptors guest writting refactoring igb: RX payload guest writting refactoring igb: add IPv6 extended headers traffic detection igb: packet-split descriptors support e1000e: rename e1000e_ba_state and e1000e_write_hdr_to_rx_buffers Yuri Benditovich (2): tap: Add check for USO features virtio-net: Add support for USO features MAINTAINERS | 4 + ebpf/ebpf.c | 70 ++ ebpf/ebpf.h | 31 + ebpf/ebpf_rss-stub.c | 6 + ebpf/ebpf_rss.c | 150 ++- ebpf/ebpf_rss.h | 10 + ebpf/meson.build | 2 +- ebpf/rss.bpf.skeleton.h | 1460 ++++++++++++----------- hmp-commands.hx | 3 + hw/core/machine.c | 4 + hw/net/e1000e_core.c | 80 +- hw/net/igb_core.c | 732 ++++++++---- hw/net/igb_regs.h | 20 +- hw/net/trace-events | 6 +- hw/net/vhost_net.c | 3 + hw/net/virtio-net.c | 90 +- hw/net/vmxnet3.c | 2 + include/hw/virtio/virtio-net.h | 1 + include/net/net.h | 7 +- meson.build | 19 +- meson_options.txt | 2 + net/af-xdp.c | 526 ++++++++ net/clients.h | 5 + net/meson.build | 3 + net/net.c | 19 +- net/tap-bsd.c | 7 +- net/tap-linux.c | 27 +- net/tap-linux.h | 2 + net/tap-solaris.c | 7 +- net/tap-stub.c | 7 +- net/tap-win32.c | 2 +- net/tap.c | 18 +- net/tap_int.h | 4 +- net/vhost-vdpa.c | 3 + qapi/ebpf.json | 66 + qapi/meson.build | 1 + qapi/net.json | 58 + qapi/qapi-schema.json | 1 + qemu-options.hx | 70 +- scripts/ci/org.centos/stream/8/x86_64/configure | 1 + scripts/meson-buildoptions.sh | 3 + tests/docker/dockerfiles/debian-amd64.docker | 1 + tests/qtest/libqos/igb.c | 5 + tools/ebpf/rss.bpf.c | 5 +- 44 files changed, 2518 insertions(+), 1025 deletions(-) create mode 100644 ebpf/ebpf.c create mode 100644 ebpf/ebpf.h create mode 100644 net/af-xdp.c create mode 100644 qapi/ebpf.json