| Message ID | 4388a0c540026486264dfe35efc4909743f4d5cc.1700010534.git.ehem+openwrt@m5p.com |
|---|---|
| State | Superseded, archived |
| Delegated to: | Chuanhong Guo |
| Headers | show |
| Series | Misc kernel config cleanup and small adjustments | expand |
Hi On 2023-11-14, Elliott Mitchell wrote: > Date: Thu, 30 Mar 2023 16:30:49 -0700 > > Full amd64 support isn't really appropriate for most situations > OpenWRT is deployed. Whereas x86-x32 seems extremely appropriate for > these situations. As such enable x86-x32 support. > > CONFIG_ARCH_MMAP_RND_COMPAT_BITS is required to follow along, > otherwise the kernel build breaks. > > Signed-off-by: Elliott Mitchell <ehem+openwrt@m5p.com> > --- > > I suggest OpenWRT should place some effort towards x86-x32. x86-x32 > seems a rather superior generic target for OpenWRT. Only issue is > it could be valuable to have at least minimal amd64 userland support > alongside the x86-x32 version. > > Note, this simply enables kernel support. Until userspace building > is added, this does almost nothing. What would be the reason for enabling x32? There is very little upstream buy-in for x32, when this question came up for Debian, it was rejected to be enabled for -among other reasons- concerns about the the system call ABI and its security hardening, as well as concerns about the long term ABI compatibility (the later of which probably not that relevant for OpenWrt, the former however is). I do understand that this is a pet peeve among the high-density virtualization crowd, but anywhere else, x32 as a concept is dead (and never took off in the first place). Talking about x86_64, as we have it right now, I don't see much of a reason to change the current concept of 64-bit kernel and pure64 userland (and that's far from being a minimal setup, running on the bare iron and having loaded quite bulky adblock lists): # free -m total used free shared buff/cache available Mem: 3924052 75720 3640732 3424 207600 3768960 Swap: 0 0 0 *If* (and imho, again purely my own irrelevant opinion, that is a big IF) x32 would really be deemed worthwhile for OpenWrt, it still doesn't make sense to enable this whole userspace ABI for the x86_64 kernel now, before your desired additional patches are available (and even then, you'd probably want a dedicated x32 subtarget instead of killing off the pure64 target for OpenWrt - something I'd personally be even less in favour of). What's the purpose for this patch now? OpenWrt doesn't ship anything needing the x32 (yet), the only use for it would be running x32 binaries in a non-OpenWrt x32 chroot (and there aren't that many general purpose distributions available to choose from) on top of OpenWrt. That aside, I may be confused by the config stacking order, but... If I read it correctly, you enable CONFIG_X86_X32 in target/linux/x86/config-*, while explicitly disabling it in the only subtarget config (target/linux/x86/64/config-*) where it would matter, is that really the intended way round and in line with the commit description? Regards Stefan Lippers-Hollmann
Hi On 2023-11-15, Stefan Lippers-Hollmann wrote: > On 2023-11-14, Elliott Mitchell wrote: > > Date: Thu, 30 Mar 2023 16:30:49 -0700 > > > > Full amd64 support isn't really appropriate for most situations > > OpenWRT is deployed. Whereas x86-x32 seems extremely appropriate for > > these situations. As such enable x86-x32 support. [...] > What would be the reason for enabling x32? > There is very little upstream buy-in for x32, when this question came > up for Debian, it was rejected to be enabled for -among other reasons- > concerns about the the system call ABI and its security hardening, as > well as concerns about the long term ABI compatibility (the later of > which probably not that relevant for OpenWrt, the former however is). Just to add some references to this: - Can we drop upstream Linux x32 support? https://lkml.org/lkml/2018/12/10/1145 the whole thread is interesting and doesn't display too much sympathy for x32 - information about the never merged/ defunct x32 port proposed for Debian https://wiki.debian.org/X32Port Regards Stefan Lippers-Hollmann
On Wed, Nov 15, 2023 at 05:35:11AM +0100, Stefan Lippers-Hollmann wrote: > > On 2023-11-14, Elliott Mitchell wrote: > > Date: Thu, 30 Mar 2023 16:30:49 -0700 > > > > Full amd64 support isn't really appropriate for most situations > > OpenWRT is deployed. Whereas x86-x32 seems extremely appropriate for > > these situations. As such enable x86-x32 support. > > > > CONFIG_ARCH_MMAP_RND_COMPAT_BITS is required to follow along, > > otherwise the kernel build breaks. > > > > Signed-off-by: Elliott Mitchell <ehem+openwrt@m5p.com> > > --- > > > > I suggest OpenWRT should place some effort towards x86-x32. x86-x32 > > seems a rather superior generic target for OpenWRT. Only issue is > > it could be valuable to have at least minimal amd64 userland support > > alongside the x86-x32 version. > > > > Note, this simply enables kernel support. Until userspace building > > is added, this does almost nothing. > > What would be the reason for enabling x32? > There is very little upstream buy-in for x32, when this question came > up for Debian, it was rejected to be enabled for -among other reasons- > concerns about the the system call ABI and its security hardening, as > well as concerns about the long term ABI compatibility (the later of > which probably not that relevant for OpenWrt, the former however is). What I see seems like x32 is still making progress on Debian. I note musl has a x32 target. While it still isn't a release architecture for Debian, I could see it becoming one. > I do understand that this is a pet peeve among the high-density > virtualization crowd, but anywhere else, x32 as a concept is dead (and > never took off in the first place). In my view x32 has some substantial use cases. Have you ever seen `ls`, `rm`, `grep`, `find`, or `test` needing more than 4GB of memory? Any of these needing that much memory would suggest something was wrong (or perhaps someone was trying to abuse them in an impressive way). For quite a number of shell utilities being able to use more than 4GB of memory is pointless. This is speculation, but I suspect there would be substantial value in having mixed systems where some utilities were x32 and some were amd64. > *If* (and imho, again purely my own irrelevant opinion, that is a > big IF) x32 would really be deemed worthwhile for OpenWrt, it still > doesn't make sense to enable this whole userspace ABI for the x86_64 > kernel now, before your desired additional patches are available > (and even then, you'd probably want a dedicated x32 subtarget instead > of killing off the pure64 target for OpenWrt - something I'd > personally be even less in favour of). Enabling the kernel support is always the first step. How does one test userspace if the kernel won't even load the executables? > That aside, I may be confused by the config stacking order, but... > If I read it correctly, you enable CONFIG_X86_X32 in > target/linux/x86/config-*, while explicitly disabling it in the > only subtarget config (target/linux/x86/64/config-*) where it > would matter, is that really the intended way round and in line > with the commit description? Previously x32 was explicitly disabled in x86/64/config subtarget. This removes that, then enables it in x86/config target. On Wed, Nov 15, 2023 at 05:41:54AM +0100, Stefan Lippers-Hollmann wrote: > > On 2023-11-15, Stefan Lippers-Hollmann wrote: > > What would be the reason for enabling x32? > > There is very little upstream buy-in for x32, when this question came > > up for Debian, it was rejected to be enabled for -among other reasons- > > concerns about the the system call ABI and its security hardening, as > > well as concerns about the long term ABI compatibility (the later of > > which probably not that relevant for OpenWrt, the former however is). > > Just to add some references to this: > - Can we drop upstream Linux x32 support? > https://lkml.org/lkml/2018/12/10/1145 > the whole thread is interesting and doesn't display too much > sympathy for x32 Yes, and did it get dropped? No, it didn't. At least some of the issues brought up there have been fixed. > - information about the never merged/ defunct x32 port proposed for > Debian > https://wiki.debian.org/X32Port There are pointers to many bugs which were found and fixed. There don't appear to be that many unresolved ones. Unfortunately I don't know whether it is likely to be part of the next formal release. I also note even the most extreme of the OpenWRT virtual machine suggestions doesn't suggest more than 4GB of memory. I don't know the future, but enabling kernel support is the first step.
Hi On 2023-11-14, Elliott Mitchell wrote: > On Wed, Nov 15, 2023 at 05:35:11AM +0100, Stefan Lippers-Hollmann wrote: > > On 2023-11-14, Elliott Mitchell wrote: [...] > > What would be the reason for enabling x32? > > There is very little upstream buy-in for x32, when this question came > > up for Debian, it was rejected to be enabled for -among other reasons- > > concerns about the the system call ABI and its security hardening, as > > well as concerns about the long term ABI compatibility (the later of > > which probably not that relevant for OpenWrt, the former however is). > > What I see seems like x32 is still making progress on Debian. I note > musl has a x32 target. While it still isn't a release architecture for > Debian, I could see it becoming one. I wouldn't really consider this to be very likely, if at all, it has become even less likely than it was over a decade ago. > > I do understand that this is a pet peeve among the high-density > > virtualization crowd, but anywhere else, x32 as a concept is dead (and > > never took off in the first place). > > In my view x32 has some substantial use cases. Have you ever seen `ls`, > `rm`, `grep`, `find`, or `test` needing more than 4GB of memory? Any > of these needing that much memory would suggest something was wrong > (or perhaps someone was trying to abuse them in an impressive way). For > quite a number of shell utilities being able to use more than 4GB of > memory is pointless. The counter question would be, does it hurt? Yes, the x32 proponents (mostly from the high-density virtualization crowd and, as Linus called it "extreme benchmarking" environments) will point out that pure-x32 uses slightly less disk space and slightly less RAM than pure64, but we are talking about very little in today's world. > This is speculation, but I suspect there would be substantial value in > having mixed systems where some utilities were x32 and some were amd64. Please point out what exactly those would be, especially in the context of OpenWrt. If you really want to mix x32 and amd64 binaries, you will need a full library set (binary packages - co-installable) for both x32 and x86_64 installed and in RAM - which is completely detrimental to any of the gains promised above. x32 only works, if you have (aside from the kernel itself) a pure-x32 userspace, or your losses in terms of additional disk space and RAM usage outweigh your gains by an order of magnitude. In a general purpose distribution you might ignore that by claiming that 'only' your big database management system (and its dependencies) would have to be a 64 binary (or some other -very isolated, compared to the rest of the running system- big component), but the situation for OpenWrt is different. This kind of mixed userspace would be a nightmare for complexity on the source side - and/or would require Debian-style multiarch support. Neither of which is very likely to succeed for OpenWrt (let alone for opkg). > > *If* (and imho, again purely my own irrelevant opinion, that is a > > big IF) x32 would really be deemed worthwhile for OpenWrt, it still > > doesn't make sense to enable this whole userspace ABI for the x86_64 > > kernel now, before your desired additional patches are available > > (and even then, you'd probably want a dedicated x32 subtarget instead > > of killing off the pure64 target for OpenWrt - something I'd > > personally be even less in favour of). > > Enabling the kernel support is always the first step. How does one test > userspace if the kernel won't even load the executables? In a general purpose distribution one might say that, however even there one would expect quite a lot of testing having happened before enabling a complete new/ additional ABI to userspace (duplicate to the native one), something akin to the semi-official debian-ports infrastructure, used for bootstrapping of new ports or retiring old ones that still have some form of interest. But in this case there is nothing to test in isolation, as there are no x32 binaries compatible with OpenWrt (you could only test inside a non-OpenWrt x32 chroot, but that's barely grounds for enabling x32 in OpenWrt's kernel, just in case; otherwise float emulation would be enabled for mips, to facilitate Debian chroots - spoiler, it's not). In contrast to the changes necessary (buildsystem, image generation, kernel configs, etc.) to test an OpenWrt x32 subtarget, a lazy-man's two-line patch of enabling X86_X32 && X86_X32_ABI would be trivial in comparison to anything of the rest that would be needed. Given that OpenWrt doesn't have any binary compatibility between major versions and needs to get rebuilt in total for each new revision, there is no need to stage- and phase in additions like this, add the new sub-target in its entirety (or not). [...] > On Wed, Nov 15, 2023 at 05:41:54AM +0100, Stefan Lippers-Hollmann wrote: > > On 2023-11-15, Stefan Lippers-Hollmann wrote: > > > What would be the reason for enabling x32? > > > There is very little upstream buy-in for x32, when this question came > > > up for Debian, it was rejected to be enabled for -among other reasons- > > > concerns about the the system call ABI and its security hardening, as > > > well as concerns about the long term ABI compatibility (the later of > > > which probably not that relevant for OpenWrt, the former however is). > > > > Just to add some references to this: > > - Can we drop upstream Linux x32 support? > > https://lkml.org/lkml/2018/12/10/1145 > > the whole thread is interesting and doesn't display too much > > sympathy for x32 > > Yes, and did it get dropped? No, it didn't. At least some of the issues > brought up there have been fixed. It didn't get any more traction either, rather to the contrary (disk- and RAM are growing in modern hardware, what would be the real gains here). > > - information about the never merged/ defunct x32 port proposed for > > Debian > > https://wiki.debian.org/X32Port > > There are pointers to many bugs which were found and fixed. There don't > appear to be that many unresolved ones. Unfortunately I don't know > whether it is likely to be part of the next formal release. > I also note even the most extreme of the OpenWRT virtual machine > suggestions doesn't suggest more than 4GB of memory. and as show cased, a pure64 OpenWrt can get away with ~16 MB disk space (using a sparse image, rather than the ~121 MB raw disk image) and easily gets away with ~96 MB RAM with its default package set - how much do you think x32 would improve that (we're talking about x86_64 hardware here, not legacy i386 systems)? Is it worth the effort depicted above? (and yes, I've tested both with 23.05.0 on qemu; 12 MB qcow2 sparse image, 96 MB RAM (64 MB throws OOM)). Even the runtime performance improvements should be negligible, all the routing happens in kernelspace (for the most part), which would remain x86_64 - any speed improvements on the userspace would be (apart from being minimal in isolation) completely irrelevant to routing performance. dnsmasq, ls, cp, rm and friends being 5% faster or slower doesn't really matter (and around 5% is the most even its proponents claim). > I don't know the future, but enabling kernel support is the first step. If you wanted to add it to Debian (with working a multi-arch implementation and organically grown repositories (which aren't rebuild aside from sourceful uploads or targeted binNMUs), you would be right - for OpenWrt, no, there really is no need to enable it, before you have the rest of the x32 subtarget ready to be merged (and I can't imagine any reason to enable it for the current OpenWrt x86_64 subtarget at all). For OpenWrt, I could imagine only two approaches to this: - make the x86_64 subtarget x32-only - add a new x32-only subtarget and leave x86_64 as it is Neither would really be 'sensible', but the only workable approach would imho be the later, even if the intention was (I hope not to witness that) to kill off the old x86_64 target. Yes, as laid out before, there isn't any strict need for OpenWrt to remain binary compatible between releases, however this would seriously mess up base vs. ports for (at least) a few weeks (broken packages, package relations, implicit expectations by build recipes throughout the tree) - and anyone doing this in one step (archive wide), without prior testing, would be insane, so the later step of adding a new subtarget in parallel to the old one would be necessary for multiple months (if not at least a stable release cycle shipping both) either way. So I really don't see any reason to enable x32 for the x86_64 subtarget, there's nothing to be gained, just major disadvantages. While I'm not a proponent for a pure-x32 subtarget either (at all), this would be the only workable approach to introduce it. Regards Stefan Lippers-Hollmann
On Wed, Nov 15, 2023 at 10:36:21AM +0100, Stefan Lippers-Hollmann wrote: > > On 2023-11-14, Elliott Mitchell wrote: > > > I don't know the future, but enabling kernel support is the first step. > > If you wanted to add it to Debian (with working a multi-arch > implementation and organically grown repositories (which aren't > rebuild aside from sourceful uploads or targeted binNMUs), you would > be right - for OpenWrt, no, there really is no need to enable it, > before you have the rest of the x32 subtarget ready to be merged (and > I can't imagine any reason to enable it for the current OpenWrt x86_64 > subtarget at all). So you would be against enabling the kernel support before userspace is ready? Meanwhile everyone else would be against working on userspace without it being enabled in the kernel... That doesn't really work too well. > For OpenWrt, I could imagine only two approaches to this: > - make the x86_64 subtarget x32-only > - add a new x32-only subtarget and leave x86_64 as it is > Neither would really be 'sensible', but the only workable approach > would imho be the later, even if the intention was (I hope not to > witness that) to kill off the old x86_64 target. I agree with this. One comment though, I suspect OpenWRT VM installations will remain below 4GB of memory for quite some time. As such x32 would be a "free" performance boost/memory reduction if upstream sources support x32 (yeah, this is a big if). > So I really don't see any reason to enable x32 for the x86_64 subtarget, > there's nothing to be gained, just major disadvantages. > While I'm not a proponent for a pure-x32 subtarget either (at all), this > would be the only workable approach to introduce it. At a glance it looks like some useful gains for not too much cost. From my perspective though this isn't a crucial part of my goal. So I'm likely to drop further discussion as this the cost:benefit isn't there.
diff --git a/target/linux/x86/64/config-5.15 b/target/linux/x86/64/config-5.15 index 1db3180c84..ea0a51f251 100644 --- a/target/linux/x86/64/config-5.15 +++ b/target/linux/x86/64/config-5.15 @@ -502,7 +502,6 @@ CONFIG_X86_PM_TIMER=y # CONFIG_X86_POWERNOW_K8 is not set # CONFIG_X86_VSYSCALL_EMULATION is not set CONFIG_X86_X2APIC=y -# CONFIG_X86_X32 is not set CONFIG_XEN=y CONFIG_XENFS=y CONFIG_XEN_512GB=y diff --git a/target/linux/x86/64/config-6.1 b/target/linux/x86/64/config-6.1 index 6609f555ba..56ec0c56fd 100644 --- a/target/linux/x86/64/config-6.1 +++ b/target/linux/x86/64/config-6.1 @@ -557,8 +557,6 @@ CONFIG_X86_PM_TIMER=y # CONFIG_X86_POWERNOW_K8 is not set # CONFIG_X86_VSYSCALL_EMULATION is not set CONFIG_X86_X2APIC=y -# CONFIG_X86_X32 is not set -# CONFIG_X86_X32_ABI is not set CONFIG_XEN=y CONFIG_XENFS=y CONFIG_XEN_512GB=y diff --git a/target/linux/x86/config-5.15 b/target/linux/x86/config-5.15 index 71f06ad470..e93e46414a 100644 --- a/target/linux/x86/config-5.15 +++ b/target/linux/x86/config-5.15 @@ -13,6 +13,7 @@ CONFIG_ARCH_MAY_HAVE_PC_FDC=y CONFIG_ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE=y CONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=y CONFIG_ARCH_MIGHT_HAVE_PC_SERIO=y +CONFIG_ARCH_MMAP_RND_COMPAT_BITS=8 CONFIG_ARCH_NR_GPIO=512 CONFIG_ARCH_RANDOM=y CONFIG_ARCH_SELECT_MEMORY_MODEL=y @@ -429,6 +430,7 @@ CONFIG_X86_UP_IOAPIC=y CONFIG_X86_USE_PPRO_CHECKSUM=y CONFIG_X86_VERBOSE_BOOTUP=y CONFIG_X86_VMX_FEATURE_NAMES=y +CONFIG_X86_X32=y CONFIG_XZ_DEC_BCJ=y CONFIG_XZ_DEC_X86=y CONFIG_ZLIB_INFLATE=y diff --git a/target/linux/x86/config-6.1 b/target/linux/x86/config-6.1 index d6f8f6180b..a7affa9e56 100644 --- a/target/linux/x86/config-6.1 +++ b/target/linux/x86/config-6.1 @@ -14,6 +14,7 @@ CONFIG_ARCH_MAY_HAVE_PC_FDC=y CONFIG_ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE=y CONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=y CONFIG_ARCH_MIGHT_HAVE_PC_SERIO=y +CONFIG_ARCH_MMAP_RND_COMPAT_BITS=8 CONFIG_ARCH_NR_GPIO=512 CONFIG_ARCH_SELECT_MEMORY_MODEL=y CONFIG_ARCH_SPARSEMEM_ENABLE=y @@ -451,6 +452,8 @@ CONFIG_X86_UP_IOAPIC=y CONFIG_X86_USE_PPRO_CHECKSUM=y CONFIG_X86_VERBOSE_BOOTUP=y CONFIG_X86_VMX_FEATURE_NAMES=y +CONFIG_X86_X32=y +CONFIG_X86_X32_ABI=y CONFIG_XZ_DEC_BCJ=y CONFIG_XZ_DEC_X86=y CONFIG_ZLIB_INFLATE=y
Date: Thu, 30 Mar 2023 16:30:49 -0700 Full amd64 support isn't really appropriate for most situations OpenWRT is deployed. Whereas x86-x32 seems extremely appropriate for these situations. As such enable x86-x32 support. CONFIG_ARCH_MMAP_RND_COMPAT_BITS is required to follow along, otherwise the kernel build breaks. Signed-off-by: Elliott Mitchell <ehem+openwrt@m5p.com> --- I suggest OpenWRT should place some effort towards x86-x32. x86-x32 seems a rather superior generic target for OpenWRT. Only issue is it could be valuable to have at least minimal amd64 userland support alongside the x86-x32 version. Note, this simply enables kernel support. Until userspace building is added, this does almost nothing. --- target/linux/x86/64/config-5.15 | 1 - target/linux/x86/64/config-6.1 | 2 -- target/linux/x86/config-5.15 | 2 ++ target/linux/x86/config-6.1 | 3 +++ 4 files changed, 5 insertions(+), 3 deletions(-)