| Message ID | 20200605121342.534811-1-imammedo@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | [v2] numa: forbid '-numa node, mem' for 5.1 and newer machine types | expand |
Patchew URL: https://patchew.org/QEMU/20200605121342.534811-1-imammedo@redhat.com/
Hi,
This series failed the asan build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
=== TEST SCRIPT END ===
CC tools/virtiofsd/helper.o
CC tools/virtiofsd/seccomp.o
Warning, treated as error:
/tmp/qemu-test/src/docs/system/deprecated.rst:507:Title underline too short.
``-numa node,mem=``\ *size* (removed in 5.1)
---
BUNZIP2 pc-bios/edk2-i386-code.fd.bz2
BUNZIP2 pc-bios/edk2-arm-vars.fd.bz2
BUNZIP2 pc-bios/edk2-aarch64-code.fd.bz2
make: *** [Makefile:1114: .docs_system_qemu.1_docs_system_qemu-block-drivers.7_docs_system_qemu-cpu-models.7.sentinel.] Error 2
make: *** Deleting file '.docs_system_qemu.1_docs_system_qemu-block-drivers.7_docs_system_qemu-cpu-models.7.sentinel.'
make: *** Waiting for unfinished jobs....
Warning, treated as error:
/tmp/qemu-test/src/docs/system/deprecated.rst:507:Title underline too short.
``-numa node,mem=``\ *size* (removed in 5.1)
'''''''''''''''''''''''''''''''''''''''
make: *** [Makefile:1103: docs/system/index.html] Error 2
Traceback (most recent call last):
File "./tests/docker/docker.py", line 665, in <module>
sys.exit(main())
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=36f1410148bd4474ae0381f8fbdf93b6', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=x86_64-softmmu', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-f_yjtmn7/src/docker-src.2020-06-05-10.06.36.18226:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-debug']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=36f1410148bd4474ae0381f8fbdf93b6
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-f_yjtmn7/src'
make: *** [docker-run-test-debug@fedora] Error 2
real 3m53.414s
user 0m7.982s
The full log is available at
http://patchew.org/logs/20200605121342.534811-1-imammedo@redhat.com/testing.asan/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Patchew URL: https://patchew.org/QEMU/20200605121342.534811-1-imammedo@redhat.com/
Hi,
This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.
=== TEST SCRIPT BEGIN ===
#! /bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===
CC hw/display/edid-generate.o
CC blockdev.o
Warning, treated as error:
/tmp/qemu-test/src/docs/system/deprecated.rst:507:Title underline too short.
``-numa node,mem=``\ *size* (removed in 5.1)
---
CC job-qmp.o
CC os-win32.o
Warning, treated as error:
/tmp/qemu-test/src/docs/system/deprecated.rst:507:Title underline too short.
``-numa node,mem=``\ *size* (removed in 5.1)
---
CC qapi/qapi-commands-crypto.o
CC qapi/qapi-commands-introspect.o
CC qapi/qapi-commands-job.o
make: *** [Makefile:1114: .docs_system_qemu.1_docs_system_qemu-block-drivers.7_docs_system_qemu-cpu-models.7.sentinel.] Error 2
make: *** Deleting file '.docs_system_qemu.1_docs_system_qemu-block-drivers.7_docs_system_qemu-cpu-models.7.sentinel.'
make: *** Waiting for unfinished jobs....
make: *** [Makefile:1103: docs/system/index.html] Error 2
Traceback (most recent call last):
File "./tests/docker/docker.py", line 665, in <module>
sys.exit(main())
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=74ef85fd49af466999d8de2438cc0177', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-mlk2hjle/src/docker-src.2020-06-05-10.11.35.25260:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=74ef85fd49af466999d8de2438cc0177
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-mlk2hjle/src'
make: *** [docker-run-test-mingw@fedora] Error 2
real 2m22.940s
user 0m8.002s
The full log is available at
http://patchew.org/logs/20200605121342.534811-1-imammedo@redhat.com/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
On 05/06/20 14:13, Igor Mammedov wrote: > Deprecation period is run out and it's a time to flip the switch > introduced by cd5ff8333a. Disable legacy option for new machine > types (since 5.1) and amend documentation. > > '-numa node,memdev' shall be used instead of disabled option > with new machine types. > > Signed-off-by: Igor Mammedov <imammedo@redhat.com> > Reviewed-by: Michal Privoznik <mprivozn@redhat.com> > --- > v1: > - rebased on top of current master > - move compat mode from 4.2 to 5.0 > v2: > - move deprection text to recently removed section > - pick up reviewed-bys > > CC: peter.maydell@linaro.org > CC: ehabkost@redhat.com > CC: marcel.apfelbaum@gmail.com > CC: mst@redhat.com > CC: pbonzini@redhat.com > CC: rth@twiddle.net > CC: david@gibson.dropbear.id.au > CC: libvir-list@redhat.com > CC: qemu-arm@nongnu.org > CC: qemu-ppc@nongnu.org > CC: eblake@redhat.com > --- > docs/system/deprecated.rst | 37 ++++++++++++++++++++----------------- > hw/arm/virt.c | 2 +- > hw/core/numa.c | 6 ++++++ > hw/i386/pc.c | 1 - > hw/i386/pc_piix.c | 1 + > hw/i386/pc_q35.c | 1 + > hw/ppc/spapr.c | 2 +- > qemu-options.hx | 9 +++++---- > 8 files changed, 35 insertions(+), 24 deletions(-) > > diff --git a/docs/system/deprecated.rst b/docs/system/deprecated.rst > index f0061f94aa..6f717e4a1d 100644 > --- a/docs/system/deprecated.rst > +++ b/docs/system/deprecated.rst > @@ -101,23 +101,6 @@ error in the future. > The ``-realtime mlock=on|off`` argument has been replaced by the > ``-overcommit mem-lock=on|off`` argument. > > -``-numa node,mem=``\ *size* (since 4.1) > -''''''''''''''''''''''''''''''''''''''' > - > -The parameter ``mem`` of ``-numa node`` is used to assign a part of > -guest RAM to a NUMA node. But when using it, it's impossible to manage specified > -RAM chunk on the host side (like bind it to a host node, setting bind policy, ...), > -so guest end-ups with the fake NUMA configuration with suboptiomal performance. > -However since 2014 there is an alternative way to assign RAM to a NUMA node > -using parameter ``memdev``, which does the same as ``mem`` and adds > -means to actualy manage node RAM on the host side. Use parameter ``memdev`` > -with *memory-backend-ram* backend as an replacement for parameter ``mem`` > -to achieve the same fake NUMA effect or a properly configured > -*memory-backend-file* backend to actually benefit from NUMA configuration. > -In future new machine versions will not accept the option but it will still > -work with old machine types. User can check QAPI schema to see if the legacy > -option is supported by looking at MachineInfo::numa-mem-supported property. > - > ``-numa`` node (without memory specified) (since 4.1) > ''''''''''''''''''''''''''''''''''''''''''''''''''''' > > @@ -512,3 +495,23 @@ long starting at 1MiB, the old command:: > can be rewritten as:: > > qemu-nbd -t --image-opts driver=raw,offset=1M,size=100M,file.driver=qcow2,file.file.driver=file,file.file.filename=file.qcow2 > + > +Command line options > +-------------------- > + > +``-numa node,mem=``\ *size* (removed in 5.1) > +''''''''''''''''''''''''''''''''''''''' > + > +The parameter ``mem`` of ``-numa node`` is used to assign a part of > +guest RAM to a NUMA node. But when using it, it's impossible to manage specified > +RAM chunk on the host side (like bind it to a host node, setting bind policy, ...), > +so guest end-ups with the fake NUMA configuration with suboptiomal performance. > +However since 2014 there is an alternative way to assign RAM to a NUMA node > +using parameter ``memdev``, which does the same as ``mem`` and adds > +means to actualy manage node RAM on the host side. Use parameter ``memdev`` > +with *memory-backend-ram* backend as an replacement for parameter ``mem`` > +to achieve the same fake NUMA effect or a properly configured > +*memory-backend-file* backend to actually benefit from NUMA configuration. > +In future new machine versions will not accept the option but it will still > +work with old machine types. User can check QAPI schema to see if the legacy > +option is supported by looking at MachineInfo::numa-mem-supported property. > diff --git a/hw/arm/virt.c b/hw/arm/virt.c > index 37462a6f78..063d4703f7 100644 > --- a/hw/arm/virt.c > +++ b/hw/arm/virt.c > @@ -2262,7 +2262,6 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) > hc->pre_plug = virt_machine_device_pre_plug_cb; > hc->plug = virt_machine_device_plug_cb; > hc->unplug_request = virt_machine_device_unplug_request_cb; > - mc->numa_mem_supported = true; > mc->nvdimm_supported = true; > mc->auto_enable_numa_with_memhp = true; > mc->default_ram_id = "mach-virt.ram"; > @@ -2375,6 +2374,7 @@ DEFINE_VIRT_MACHINE_AS_LATEST(5, 1) > static void virt_machine_5_0_options(MachineClass *mc) > { > virt_machine_5_1_options(mc); > + mc->numa_mem_supported = true; > } > DEFINE_VIRT_MACHINE(5, 0) > > diff --git a/hw/core/numa.c b/hw/core/numa.c > index 316bc50d75..05be412e59 100644 > --- a/hw/core/numa.c > +++ b/hw/core/numa.c > @@ -117,6 +117,12 @@ static void parse_numa_node(MachineState *ms, NumaNodeOptions *node, > } > > if (node->has_mem) { > + if (!mc->numa_mem_supported) { > + error_setg(errp, "Parameter -numa node,mem is not supported by this" > + " machine type. Use -numa node,memdev instead"); > + return; > + } > + > numa_info[nodenr].node_mem = node->mem; > if (!qtest_enabled()) { > warn_report("Parameter -numa node,mem is deprecated," > diff --git a/hw/i386/pc.c b/hw/i386/pc.c > index 2128f3d6fe..a86136069c 100644 > --- a/hw/i386/pc.c > +++ b/hw/i386/pc.c > @@ -1960,7 +1960,6 @@ static void pc_machine_class_init(ObjectClass *oc, void *data) > hc->unplug = pc_machine_device_unplug_cb; > mc->default_cpu_type = TARGET_DEFAULT_CPU_TYPE; > mc->nvdimm_supported = true; > - mc->numa_mem_supported = true; > mc->default_ram_id = "pc.ram"; > > object_class_property_add(oc, PC_MACHINE_DEVMEM_REGION_SIZE, "int", > diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c > index f66e1d73ce..8955436efa 100644 > --- a/hw/i386/pc_piix.c > +++ b/hw/i386/pc_piix.c > @@ -438,6 +438,7 @@ static void pc_i440fx_5_0_machine_options(MachineClass *m) > pc_i440fx_5_1_machine_options(m); > m->alias = NULL; > m->is_default = false; > + m->numa_mem_supported = true; > compat_props_add(m->compat_props, hw_compat_5_0, hw_compat_5_0_len); > compat_props_add(m->compat_props, pc_compat_5_0, pc_compat_5_0_len); > } > diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c > index 4ba8ac8774..af6b32adb7 100644 > --- a/hw/i386/pc_q35.c > +++ b/hw/i386/pc_q35.c > @@ -366,6 +366,7 @@ static void pc_q35_5_0_machine_options(MachineClass *m) > { > pc_q35_5_1_machine_options(m); > m->alias = NULL; > + m->numa_mem_supported = true; > compat_props_add(m->compat_props, hw_compat_5_0, hw_compat_5_0_len); > compat_props_add(m->compat_props, pc_compat_5_0, pc_compat_5_0_len); > } > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c > index 3b1a5ed865..210d61d1b3 100644 > --- a/hw/ppc/spapr.c > +++ b/hw/ppc/spapr.c > @@ -4511,7 +4511,6 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data) > * in which LMBs are represented and hot-added > */ > mc->numa_mem_align_shift = 28; > - mc->numa_mem_supported = true; > mc->auto_enable_numa = true; > > smc->default_caps.caps[SPAPR_CAP_HTM] = SPAPR_CAP_OFF; > @@ -4599,6 +4598,7 @@ static void spapr_machine_5_0_class_options(MachineClass *mc) > { > spapr_machine_5_1_class_options(mc); > compat_props_add(mc->compat_props, hw_compat_5_0, hw_compat_5_0_len); > + mc->numa_mem_supported = true; > } > > DEFINE_SPAPR_MACHINE(5_0, "5.0", false); > diff --git a/qemu-options.hx b/qemu-options.hx > index 93bde2bbc8..196f468786 100644 > --- a/qemu-options.hx > +++ b/qemu-options.hx > @@ -239,10 +239,11 @@ SRST > -numa node,nodeid=0 -numa node,nodeid=1 \ > -numa cpu,node-id=0,socket-id=0 -numa cpu,node-id=1,socket-id=1 > > - '\ ``mem``\ ' assigns a given RAM amount to a node. '\ ``memdev``\ ' > - assigns RAM from a given memory backend device to a node. If > - '\ ``mem``\ ' and '\ ``memdev``\ ' are omitted in all nodes, RAM is > - split equally between them. > + Legacy '\ ``mem``\ ' assigns a given RAM amount to a node (not supported > + for 5.1 and newer machine types). '\ ``memdev``\ ' assigns RAM from > + a given memory backend device to a node. If '\ ``mem``\ ' and > + '\ ``memdev``\ ' are omitted in all nodes, RAM is split equally between them. > + > > '\ ``mem``\ ' and '\ ``memdev``\ ' are mutually exclusive. > Furthermore, if one node uses '\ ``memdev``\ ', all of them have to > Queued, thanks. Paolo
diff --git a/docs/system/deprecated.rst b/docs/system/deprecated.rst index f0061f94aa..6f717e4a1d 100644 --- a/docs/system/deprecated.rst +++ b/docs/system/deprecated.rst @@ -101,23 +101,6 @@ error in the future. The ``-realtime mlock=on|off`` argument has been replaced by the ``-overcommit mem-lock=on|off`` argument. -``-numa node,mem=``\ *size* (since 4.1) -''''''''''''''''''''''''''''''''''''''' - -The parameter ``mem`` of ``-numa node`` is used to assign a part of -guest RAM to a NUMA node. But when using it, it's impossible to manage specified -RAM chunk on the host side (like bind it to a host node, setting bind policy, ...), -so guest end-ups with the fake NUMA configuration with suboptiomal performance. -However since 2014 there is an alternative way to assign RAM to a NUMA node -using parameter ``memdev``, which does the same as ``mem`` and adds -means to actualy manage node RAM on the host side. Use parameter ``memdev`` -with *memory-backend-ram* backend as an replacement for parameter ``mem`` -to achieve the same fake NUMA effect or a properly configured -*memory-backend-file* backend to actually benefit from NUMA configuration. -In future new machine versions will not accept the option but it will still -work with old machine types. User can check QAPI schema to see if the legacy -option is supported by looking at MachineInfo::numa-mem-supported property. - ``-numa`` node (without memory specified) (since 4.1) ''''''''''''''''''''''''''''''''''''''''''''''''''''' @@ -512,3 +495,23 @@ long starting at 1MiB, the old command:: can be rewritten as:: qemu-nbd -t --image-opts driver=raw,offset=1M,size=100M,file.driver=qcow2,file.file.driver=file,file.file.filename=file.qcow2 + +Command line options +-------------------- + +``-numa node,mem=``\ *size* (removed in 5.1) +''''''''''''''''''''''''''''''''''''''' + +The parameter ``mem`` of ``-numa node`` is used to assign a part of +guest RAM to a NUMA node. But when using it, it's impossible to manage specified +RAM chunk on the host side (like bind it to a host node, setting bind policy, ...), +so guest end-ups with the fake NUMA configuration with suboptiomal performance. +However since 2014 there is an alternative way to assign RAM to a NUMA node +using parameter ``memdev``, which does the same as ``mem`` and adds +means to actualy manage node RAM on the host side. Use parameter ``memdev`` +with *memory-backend-ram* backend as an replacement for parameter ``mem`` +to achieve the same fake NUMA effect or a properly configured +*memory-backend-file* backend to actually benefit from NUMA configuration. +In future new machine versions will not accept the option but it will still +work with old machine types. User can check QAPI schema to see if the legacy +option is supported by looking at MachineInfo::numa-mem-supported property. diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 37462a6f78..063d4703f7 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2262,7 +2262,6 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) hc->pre_plug = virt_machine_device_pre_plug_cb; hc->plug = virt_machine_device_plug_cb; hc->unplug_request = virt_machine_device_unplug_request_cb; - mc->numa_mem_supported = true; mc->nvdimm_supported = true; mc->auto_enable_numa_with_memhp = true; mc->default_ram_id = "mach-virt.ram"; @@ -2375,6 +2374,7 @@ DEFINE_VIRT_MACHINE_AS_LATEST(5, 1) static void virt_machine_5_0_options(MachineClass *mc) { virt_machine_5_1_options(mc); + mc->numa_mem_supported = true; } DEFINE_VIRT_MACHINE(5, 0) diff --git a/hw/core/numa.c b/hw/core/numa.c index 316bc50d75..05be412e59 100644 --- a/hw/core/numa.c +++ b/hw/core/numa.c @@ -117,6 +117,12 @@ static void parse_numa_node(MachineState *ms, NumaNodeOptions *node, } if (node->has_mem) { + if (!mc->numa_mem_supported) { + error_setg(errp, "Parameter -numa node,mem is not supported by this" + " machine type. Use -numa node,memdev instead"); + return; + } + numa_info[nodenr].node_mem = node->mem; if (!qtest_enabled()) { warn_report("Parameter -numa node,mem is deprecated," diff --git a/hw/i386/pc.c b/hw/i386/pc.c index 2128f3d6fe..a86136069c 100644 --- a/hw/i386/pc.c +++ b/hw/i386/pc.c @@ -1960,7 +1960,6 @@ static void pc_machine_class_init(ObjectClass *oc, void *data) hc->unplug = pc_machine_device_unplug_cb; mc->default_cpu_type = TARGET_DEFAULT_CPU_TYPE; mc->nvdimm_supported = true; - mc->numa_mem_supported = true; mc->default_ram_id = "pc.ram"; object_class_property_add(oc, PC_MACHINE_DEVMEM_REGION_SIZE, "int", diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c index f66e1d73ce..8955436efa 100644 --- a/hw/i386/pc_piix.c +++ b/hw/i386/pc_piix.c @@ -438,6 +438,7 @@ static void pc_i440fx_5_0_machine_options(MachineClass *m) pc_i440fx_5_1_machine_options(m); m->alias = NULL; m->is_default = false; + m->numa_mem_supported = true; compat_props_add(m->compat_props, hw_compat_5_0, hw_compat_5_0_len); compat_props_add(m->compat_props, pc_compat_5_0, pc_compat_5_0_len); } diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c index 4ba8ac8774..af6b32adb7 100644 --- a/hw/i386/pc_q35.c +++ b/hw/i386/pc_q35.c @@ -366,6 +366,7 @@ static void pc_q35_5_0_machine_options(MachineClass *m) { pc_q35_5_1_machine_options(m); m->alias = NULL; + m->numa_mem_supported = true; compat_props_add(m->compat_props, hw_compat_5_0, hw_compat_5_0_len); compat_props_add(m->compat_props, pc_compat_5_0, pc_compat_5_0_len); } diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index 3b1a5ed865..210d61d1b3 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -4511,7 +4511,6 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data) * in which LMBs are represented and hot-added */ mc->numa_mem_align_shift = 28; - mc->numa_mem_supported = true; mc->auto_enable_numa = true; smc->default_caps.caps[SPAPR_CAP_HTM] = SPAPR_CAP_OFF; @@ -4599,6 +4598,7 @@ static void spapr_machine_5_0_class_options(MachineClass *mc) { spapr_machine_5_1_class_options(mc); compat_props_add(mc->compat_props, hw_compat_5_0, hw_compat_5_0_len); + mc->numa_mem_supported = true; } DEFINE_SPAPR_MACHINE(5_0, "5.0", false); diff --git a/qemu-options.hx b/qemu-options.hx index 93bde2bbc8..196f468786 100644 --- a/qemu-options.hx +++ b/qemu-options.hx @@ -239,10 +239,11 @@ SRST -numa node,nodeid=0 -numa node,nodeid=1 \ -numa cpu,node-id=0,socket-id=0 -numa cpu,node-id=1,socket-id=1 - '\ ``mem``\ ' assigns a given RAM amount to a node. '\ ``memdev``\ ' - assigns RAM from a given memory backend device to a node. If - '\ ``mem``\ ' and '\ ``memdev``\ ' are omitted in all nodes, RAM is - split equally between them. + Legacy '\ ``mem``\ ' assigns a given RAM amount to a node (not supported + for 5.1 and newer machine types). '\ ``memdev``\ ' assigns RAM from + a given memory backend device to a node. If '\ ``mem``\ ' and + '\ ``memdev``\ ' are omitted in all nodes, RAM is split equally between them. + '\ ``mem``\ ' and '\ ``memdev``\ ' are mutually exclusive. Furthermore, if one node uses '\ ``memdev``\ ', all of them have to