| Message ID | 1567231103-13237-3-git-send-email-linyunsheng@huawei.com |
|---|---|
| State | Not Applicable |
| Delegated to: | David Miller |
| Headers | show |
| Series | check the node id consistently across different arches | expand |
On Sat, Aug 31, 2019 at 01:58:16PM +0800, Yunsheng Lin wrote: > According to Section 6.2.14 from ACPI spec 6.3 [1], the setting > of proximity domain is optional, as below: > > This optional object is used to describe proximity domain > associations within a machine. _PXM evaluates to an integer > that identifies a device as belonging to a Proximity Domain > defined in the System Resource Affinity Table (SRAT). That's just words.. what does it actually mean? > This patch checks node id with the below case before returning > node_to_cpumask_map[node]: > 1. if node_id >= nr_node_ids, return cpu_none_mask > 2. if node_id < 0, return cpu_online_mask > 3. if node_to_cpumask_map[node_id] is NULL, return cpu_online_mask > > [1] https://uefi.org/sites/default/files/resources/ACPI_6_3_final_Jan30.pdf > > Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com> > --- > arch/x86/include/asm/topology.h | 6 ++++++ > arch/x86/mm/numa.c | 2 +- > 2 files changed, 7 insertions(+), 1 deletion(-) > > diff --git a/arch/x86/include/asm/topology.h b/arch/x86/include/asm/topology.h > index 4b14d23..f36e9c8 100644 > --- a/arch/x86/include/asm/topology.h > +++ b/arch/x86/include/asm/topology.h > @@ -69,6 +69,12 @@ extern const struct cpumask *cpumask_of_node(int node); > /* Returns a pointer to the cpumask of CPUs on Node 'node'. */ > static inline const struct cpumask *cpumask_of_node(int node) > { > + if (node >= nr_node_ids) > + return cpu_none_mask; > + > + if (node < 0 || !node_to_cpumask_map[node]) > + return cpu_online_mask; > + > return node_to_cpumask_map[node]; > } > #endif I _reallly_ hate this. Users are expected to use valid numa ids. Now we're adding all this checking to all users. Why do we want to do that? Using '(unsigned)node >= nr_nods_ids' is an error. > diff --git a/arch/x86/mm/numa.c b/arch/x86/mm/numa.c > index e6dad60..5e393d2 100644 > --- a/arch/x86/mm/numa.c > +++ b/arch/x86/mm/numa.c > @@ -868,7 +868,7 @@ const struct cpumask *cpumask_of_node(int node) > dump_stack(); > return cpu_none_mask; > } > - if (node_to_cpumask_map[node] == NULL) { > + if (node < 0 || !node_to_cpumask_map[node]) { > printk(KERN_WARNING > "cpumask_of_node(%d): no node_to_cpumask_map!\n", > node); > -- > 2.8.1 >
On 2019/8/31 16:55, Peter Zijlstra wrote: > On Sat, Aug 31, 2019 at 01:58:16PM +0800, Yunsheng Lin wrote: >> According to Section 6.2.14 from ACPI spec 6.3 [1], the setting >> of proximity domain is optional, as below: >> >> This optional object is used to describe proximity domain >> associations within a machine. _PXM evaluates to an integer >> that identifies a device as belonging to a Proximity Domain >> defined in the System Resource Affinity Table (SRAT). > > That's just words.. what does it actually mean? It means the dev_to_node(dev) may return -1 if the bios does not implement the proximity domain feature, user may use that value to call cpumask_of_node and cpumask_of_node does not protect itself from node id being -1, which causes out of bound access. > >> This patch checks node id with the below case before returning >> node_to_cpumask_map[node]: >> 1. if node_id >= nr_node_ids, return cpu_none_mask >> 2. if node_id < 0, return cpu_online_mask >> 3. if node_to_cpumask_map[node_id] is NULL, return cpu_online_mask >> >> [1] https://uefi.org/sites/default/files/resources/ACPI_6_3_final_Jan30.pdf >> >> Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com> >> --- >> arch/x86/include/asm/topology.h | 6 ++++++ >> arch/x86/mm/numa.c | 2 +- >> 2 files changed, 7 insertions(+), 1 deletion(-) >> >> diff --git a/arch/x86/include/asm/topology.h b/arch/x86/include/asm/topology.h >> index 4b14d23..f36e9c8 100644 >> --- a/arch/x86/include/asm/topology.h >> +++ b/arch/x86/include/asm/topology.h >> @@ -69,6 +69,12 @@ extern const struct cpumask *cpumask_of_node(int node); >> /* Returns a pointer to the cpumask of CPUs on Node 'node'. */ >> static inline const struct cpumask *cpumask_of_node(int node) >> { >> + if (node >= nr_node_ids) >> + return cpu_none_mask; >> + >> + if (node < 0 || !node_to_cpumask_map[node]) >> + return cpu_online_mask; >> + >> return node_to_cpumask_map[node]; >> } >> #endif > > I _reallly_ hate this. Users are expected to use valid numa ids. Now > we're adding all this checking to all users. Why do we want to do that? As above, the dev_to_node(dev) may return -1. > > Using '(unsigned)node >= nr_nods_ids' is an error. 'node >= nr_node_ids' can be dropped if all user is expected to not call cpumask_of_node with node id greater or equal to nr_nods_ids. From what I can see, the problem can be fixed in three place: 1. Make user dev_to_node return a valid node id even when proximity domain is not set by bios(or node id set by buggy bios is not valid), which may need info from the numa system to make sure it will return a valid node. 2. User that call cpumask_of_node should ensure the node id is valid before calling cpumask_of_node, and user also need some info to make ensure node id is valid. 3. Make sure cpumask_of_node deal with invalid node id as this patchset. Which one do you prefer to make sure node id is valid, or do you have any better idea? Any detail advice and suggestion will be very helpful, thanks. > >> diff --git a/arch/x86/mm/numa.c b/arch/x86/mm/numa.c >> index e6dad60..5e393d2 100644 >> --- a/arch/x86/mm/numa.c >> +++ b/arch/x86/mm/numa.c >> @@ -868,7 +868,7 @@ const struct cpumask *cpumask_of_node(int node) >> dump_stack(); >> return cpu_none_mask; >> } >> - if (node_to_cpumask_map[node] == NULL) { >> + if (node < 0 || !node_to_cpumask_map[node]) { >> printk(KERN_WARNING >> "cpumask_of_node(%d): no node_to_cpumask_map!\n", >> node); >> -- >> 2.8.1 >> > > . >
On Sat, Aug 31, 2019 at 06:09:39PM +0800, Yunsheng Lin wrote: > > > On 2019/8/31 16:55, Peter Zijlstra wrote: > > On Sat, Aug 31, 2019 at 01:58:16PM +0800, Yunsheng Lin wrote: > >> According to Section 6.2.14 from ACPI spec 6.3 [1], the setting > >> of proximity domain is optional, as below: > >> > >> This optional object is used to describe proximity domain > >> associations within a machine. _PXM evaluates to an integer > >> that identifies a device as belonging to a Proximity Domain > >> defined in the System Resource Affinity Table (SRAT). > > > > That's just words.. what does it actually mean? > > It means the dev_to_node(dev) may return -1 if the bios does not > implement the proximity domain feature, user may use that value > to call cpumask_of_node and cpumask_of_node does not protect itself > from node id being -1, which causes out of bound access. > >> @@ -69,6 +69,12 @@ extern const struct cpumask *cpumask_of_node(int node); > >> /* Returns a pointer to the cpumask of CPUs on Node 'node'. */ > >> static inline const struct cpumask *cpumask_of_node(int node) > >> { > >> + if (node >= nr_node_ids) > >> + return cpu_none_mask; > >> + > >> + if (node < 0 || !node_to_cpumask_map[node]) > >> + return cpu_online_mask; > >> + > >> return node_to_cpumask_map[node]; > >> } > >> #endif > > > > I _reallly_ hate this. Users are expected to use valid numa ids. Now > > we're adding all this checking to all users. Why do we want to do that? > > As above, the dev_to_node(dev) may return -1. > > > > > Using '(unsigned)node >= nr_nods_ids' is an error. > > 'node >= nr_node_ids' can be dropped if all user is expected to not call > cpumask_of_node with node id greater or equal to nr_nods_ids. you copied my typo :-) > From what I can see, the problem can be fixed in three place: > 1. Make user dev_to_node return a valid node id even when proximity > domain is not set by bios(or node id set by buggy bios is not valid), > which may need info from the numa system to make sure it will return > a valid node. > > 2. User that call cpumask_of_node should ensure the node id is valid > before calling cpumask_of_node, and user also need some info to > make ensure node id is valid. > > 3. Make sure cpumask_of_node deal with invalid node id as this patchset. > > Which one do you prefer to make sure node id is valid, or do you > have any better idea? > > Any detail advice and suggestion will be very helpful, thanks. 1) because even it is not set, the device really does belong to a node. It is impossible a device will have magic uniform access to memory when CPUs cannot. 2) is already true today, cpumask_of_node() requires a valid node_id. 3) is just wrong and increases overhead for everyone.
On 2019/9/1 0:12, Peter Zijlstra wrote: > On Sat, Aug 31, 2019 at 06:09:39PM +0800, Yunsheng Lin wrote: >> >> >> On 2019/8/31 16:55, Peter Zijlstra wrote: >>> On Sat, Aug 31, 2019 at 01:58:16PM +0800, Yunsheng Lin wrote: >>>> According to Section 6.2.14 from ACPI spec 6.3 [1], the setting >>>> of proximity domain is optional, as below: >>>> >>>> This optional object is used to describe proximity domain >>>> associations within a machine. _PXM evaluates to an integer >>>> that identifies a device as belonging to a Proximity Domain >>>> defined in the System Resource Affinity Table (SRAT). >>> >>> That's just words.. what does it actually mean? >> >> It means the dev_to_node(dev) may return -1 if the bios does not >> implement the proximity domain feature, user may use that value >> to call cpumask_of_node and cpumask_of_node does not protect itself >> from node id being -1, which causes out of bound access. > >>>> @@ -69,6 +69,12 @@ extern const struct cpumask *cpumask_of_node(int node); >>>> /* Returns a pointer to the cpumask of CPUs on Node 'node'. */ >>>> static inline const struct cpumask *cpumask_of_node(int node) >>>> { >>>> + if (node >= nr_node_ids) >>>> + return cpu_none_mask; >>>> + >>>> + if (node < 0 || !node_to_cpumask_map[node]) >>>> + return cpu_online_mask; >>>> + >>>> return node_to_cpumask_map[node]; >>>> } >>>> #endif >>> >>> I _reallly_ hate this. Users are expected to use valid numa ids. Now >>> we're adding all this checking to all users. Why do we want to do that? >> >> As above, the dev_to_node(dev) may return -1. >> >>> >>> Using '(unsigned)node >= nr_nods_ids' is an error. >> >> 'node >= nr_node_ids' can be dropped if all user is expected to not call >> cpumask_of_node with node id greater or equal to nr_nods_ids. > > you copied my typo :-) I did note the typo, corrected the first one, but missed the second one :) > >> From what I can see, the problem can be fixed in three place: >> 1. Make user dev_to_node return a valid node id even when proximity >> domain is not set by bios(or node id set by buggy bios is not valid), >> which may need info from the numa system to make sure it will return >> a valid node. >> >> 2. User that call cpumask_of_node should ensure the node id is valid >> before calling cpumask_of_node, and user also need some info to >> make ensure node id is valid. >> >> 3. Make sure cpumask_of_node deal with invalid node id as this patchset. >> >> Which one do you prefer to make sure node id is valid, or do you >> have any better idea? >> >> Any detail advice and suggestion will be very helpful, thanks. > > 1) because even it is not set, the device really does belong to a node. > It is impossible a device will have magic uniform access to memory when > CPUs cannot. So it means dev_to_node() will return either NUMA_NO_NODE or a valid node id? > > 2) is already true today, cpumask_of_node() requires a valid node_id. Ok, most of the user does check node_id before calling cpumask_of_node(), but does a little different type of checking: 1) some does " < 0" check; 2) some does "== NUMA_NO_NODE" check; 3) some does ">= MAX_NUMNODES" check; 4) some does "< 0 || >= MAX_NUMNODES || !node_online(node)" check. > > 3) is just wrong and increases overhead for everyone. Ok, cpumask_of_node() is also used in some critical path such as scheduling, which may not need those checking, the overhead is unnecessary. But for non-critical path such as setup or configuration path, it better to have consistent checking, and also simplify the user code that calls cpumask_of_node(). Do you think it is worth the trouble to add a new function such as cpumask_of_node_check(maybe some other name) to do consistent checking? Or caller just simply check if dev_to_node()'s return value is NUMA_NO_NODE before calling cpumask_of_node()? > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel > > . >
On Mon, Sep 02, 2019 at 01:46:51PM +0800, Yunsheng Lin wrote: > On 2019/9/1 0:12, Peter Zijlstra wrote: > > 1) because even it is not set, the device really does belong to a node. > > It is impossible a device will have magic uniform access to memory when > > CPUs cannot. > > So it means dev_to_node() will return either NUMA_NO_NODE or a > valid node id? NUMA_NO_NODE := -1, which is not a valid node number. It is also, like I said, not a valid device location on a NUMA system. Just because ACPI/BIOS is shit, doesn't mean the device doesn't have a node association. It just means we don't know and might have to guess. > > 2) is already true today, cpumask_of_node() requires a valid node_id. > > Ok, most of the user does check node_id before calling > cpumask_of_node(), but does a little different type of checking: > > 1) some does " < 0" check; > 2) some does "== NUMA_NO_NODE" check; > 3) some does ">= MAX_NUMNODES" check; > 4) some does "< 0 || >= MAX_NUMNODES || !node_online(node)" check. The one true way is: '(unsigned)node_id >= nr_node_ids' > > 3) is just wrong and increases overhead for everyone. > > Ok, cpumask_of_node() is also used in some critical path such > as scheduling, which may not need those checking, the overhead > is unnecessary. > > But for non-critical path such as setup or configuration path, > it better to have consistent checking, and also simplify the > user code that calls cpumask_of_node(). > > Do you think it is worth the trouble to add a new function > such as cpumask_of_node_check(maybe some other name) to do > consistent checking? > > Or caller just simply check if dev_to_node()'s return value is > NUMA_NO_NODE before calling cpumask_of_node()? It is not a matter of convenience. The function is called cpumask_of_node(), when node < 0 || node >= nr_node_ids, it is not a valid node, therefore the function shouldn't return anything except an error. Also note that the CONFIG_DEBUG_PER_CPU_MAPS version of cpumask_of_node() already does this (although it wants the below fix). --- diff --git a/arch/x86/mm/numa.c b/arch/x86/mm/numa.c index e6dad600614c..5f49c10201c7 100644 --- a/arch/x86/mm/numa.c +++ b/arch/x86/mm/numa.c @@ -861,7 +861,7 @@ void numa_remove_cpu(int cpu) */ const struct cpumask *cpumask_of_node(int node) { - if (node >= nr_node_ids) { + if ((unsigned)node >= nr_node_ids) { printk(KERN_WARNING "cpumask_of_node(%d): node > nr_node_ids(%u)\n", node, nr_node_ids);
On 2019/9/2 15:25, Peter Zijlstra wrote: > On Mon, Sep 02, 2019 at 01:46:51PM +0800, Yunsheng Lin wrote: >> On 2019/9/1 0:12, Peter Zijlstra wrote: > >>> 1) because even it is not set, the device really does belong to a node. >>> It is impossible a device will have magic uniform access to memory when >>> CPUs cannot. >> >> So it means dev_to_node() will return either NUMA_NO_NODE or a >> valid node id? > > NUMA_NO_NODE := -1, which is not a valid node number. It is also, like I > said, not a valid device location on a NUMA system. > > Just because ACPI/BIOS is shit, doesn't mean the device doesn't have a > node association. It just means we don't know and might have to guess. How do we guess the device's location when ACPI/BIOS does not set it? It seems dev_to_node() does not do anything about that and leave the job to the caller or whatever function that get called with its return value, such as cpumask_of_node(). > >>> 2) is already true today, cpumask_of_node() requires a valid node_id. >> >> Ok, most of the user does check node_id before calling >> cpumask_of_node(), but does a little different type of checking: >> >> 1) some does " < 0" check; >> 2) some does "== NUMA_NO_NODE" check; >> 3) some does ">= MAX_NUMNODES" check; >> 4) some does "< 0 || >= MAX_NUMNODES || !node_online(node)" check. > > The one true way is: > > '(unsigned)node_id >= nr_node_ids' I missed the magic of the "unsigned" in your previous reply. > >>> 3) is just wrong and increases overhead for everyone. >> >> Ok, cpumask_of_node() is also used in some critical path such >> as scheduling, which may not need those checking, the overhead >> is unnecessary. >> >> But for non-critical path such as setup or configuration path, >> it better to have consistent checking, and also simplify the >> user code that calls cpumask_of_node(). >> >> Do you think it is worth the trouble to add a new function >> such as cpumask_of_node_check(maybe some other name) to do >> consistent checking? >> >> Or caller just simply check if dev_to_node()'s return value is >> NUMA_NO_NODE before calling cpumask_of_node()? > > It is not a matter of convenience. The function is called > cpumask_of_node(), when node < 0 || node >= nr_node_ids, it is not a > valid node, therefore the function shouldn't return anything except an > error. what do you mean by error? What I can think is three type of errors: 1) return NULL, this way it seems cpumask_of_node() also leave the job to the function that calls it. 2) cpu_none_mask, I am not sure what this means, maybe it means there is no cpu on the same node with the device? 3) give a warning, stack dump, or even a BUG_ON? I would prefer the second one, and implement the third one when the CONFIG_DEBUG_PER_CPU_MAPS is selected. Any suggestion? > > Also note that the CONFIG_DEBUG_PER_CPU_MAPS version of > cpumask_of_node() already does this (although it wants the below fix). Thanks for the note and example.
On Mon, Sep 02, 2019 at 08:25:24PM +0800, Yunsheng Lin wrote: > On 2019/9/2 15:25, Peter Zijlstra wrote: > > On Mon, Sep 02, 2019 at 01:46:51PM +0800, Yunsheng Lin wrote: > >> On 2019/9/1 0:12, Peter Zijlstra wrote: > > > >>> 1) because even it is not set, the device really does belong to a node. > >>> It is impossible a device will have magic uniform access to memory when > >>> CPUs cannot. > >> > >> So it means dev_to_node() will return either NUMA_NO_NODE or a > >> valid node id? > > > > NUMA_NO_NODE := -1, which is not a valid node number. It is also, like I > > said, not a valid device location on a NUMA system. > > > > Just because ACPI/BIOS is shit, doesn't mean the device doesn't have a > > node association. It just means we don't know and might have to guess. > > How do we guess the device's location when ACPI/BIOS does not set it? See device_add(), it looks to the device's parent and on NO_NODE, puts it there. Lacking any hints, just stick it to node0 and print a FW_BUG or something. > It seems dev_to_node() does not do anything about that and leave the > job to the caller or whatever function that get called with its return > value, such as cpumask_of_node(). Well, dev_to_node() doesn't do anything; nor should it. It are the callers of set_dev_node() that should be taking care. Also note how device_add() sets the device node to the parent device's node on NUMA_NO_NODE. Arguably we should change it to complain when it finds NUMA_NO_NODE and !parent. --- drivers/base/core.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/drivers/base/core.c b/drivers/base/core.c index f0dd8e38fee3..2caf204966a0 100644 --- a/drivers/base/core.c +++ b/drivers/base/core.c @@ -2120,8 +2120,16 @@ int device_add(struct device *dev) dev->kobj.parent = kobj; /* use parent numa_node */ - if (parent && (dev_to_node(dev) == NUMA_NO_NODE)) - set_dev_node(dev, dev_to_node(parent)); + if (dev_to_node(dev) == NUMA_NO_NODE) { + if (parent) + set_dev_node(dev, dev_to_node(parent)); +#ifdef CONFIG_NUMA + else { + pr_err("device: '%s': has no assigned NUMA node\n", dev_name(dev)); + set_dev_node(dev, 0); + } +#endif + } /* first, register with generic layer. */ /* we require the name to be set before, and pass NULL */
* Peter Zijlstra <peterz@infradead.org> wrote: > Also note that the CONFIG_DEBUG_PER_CPU_MAPS version of > cpumask_of_node() already does this (although it wants the below fix). > > --- > diff --git a/arch/x86/mm/numa.c b/arch/x86/mm/numa.c > index e6dad600614c..5f49c10201c7 100644 > --- a/arch/x86/mm/numa.c > +++ b/arch/x86/mm/numa.c > @@ -861,7 +861,7 @@ void numa_remove_cpu(int cpu) > */ > const struct cpumask *cpumask_of_node(int node) > { > - if (node >= nr_node_ids) { > + if ((unsigned)node >= nr_node_ids) { > printk(KERN_WARNING > "cpumask_of_node(%d): node > nr_node_ids(%u)\n", > node, nr_node_ids); Nitpicking: please also fix the kernel message to say ">=". With that: Acked-by: Ingo Molnar <mingo@kernel.org> Note that: - arch/arm64/mm/numa.c copied the same sign bug (or unrobustness if we don't want to call it a bug). - Kudos to the mm/memcontrol.c and kernel/bpf/syscall.c teams for not copying that bug. Booo for none of them fixing the buggy pattern elsewhere in the kernel ;-) Thanks, Ingo
* Peter Zijlstra <peterz@infradead.org> wrote: > On Mon, Sep 02, 2019 at 08:25:24PM +0800, Yunsheng Lin wrote: > > On 2019/9/2 15:25, Peter Zijlstra wrote: > > > On Mon, Sep 02, 2019 at 01:46:51PM +0800, Yunsheng Lin wrote: > > >> On 2019/9/1 0:12, Peter Zijlstra wrote: > > > > > >>> 1) because even it is not set, the device really does belong to a node. > > >>> It is impossible a device will have magic uniform access to memory when > > >>> CPUs cannot. > > >> > > >> So it means dev_to_node() will return either NUMA_NO_NODE or a > > >> valid node id? > > > > > > NUMA_NO_NODE := -1, which is not a valid node number. It is also, like I > > > said, not a valid device location on a NUMA system. > > > > > > Just because ACPI/BIOS is shit, doesn't mean the device doesn't have a > > > node association. It just means we don't know and might have to guess. > > > > How do we guess the device's location when ACPI/BIOS does not set it? > > See device_add(), it looks to the device's parent and on NO_NODE, puts > it there. > > Lacking any hints, just stick it to node0 and print a FW_BUG or > something. > > > It seems dev_to_node() does not do anything about that and leave the > > job to the caller or whatever function that get called with its return > > value, such as cpumask_of_node(). > > Well, dev_to_node() doesn't do anything; nor should it. It are the > callers of set_dev_node() that should be taking care. > > Also note how device_add() sets the device node to the parent device's > node on NUMA_NO_NODE. Arguably we should change it to complain when it > finds NUMA_NO_NODE and !parent. > > --- > drivers/base/core.c | 12 ++++++++++-- > 1 file changed, 10 insertions(+), 2 deletions(-) > > diff --git a/drivers/base/core.c b/drivers/base/core.c > index f0dd8e38fee3..2caf204966a0 100644 > --- a/drivers/base/core.c > +++ b/drivers/base/core.c > @@ -2120,8 +2120,16 @@ int device_add(struct device *dev) > dev->kobj.parent = kobj; > > /* use parent numa_node */ > - if (parent && (dev_to_node(dev) == NUMA_NO_NODE)) > - set_dev_node(dev, dev_to_node(parent)); > + if (dev_to_node(dev) == NUMA_NO_NODE) { > + if (parent) > + set_dev_node(dev, dev_to_node(parent)); > +#ifdef CONFIG_NUMA > + else { > + pr_err("device: '%s': has no assigned NUMA node\n", dev_name(dev)); > + set_dev_node(dev, 0); > + } > +#endif BTW., is firmware required to always provide a NUMA node on NUMA systems? I.e. do we really want this warning on non-NUMA systems that don't assign NUMA nodes? Also, even on NUMA systems, is firmware required to provide a NUMA node - i.e. is it in principle invalid to offer no NUMA binding? Thanks, Ingo
On Mon, Sep 02, 2019 at 08:22:52PM +0200, Ingo Molnar wrote: > > * Peter Zijlstra <peterz@infradead.org> wrote: > > diff --git a/drivers/base/core.c b/drivers/base/core.c > > index f0dd8e38fee3..2caf204966a0 100644 > > --- a/drivers/base/core.c > > +++ b/drivers/base/core.c > > @@ -2120,8 +2120,16 @@ int device_add(struct device *dev) > > dev->kobj.parent = kobj; > > > > /* use parent numa_node */ > > - if (parent && (dev_to_node(dev) == NUMA_NO_NODE)) > > - set_dev_node(dev, dev_to_node(parent)); > > + if (dev_to_node(dev) == NUMA_NO_NODE) { > > + if (parent) > > + set_dev_node(dev, dev_to_node(parent)); > > +#ifdef CONFIG_NUMA > > + else { > > + pr_err("device: '%s': has no assigned NUMA node\n", dev_name(dev)); > > + set_dev_node(dev, 0); > > + } > > +#endif > > BTW., is firmware required to always provide a NUMA node on NUMA systems? > > I.e. do we really want this warning on non-NUMA systems that don't assign > NUMA nodes? Good point; we might have to exclude nr_node_ids==1 systems from warning. > Also, even on NUMA systems, is firmware required to provide a NUMA node - > i.e. is it in principle invalid to offer no NUMA binding? I think so; a device needs to be _somewhere_, right? Typically though; devices are on a PCI bus, and the PCI bridge itself will have a NUMA binding and then the above parent rule will make everything just work. But I don't see how you can be outside of the NUMA topology.
On 2019/9/2 20:56, Peter Zijlstra wrote: > On Mon, Sep 02, 2019 at 08:25:24PM +0800, Yunsheng Lin wrote: >> On 2019/9/2 15:25, Peter Zijlstra wrote: >>> On Mon, Sep 02, 2019 at 01:46:51PM +0800, Yunsheng Lin wrote: >>>> On 2019/9/1 0:12, Peter Zijlstra wrote: >>> >>>>> 1) because even it is not set, the device really does belong to a node. >>>>> It is impossible a device will have magic uniform access to memory when >>>>> CPUs cannot. >>>> >>>> So it means dev_to_node() will return either NUMA_NO_NODE or a >>>> valid node id? >>> >>> NUMA_NO_NODE := -1, which is not a valid node number. It is also, like I >>> said, not a valid device location on a NUMA system. >>> >>> Just because ACPI/BIOS is shit, doesn't mean the device doesn't have a >>> node association. It just means we don't know and might have to guess. >> >> How do we guess the device's location when ACPI/BIOS does not set it? > > See device_add(), it looks to the device's parent and on NO_NODE, puts > it there. > > Lacking any hints, just stick it to node0 and print a FW_BUG or > something. > >> It seems dev_to_node() does not do anything about that and leave the >> job to the caller or whatever function that get called with its return >> value, such as cpumask_of_node(). > > Well, dev_to_node() doesn't do anything; nor should it. It are the > callers of set_dev_node() that should be taking care. > > Also note how device_add() sets the device node to the parent device's > node on NUMA_NO_NODE. Arguably we should change it to complain when it > finds NUMA_NO_NODE and !parent. Is it possible that the node id set by device_add() become invalid if the node is offlined, then dev_to_node() may return a invalid node id. From the comment in select_fallback_rq(), it seems that a node can be offlined, not sure if node offline process has taken cared of that? /* * If the node that the CPU is on has been offlined, cpu_to_node() * will return -1. There is no CPU on the node, and we should * select the CPU on the other node. */ With the above assumption that a device is always on a valid node, the node id returned from dev_to_node() can be safely passed to cpumask_of_node() without any checking? > > --- > drivers/base/core.c | 12 ++++++++++-- > 1 file changed, 10 insertions(+), 2 deletions(-) > > diff --git a/drivers/base/core.c b/drivers/base/core.c > index f0dd8e38fee3..2caf204966a0 100644 > --- a/drivers/base/core.c > +++ b/drivers/base/core.c > @@ -2120,8 +2120,16 @@ int device_add(struct device *dev) > dev->kobj.parent = kobj; > > /* use parent numa_node */ > - if (parent && (dev_to_node(dev) == NUMA_NO_NODE)) > - set_dev_node(dev, dev_to_node(parent)); > + if (dev_to_node(dev) == NUMA_NO_NODE) { > + if (parent) > + set_dev_node(dev, dev_to_node(parent)); > +#ifdef CONFIG_NUMA > + else { > + pr_err("device: '%s': has no assigned NUMA node\n", dev_name(dev)); > + set_dev_node(dev, 0); > + } > +#endif > + } > > /* first, register with generic layer. */ > /* we require the name to be set before, and pass NULL */ > > . >
On Tue, Sep 03, 2019 at 02:19:04PM +0800, Yunsheng Lin wrote: > On 2019/9/2 20:56, Peter Zijlstra wrote: > > On Mon, Sep 02, 2019 at 08:25:24PM +0800, Yunsheng Lin wrote: > >> On 2019/9/2 15:25, Peter Zijlstra wrote: > >>> On Mon, Sep 02, 2019 at 01:46:51PM +0800, Yunsheng Lin wrote: > >>>> On 2019/9/1 0:12, Peter Zijlstra wrote: > >>> > >>>>> 1) because even it is not set, the device really does belong to a node. > >>>>> It is impossible a device will have magic uniform access to memory when > >>>>> CPUs cannot. > >>>> > >>>> So it means dev_to_node() will return either NUMA_NO_NODE or a > >>>> valid node id? > >>> > >>> NUMA_NO_NODE := -1, which is not a valid node number. It is also, like I > >>> said, not a valid device location on a NUMA system. > >>> > >>> Just because ACPI/BIOS is shit, doesn't mean the device doesn't have a > >>> node association. It just means we don't know and might have to guess. > >> > >> How do we guess the device's location when ACPI/BIOS does not set it? > > > > See device_add(), it looks to the device's parent and on NO_NODE, puts > > it there. > > > > Lacking any hints, just stick it to node0 and print a FW_BUG or > > something. > > > >> It seems dev_to_node() does not do anything about that and leave the > >> job to the caller or whatever function that get called with its return > >> value, such as cpumask_of_node(). > > > > Well, dev_to_node() doesn't do anything; nor should it. It are the > > callers of set_dev_node() that should be taking care. > > > > Also note how device_add() sets the device node to the parent device's > > node on NUMA_NO_NODE. Arguably we should change it to complain when it > > finds NUMA_NO_NODE and !parent. > > Is it possible that the node id set by device_add() become invalid > if the node is offlined, then dev_to_node() may return a invalid > node id. In that case I would expect the device to go away too. Once the memory controller goes away, the PCI bus connected to it cannot continue to function. > From the comment in select_fallback_rq(), it seems that a node can > be offlined, not sure if node offline process has taken cared of that? > > /* > * If the node that the CPU is on has been offlined, cpu_to_node() > * will return -1. There is no CPU on the node, and we should > * select the CPU on the other node. > */ Ugh, so I disagree with that notion. cpu_to_node() mapping should be fixed, you simply cannot change it after boot, too much stuff relies on it. Setting cpu_to_node to -1 on node offline is just wrong. But alas, it seems this is already so. > With the above assumption that a device is always on a valid node, > the node id returned from dev_to_node() can be safely passed to > cpumask_of_node() without any checking?
On 2019/9/3 15:11, Peter Zijlstra wrote: > On Tue, Sep 03, 2019 at 02:19:04PM +0800, Yunsheng Lin wrote: >> On 2019/9/2 20:56, Peter Zijlstra wrote: >>> On Mon, Sep 02, 2019 at 08:25:24PM +0800, Yunsheng Lin wrote: >>>> On 2019/9/2 15:25, Peter Zijlstra wrote: >>>>> On Mon, Sep 02, 2019 at 01:46:51PM +0800, Yunsheng Lin wrote: >>>>>> On 2019/9/1 0:12, Peter Zijlstra wrote: >>>>> >>>>>>> 1) because even it is not set, the device really does belong to a node. >>>>>>> It is impossible a device will have magic uniform access to memory when >>>>>>> CPUs cannot. >>>>>> >>>>>> So it means dev_to_node() will return either NUMA_NO_NODE or a >>>>>> valid node id? >>>>> >>>>> NUMA_NO_NODE := -1, which is not a valid node number. It is also, like I >>>>> said, not a valid device location on a NUMA system. >>>>> >>>>> Just because ACPI/BIOS is shit, doesn't mean the device doesn't have a >>>>> node association. It just means we don't know and might have to guess. >>>> >>>> How do we guess the device's location when ACPI/BIOS does not set it? >>> >>> See device_add(), it looks to the device's parent and on NO_NODE, puts >>> it there. >>> >>> Lacking any hints, just stick it to node0 and print a FW_BUG or >>> something. >>> >>>> It seems dev_to_node() does not do anything about that and leave the >>>> job to the caller or whatever function that get called with its return >>>> value, such as cpumask_of_node(). >>> >>> Well, dev_to_node() doesn't do anything; nor should it. It are the >>> callers of set_dev_node() that should be taking care. >>> >>> Also note how device_add() sets the device node to the parent device's >>> node on NUMA_NO_NODE. Arguably we should change it to complain when it >>> finds NUMA_NO_NODE and !parent. >> >> Is it possible that the node id set by device_add() become invalid >> if the node is offlined, then dev_to_node() may return a invalid >> node id. > > In that case I would expect the device to go away too. Once the memory > controller goes away, the PCI bus connected to it cannot continue to > function. Ok. To summarize the discussion in order to for me to understand it correctly: 1) Make sure device_add() set to default node0 to a device if ACPI/BIOS does not set the node id and it has not no parent device. 2) Use '(unsigned)node_id >= nr_node_ids' to fix the CONFIG_DEBUG_PER_CPU_MAPS version of cpumask_of_node() for x86 and arm64, x86 just has a fix from you now, a patch for arm64 is also needed. 3) Maybe fix some other the sign bug for node id checking through the kernel using the '(unsigned)node_id >= nr_node_ids'. Please see if I understand it correctly or miss something. Maybe I can begin by sending a patch about item one to see if everyone is ok with the idea? > >> From the comment in select_fallback_rq(), it seems that a node can >> be offlined, not sure if node offline process has taken cared of that? >> >> /* >> * If the node that the CPU is on has been offlined, cpu_to_node() >> * will return -1. There is no CPU on the node, and we should >> * select the CPU on the other node. >> */ > > Ugh, so I disagree with that notion. cpu_to_node() mapping should be > fixed, you simply cannot change it after boot, too much stuff relies on > it. > > Setting cpu_to_node to -1 on node offline is just wrong. But alas, it > seems this is already so.
diff --git a/arch/x86/include/asm/topology.h b/arch/x86/include/asm/topology.h index 4b14d23..f36e9c8 100644 --- a/arch/x86/include/asm/topology.h +++ b/arch/x86/include/asm/topology.h @@ -69,6 +69,12 @@ extern const struct cpumask *cpumask_of_node(int node); /* Returns a pointer to the cpumask of CPUs on Node 'node'. */ static inline const struct cpumask *cpumask_of_node(int node) { + if (node >= nr_node_ids) + return cpu_none_mask; + + if (node < 0 || !node_to_cpumask_map[node]) + return cpu_online_mask; + return node_to_cpumask_map[node]; } #endif diff --git a/arch/x86/mm/numa.c b/arch/x86/mm/numa.c index e6dad60..5e393d2 100644 --- a/arch/x86/mm/numa.c +++ b/arch/x86/mm/numa.c @@ -868,7 +868,7 @@ const struct cpumask *cpumask_of_node(int node) dump_stack(); return cpu_none_mask; } - if (node_to_cpumask_map[node] == NULL) { + if (node < 0 || !node_to_cpumask_map[node]) { printk(KERN_WARNING "cpumask_of_node(%d): no node_to_cpumask_map!\n", node);
According to Section 6.2.14 from ACPI spec 6.3 [1], the setting of proximity domain is optional, as below: This optional object is used to describe proximity domain associations within a machine. _PXM evaluates to an integer that identifies a device as belonging to a Proximity Domain defined in the System Resource Affinity Table (SRAT). This patch checks node id with the below case before returning node_to_cpumask_map[node]: 1. if node_id >= nr_node_ids, return cpu_none_mask 2. if node_id < 0, return cpu_online_mask 3. if node_to_cpumask_map[node_id] is NULL, return cpu_online_mask [1] https://uefi.org/sites/default/files/resources/ACPI_6_3_final_Jan30.pdf Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com> --- arch/x86/include/asm/topology.h | 6 ++++++ arch/x86/mm/numa.c | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-)