| Message ID | 20170918101709.30421-2-aik@ozlabs.ru |
|---|---|
| State | New |
| Headers | show |
| Series | memory: Reduce memory use | expand |
On 18/09/2017 12:16, Alexey Kardashevskiy wrote: > Most devices use at least one address space and every time a new address > space is added, flat views and dispatch trees are rebuild for all address > spaces. This is not a problem for a relatively small amount of devices but > even 50 virtio-pci devices use more than 8GB of RAM. > > What happens that on every flatview/dispatch rebuild, new arrays are > allocated and old ones release but the release is done via RCU so until > an entire machine is build, they are not released. > > This wraps devices creation into memory_region_transaction_begin/commit > to massively reduce amount of flat view/dispatch tree (re)allocations. > > Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> > --- > Changes: > v2: > * wrapped qemu_run_machine_init_done_notifiers() as well > --- > vl.c | 9 +++++++++ > 1 file changed, 9 insertions(+) > > diff --git a/vl.c b/vl.c > index 9e62e92aea..e4f2ece590 100644 > --- a/vl.c > +++ b/vl.c > @@ -4741,12 +4741,16 @@ int main(int argc, char **argv, char **envp) > igd_gfx_passthru(); > > /* init generic devices */ > + memory_region_transaction_begin(); > + > rom_set_order_override(FW_CFG_ORDER_OVERRIDE_DEVICE); > if (qemu_opts_foreach(qemu_find_opts("device"), > device_init_func, NULL, NULL)) { > exit(1); > } > > + memory_region_transaction_commit(); > + > cpu_synchronize_all_post_init(); > > rom_reset_order_override(); > @@ -4829,8 +4833,13 @@ int main(int argc, char **argv, char **envp) > /* TODO: once all bus devices are qdevified, this should be done > * when bus is created by qdev.c */ > qemu_register_reset(qbus_reset_all_fn, sysbus_get_default()); > + > + memory_region_transaction_begin(); > + > qemu_run_machine_init_done_notifiers(); > > + memory_region_transaction_commit(); > + > if (rom_check_and_register_reset() != 0) { > error_report("rom check and register reset failed"); > exit(1); > This should not be necessary given the other patches; the PCI devices have an empty address space at the beginning, so there are other less intrusive optimizations to do instead with the same effect: 1) as a start, the "|= root->enabled" can resolve aliases. This should be enough for the PCI device case. 2) also, after patch 2 you know that the address space has no listeners here, so the begin/commit isn't really needed. Instead you can use the open-coded loop to directly generate the FlatView. This avoids touching _all_ address spaces, which is already an improvement from O(n^2) to O(n) rebuilds on device startup. 3) you can consult the list (or hash table :)) of live FlatViews (which means you keep it live after memory_region_transaction_commit ends, and only clear it on the next call), and reuse an existing FlatView. Note that the number of distinct FlatViews should be very few, so feel free to revert from hash table to list in v4 if you prefer. 4) you can skip address_space_update_topology_pass if QTAILQ_EMPTY(&as->listeners). This can provide some startup speed improvements. Optimizations 2/3/4 should be moved to the end of the series, or even in a separate post. The first can be done in the beginning too, as you prefer. Paolo
On 19/09/17 00:54, Paolo Bonzini wrote: > On 18/09/2017 12:16, Alexey Kardashevskiy wrote: >> Most devices use at least one address space and every time a new address >> space is added, flat views and dispatch trees are rebuild for all address >> spaces. This is not a problem for a relatively small amount of devices but >> even 50 virtio-pci devices use more than 8GB of RAM. >> >> What happens that on every flatview/dispatch rebuild, new arrays are >> allocated and old ones release but the release is done via RCU so until >> an entire machine is build, they are not released. >> >> This wraps devices creation into memory_region_transaction_begin/commit >> to massively reduce amount of flat view/dispatch tree (re)allocations. >> >> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> >> --- >> Changes: >> v2: >> * wrapped qemu_run_machine_init_done_notifiers() as well >> --- >> vl.c | 9 +++++++++ >> 1 file changed, 9 insertions(+) >> >> diff --git a/vl.c b/vl.c >> index 9e62e92aea..e4f2ece590 100644 >> --- a/vl.c >> +++ b/vl.c >> @@ -4741,12 +4741,16 @@ int main(int argc, char **argv, char **envp) >> igd_gfx_passthru(); >> >> /* init generic devices */ >> + memory_region_transaction_begin(); >> + >> rom_set_order_override(FW_CFG_ORDER_OVERRIDE_DEVICE); >> if (qemu_opts_foreach(qemu_find_opts("device"), >> device_init_func, NULL, NULL)) { >> exit(1); >> } >> >> + memory_region_transaction_commit(); >> + >> cpu_synchronize_all_post_init(); >> >> rom_reset_order_override(); >> @@ -4829,8 +4833,13 @@ int main(int argc, char **argv, char **envp) >> /* TODO: once all bus devices are qdevified, this should be done >> * when bus is created by qdev.c */ >> qemu_register_reset(qbus_reset_all_fn, sysbus_get_default()); >> + >> + memory_region_transaction_begin(); >> + >> qemu_run_machine_init_done_notifiers(); >> >> + memory_region_transaction_commit(); >> + >> if (rom_check_and_register_reset() != 0) { >> error_report("rom check and register reset failed"); >> exit(1); >> > > This should not be necessary given the other patches; the PCI devices > have an empty address space at the beginning, so there are other less > intrusive optimizations to do instead with the same effect: > > 1) as a start, the "|= root->enabled" can resolve aliases. This should > be enough for the PCI device case. > > 2) also, after patch 2 you know that the address space has no listeners > here, so the begin/commit isn't really needed. Instead you can use the > open-coded loop to directly generate the FlatView. This avoids touching > _all_ address spaces, which is already an improvement from O(n^2) to > O(n) rebuilds on device startup. > > 3) you can consult the list (or hash table :)) of live FlatViews (which > means you keep it live after memory_region_transaction_commit ends, and > only clear it on the next call), and reuse an existing FlatView. Note > that the number of distinct FlatViews should be very few, I keep missing this bit - why few? Each virtio-pci device creates 2 AS, with proxy->modern_bar and pci_dev->bus_master_container_region which are unique and not aliases. Remember, 500 virtio devices is my test case ;) > so feel free > to revert from hash table to list in v4 if you prefer. >> 4) you can skip address_space_update_topology_pass if > QTAILQ_EMPTY(&as->listeners). This can provide some startup speed > improvements. > > Optimizations 2/3/4 should be moved to the end of the series, or even in > a separate post. The first can be done in the beginning too, as you prefer.
On 19/09/17 12:09, Alexey Kardashevskiy wrote: > On 19/09/17 00:54, Paolo Bonzini wrote: >> On 18/09/2017 12:16, Alexey Kardashevskiy wrote: >>> Most devices use at least one address space and every time a new address >>> space is added, flat views and dispatch trees are rebuild for all address >>> spaces. This is not a problem for a relatively small amount of devices but >>> even 50 virtio-pci devices use more than 8GB of RAM. >>> >>> What happens that on every flatview/dispatch rebuild, new arrays are >>> allocated and old ones release but the release is done via RCU so until >>> an entire machine is build, they are not released. >>> >>> This wraps devices creation into memory_region_transaction_begin/commit >>> to massively reduce amount of flat view/dispatch tree (re)allocations. >>> >>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> >>> --- >>> Changes: >>> v2: >>> * wrapped qemu_run_machine_init_done_notifiers() as well >>> --- >>> vl.c | 9 +++++++++ >>> 1 file changed, 9 insertions(+) >>> >>> diff --git a/vl.c b/vl.c >>> index 9e62e92aea..e4f2ece590 100644 >>> --- a/vl.c >>> +++ b/vl.c >>> @@ -4741,12 +4741,16 @@ int main(int argc, char **argv, char **envp) >>> igd_gfx_passthru(); >>> >>> /* init generic devices */ >>> + memory_region_transaction_begin(); >>> + >>> rom_set_order_override(FW_CFG_ORDER_OVERRIDE_DEVICE); >>> if (qemu_opts_foreach(qemu_find_opts("device"), >>> device_init_func, NULL, NULL)) { >>> exit(1); >>> } >>> >>> + memory_region_transaction_commit(); >>> + >>> cpu_synchronize_all_post_init(); >>> >>> rom_reset_order_override(); >>> @@ -4829,8 +4833,13 @@ int main(int argc, char **argv, char **envp) >>> /* TODO: once all bus devices are qdevified, this should be done >>> * when bus is created by qdev.c */ >>> qemu_register_reset(qbus_reset_all_fn, sysbus_get_default()); >>> + >>> + memory_region_transaction_begin(); >>> + >>> qemu_run_machine_init_done_notifiers(); >>> >>> + memory_region_transaction_commit(); >>> + >>> if (rom_check_and_register_reset() != 0) { >>> error_report("rom check and register reset failed"); >>> exit(1); >>> >> >> This should not be necessary given the other patches; the PCI devices >> have an empty address space at the beginning, so there are other less >> intrusive optimizations to do instead with the same effect: >> >> 1) as a start, the "|= root->enabled" can resolve aliases. This should >> be enough for the PCI device case. >> >> 2) also, after patch 2 you know that the address space has no listeners >> here, so the begin/commit isn't really needed. Instead you can use the >> open-coded loop to directly generate the FlatView. This avoids touching >> _all_ address spaces, which is already an improvement from O(n^2) to >> O(n) rebuilds on device startup. 03/13 does this already, no? >> >> 3) you can consult the list (or hash table :)) of live FlatViews (which >> means you keep it live after memory_region_transaction_commit ends, and >> only clear it on the next call), and reuse an existing FlatView. Note >> that the number of distinct FlatViews should be very few, > > > I keep missing this bit - why few? Each virtio-pci device creates 2 AS, > with proxy->modern_bar and pci_dev->bus_master_container_region which are > unique and not aliases. Remember, 500 virtio devices is my test case ;) More details: pci_dev->bus_master_container_region is a root and it is enabled but its only child pci_dev->bus_master_enable_region is not. Ok, in flatview_topology_update() I can render a FV, see that it is empty (view->nr==0) and share an empty FV in this case too, this halves the number of FVs (from ~1000 to ~500 for 500 virtio devices). But proxy->modern_bar (which has an modern_cfg alias which is a root of an AS) is enabled since it is created and I could disable it and enable afterwards but since a PCI device enablement is done by writing to the config space, I kind of stuck here. I can do something like this and it helps a lot (now with -S I end up having 4 FVs and much better start time) but it is kinda hacky and "memory: Postpone flatview and dispatch tree building till all devices are added" solves this better imho, no? diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c index 3268c16966..fa2cd7cf2c 100644 --- a/hw/virtio/virtio-pci.c +++ b/hw/virtio/virtio-pci.c @@ -629,6 +629,8 @@ static void virtio_write_config(PCIDevice *pci_dev, uint32_t address, VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus); struct virtio_pci_cfg_cap *cfg; + memory_region_set_enabled(&proxy->modern_bar, true); + pci_default_write_config(pci_dev, address, val, len); if (range_covers_byte(address, len, PCI_COMMAND) && @@ -662,6 +664,8 @@ static uint32_t virtio_read_config(PCIDevice *pci_dev, VirtIOPCIProxy *proxy = DO_UPCAST(VirtIOPCIProxy, pci_dev, pci_dev); struct virtio_pci_cfg_cap *cfg; + memory_region_set_enabled(&proxy->modern_bar, true); + if (proxy->config_cap && ranges_overlap(address, len, proxy->config_cap + offsetof(struct virtio_pci_cfg_cap, pci_cfg_data), @@ -1790,6 +1794,8 @@ static void virtio_pci_realize(PCIDevice *pci_dev, Error **errp) 0, memory_region_size(&proxy->modern_bar)); + memory_region_set_enabled(&proxy->modern_bar, false); + address_space_init(&proxy->modern_as, &proxy->modern_cfg, "virtio-pci-cfg-as"); > > >> so feel free >> to revert from hash table to list in v4 if you prefer. >>> 4) you can skip address_space_update_topology_pass if >> QTAILQ_EMPTY(&as->listeners). This can provide some startup speed >> improvements. >> >> Optimizations 2/3/4 should be moved to the end of the series, or even in >> a separate post. The first can be done in the beginning too, as you prefer. > > > >
On 19/09/2017 08:57, Alexey Kardashevskiy wrote: >>> 3) you can consult the list (or hash table :)) of live FlatViews (which >>> means you keep it live after memory_region_transaction_commit ends, and >>> only clear it on the next call), and reuse an existing FlatView. Note >>> that the number of distinct FlatViews should be very few, >> >> I keep missing this bit - why few? Each virtio-pci device creates 2 AS, >> with proxy->modern_bar and pci_dev->bus_master_container_region which are >> unique and not aliases. Remember, 500 virtio devices is my test case ;) Argh, no, it's me who keeps missing the bit. > More details: pci_dev->bus_master_container_region is a root and it is > enabled but its only child pci_dev->bus_master_enable_region is not. Ok, in > flatview_topology_update() I can render a FV, see that it is empty > (view->nr==0) and share an empty FV in this case too, this halves the > number of FVs (from ~1000 to ~500 for 500 virtio devices). We can also resolve "fake" aliases, obtained through a region with only one subregion. > But proxy->modern_bar (which has an modern_cfg alias which is a root of an > AS) is enabled since it is created and I could disable it and enable > afterwards but since a PCI device enablement is done by writing to the > config space, I kind of stuck here. > > I can do something like this and it helps a lot (now with -S I end up > having 4 FVs and much better start time) but it is kinda hacky and "memory: > Postpone flatview and dispatch tree building till all devices are added" > solves this better imho, no? I think modern_bar (actually modern_cfg) shouldn't have its own address space, it's a pretty wasteful way to compute the dispatch tree where a simple linked list or array would be enough: memory-region: virtio-pci 00000000fe000000-00000000fe003fff (prio 1, i/o): virtio-pci 00000000fe000000-00000000fe000fff (prio 0, i/o): virtio-pci-common 00000000fe001000-00000000fe001fff (prio 0, i/o): virtio-pci-isr 00000000fe002000-00000000fe002fff (prio 0, i/o): virtio-pci-device 00000000fe003000-00000000fe003fff (prio 0, i/o): virtio-pci-notify Paolo
On 19/09/2017 08:57, Alexey Kardashevskiy wrote: >>> 2) also, after patch 2 you know that the address space has no listeners >>> here, so the begin/commit isn't really needed. Instead you can use the >>> open-coded loop to directly generate the FlatView. This avoids touching >>> _all_ address spaces, which is already an improvement from O(n^2) to >>> O(n) rebuilds on device startup. > 03/13 does this already, no? > No, there's still as->dispatch = NULL; memory_region_update_pending |= root->enabled; memory_region_transaction_commit(); when creating the address space, which touches all address spaces. Paolo
On 19/09/17 17:40, Paolo Bonzini wrote: > On 19/09/2017 08:57, Alexey Kardashevskiy wrote: >>>> 3) you can consult the list (or hash table :)) of live FlatViews (which >>>> means you keep it live after memory_region_transaction_commit ends, and >>>> only clear it on the next call), and reuse an existing FlatView. Note >>>> that the number of distinct FlatViews should be very few, >>> >>> I keep missing this bit - why few? Each virtio-pci device creates 2 AS, >>> with proxy->modern_bar and pci_dev->bus_master_container_region which are >>> unique and not aliases. Remember, 500 virtio devices is my test case ;) > > Argh, no, it's me who keeps missing the bit. > >> More details: pci_dev->bus_master_container_region is a root and it is >> enabled but its only child pci_dev->bus_master_enable_region is not. Ok, in >> flatview_topology_update() I can render a FV, see that it is empty >> (view->nr==0) and share an empty FV in this case too, this halves the >> number of FVs (from ~1000 to ~500 for 500 virtio devices). > > We can also resolve "fake" aliases, obtained through a region with only > one subregion. I just do this now (will post soon): empty_view = generate_memory_topology(NULL); [...] new_view = generate_memory_topology(physmr); if (!new_view->nr) { flatview_ref(empty_view); flatview_unref(new_view); new_view = empty_view; [...] g_hash_table_insert(views, physmr, new_view); and that's it. For non empty FVs I need to render FVs anyway and for all-disabled/empty FVs it is really cheap so no big loss here. > >> But proxy->modern_bar (which has an modern_cfg alias which is a root of an >> AS) is enabled since it is created and I could disable it and enable >> afterwards but since a PCI device enablement is done by writing to the >> config space, I kind of stuck here. >> >> I can do something like this and it helps a lot (now with -S I end up >> having 4 FVs and much better start time) but it is kinda hacky and "memory: >> Postpone flatview and dispatch tree building till all devices are added" >> solves this better imho, no? > > I think modern_bar (actually modern_cfg) shouldn't have its own address > space, it's a pretty wasteful way to compute the dispatch tree where a > simple linked list or array would be enough: Why would we need an array there anyway? This AS is only used for virtio_address_space_write() and virtio_address_space_read() so we only need here s/virtio_address_space_write/memory_region_dispatch_write/ (and the same for reading), no? > > memory-region: virtio-pci > 00000000fe000000-00000000fe003fff (prio 1, i/o): virtio-pci > 00000000fe000000-00000000fe000fff (prio 0, i/o): virtio-pci-common > 00000000fe001000-00000000fe001fff (prio 0, i/o): virtio-pci-isr > 00000000fe002000-00000000fe002fff (prio 0, i/o): virtio-pci-device > 00000000fe003000-00000000fe003fff (prio 0, i/o): virtio-pci-notify
On 19/09/2017 09:57, Alexey Kardashevskiy wrote: >> We can also resolve "fake" aliases, obtained through a region with only >> one subregion. > I just do this now (will post soon): > > > empty_view = generate_memory_topology(NULL); > [...] > > new_view = generate_memory_topology(physmr); > > if (!new_view->nr) { > flatview_ref(empty_view); > flatview_unref(new_view); > new_view = empty_view; > > [...] > > g_hash_table_insert(views, physmr, new_view); > > and that's it. For non empty FVs I need to render FVs anyway and for > all-disabled/empty FVs it is really cheap so no big loss here. Is that enough to get 1 shared FV for all bus master address spaces? In any case, your patches provide a very good infrastructure and further optimization can be done easily on top. Paolo
On 19/09/17 17:42, Paolo Bonzini wrote: > On 19/09/2017 08:57, Alexey Kardashevskiy wrote: >>>> 2) also, after patch 2 you know that the address space has no listeners >>>> here, so the begin/commit isn't really needed. Instead you can use the >>>> open-coded loop to directly generate the FlatView. This avoids touching >>>> _all_ address spaces, which is already an improvement from O(n^2) to >>>> O(n) rebuilds on device startup. >> 03/13 does this already, no? >> > > No, there's still > > as->dispatch = NULL; > memory_region_update_pending |= root->enabled; > memory_region_transaction_commit(); > > when creating the address space, which touches all address spaces. This does not seem helping a lot - my test without such an optimization allocates 2966044 FVs and 2388768 with it, which is just 25%, i.e. worth making a patch but yet getting rid of those nasty virtio-modern address spaces will do a lot better job. What is the exact concern about the "[PATCH qemu v3 01/13] memory: Postpone flatview and dispatch tree building till all devices are added"? Is it that you missed that modern AS thing in virtio and thought we can do well even without it? Or there is something wrong with the approach?
On 19/09/2017 12:39, Alexey Kardashevskiy wrote: >> No, there's still >> >> as->dispatch = NULL; >> memory_region_update_pending |= root->enabled; >> memory_region_transaction_commit(); >> >> when creating the address space, which touches all address spaces. > > This does not seem helping a lot - my test without such an optimization > allocates 2966044 FVs and 2388768 with it, which is just 25%, i.e. worth > making a patch but yet getting rid of those nasty virtio-modern address > spaces will do a lot better job. Have you tried resolving aliases that are implemented with containers and one subregion? > What is the exact concern about the "[PATCH qemu v3 01/13] memory: Postpone > flatview and dispatch tree building till all devices are added"? Is it that > you missed that modern AS thing in virtio and thought we can do well even > without it? I am afraid that some of the special sysbus devices (not PCI, those go through the bus master DMA region and it's empty on startup) might write to memory during initialization. Thanks, Paolo > Or there is something wrong with the approach?
On 19/09/17 20:47, Paolo Bonzini wrote: > On 19/09/2017 12:39, Alexey Kardashevskiy wrote: >>> No, there's still >>> >>> as->dispatch = NULL; >>> memory_region_update_pending |= root->enabled; >>> memory_region_transaction_commit(); >>> >>> when creating the address space, which touches all address spaces. >> >> This does not seem helping a lot - my test without such an optimization >> allocates 2966044 FVs and 2388768 with it, which is just 25%, i.e. worth >> making a patch but yet getting rid of those nasty virtio-modern address >> spaces will do a lot better job. > > Have you tried resolving aliases that are implemented with containers > and one subregion? Yes, for 10 virtio-net and 8 cpus it looks like below, #5 is that shared thing. And if I use Intel e1000 instead, it is just 4 FVs. (qemu) info mtree -f -d FlatView #0 AS "virtio-pci-cfg-as", root: virtio-pci-cfg, alias virtio-pci Root memory region: virtio-pci FlatView #1 AS "I/O", root: io Root memory region: io FlatView #2 AS "pci@800000020000000", root: pci@800000020000000.iommu-root Root memory region: pci@800000020000000.iommu-root FlatView #3 AS "virtio-pci-cfg-as", root: virtio-pci-cfg, alias virtio-pci Root memory region: virtio-pci FlatView #4 AS "virtio-pci-cfg-as", root: virtio-pci-cfg, alias virtio-pci Root memory region: virtio-pci FlatView #5 AS "virtio-net-pci", root: bus master container AS "virtio-net-pci", root: bus master container AS "virtio-net-pci", root: bus master container AS "virtio-net-pci", root: bus master container AS "virtio-net-pci", root: bus master container AS "virtio-net-pci", root: bus master container AS "virtio-net-pci", root: bus master container AS "virtio-net-pci", root: bus master container AS "virtio-net-pci", root: bus master container AS "virtio-net-pci", root: bus master container Root memory region: (none) No rendered FlatView FlatView #6 AS "virtio-pci-cfg-as", root: virtio-pci-cfg, alias virtio-pci Root memory region: virtio-pci FlatView #7 AS "virtio-pci-cfg-as", root: virtio-pci-cfg, alias virtio-pci Root memory region: virtio-pci FlatView #8 AS "virtio-pci-cfg-as", root: virtio-pci-cfg, alias virtio-pci Root memory region: virtio-pci FlatView #9 AS "virtio-pci-cfg-as", root: virtio-pci-cfg, alias virtio-pci Root memory region: virtio-pci FlatView #10 AS "virtio-pci-cfg-as", root: virtio-pci-cfg, alias virtio-pci Root memory region: virtio-pci FlatView #11 AS "memory", root: system AS "cpu-memory", root: system AS "cpu-memory", root: system AS "cpu-memory", root: system AS "cpu-memory", root: system AS "cpu-memory", root: system AS "cpu-memory", root: system AS "cpu-memory", root: system AS "cpu-memory", root: system Root memory region: system FlatView #12 AS "virtio-pci-cfg-as", root: virtio-pci-cfg, alias virtio-pci Root memory region: virtio-pci FlatView #13 AS "virtio-pci-cfg-as", root: virtio-pci-cfg, alias virtio-pci Root memory region: virtio-pci >> What is the exact concern about the "[PATCH qemu v3 01/13] memory: Postpone >> flatview and dispatch tree building till all devices are added"? Is it that >> you missed that modern AS thing in virtio and thought we can do well even >> without it? > > I am afraid that some of the special sysbus devices (not PCI, those go > through the bus master DMA region and it's empty on startup) might write > to memory during initialization. And somehow QEMU guarantees that all RAM MRs are rendered to FlatViews even though some might be added via -device (can they btw?)? > > Thanks, > > Paolo > >> Or there is something wrong with the approach? >
On 19/09/2017 16:12, Alexey Kardashevskiy wrote: >>> What is the exact concern about the "[PATCH qemu v3 01/13] memory: Postpone >>> flatview and dispatch tree building till all devices are added"? Is it that >>> you missed that modern AS thing in virtio and thought we can do well even >>> without it? >> I am afraid that some of the special sysbus devices (not PCI, those go >> through the bus master DMA region and it's empty on startup) might write >> to memory during initialization. > > And somehow QEMU guarantees that all RAM MRs are rendered to FlatViews even > though some might be added via -device (can they btw?)? I don't know, but as discussed on IRC, I think this is a pretty different change from the rest of the series---which should have zero change except for going faster---so I'd prefer to keep it out for now and see where optimization can get us. Paolo
diff --git a/vl.c b/vl.c index 9e62e92aea..e4f2ece590 100644 --- a/vl.c +++ b/vl.c @@ -4741,12 +4741,16 @@ int main(int argc, char **argv, char **envp) igd_gfx_passthru(); /* init generic devices */ + memory_region_transaction_begin(); + rom_set_order_override(FW_CFG_ORDER_OVERRIDE_DEVICE); if (qemu_opts_foreach(qemu_find_opts("device"), device_init_func, NULL, NULL)) { exit(1); } + memory_region_transaction_commit(); + cpu_synchronize_all_post_init(); rom_reset_order_override(); @@ -4829,8 +4833,13 @@ int main(int argc, char **argv, char **envp) /* TODO: once all bus devices are qdevified, this should be done * when bus is created by qdev.c */ qemu_register_reset(qbus_reset_all_fn, sysbus_get_default()); + + memory_region_transaction_begin(); + qemu_run_machine_init_done_notifiers(); + memory_region_transaction_commit(); + if (rom_check_and_register_reset() != 0) { error_report("rom check and register reset failed"); exit(1);
Most devices use at least one address space and every time a new address space is added, flat views and dispatch trees are rebuild for all address spaces. This is not a problem for a relatively small amount of devices but even 50 virtio-pci devices use more than 8GB of RAM. What happens that on every flatview/dispatch rebuild, new arrays are allocated and old ones release but the release is done via RCU so until an entire machine is build, they are not released. This wraps devices creation into memory_region_transaction_begin/commit to massively reduce amount of flat view/dispatch tree (re)allocations. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> --- Changes: v2: * wrapped qemu_run_machine_init_done_notifiers() as well --- vl.c | 9 +++++++++ 1 file changed, 9 insertions(+)