| Message ID | 1444174532-2663-1-git-send-email-saurabh@cplanenetworks.com |
|---|---|
| State | Accepted |
| Headers | show |
On Tue, Oct 06, 2015 at 04:35:32PM -0700, Saurabh Mohan wrote: > On Ubuntu depmod's search priority is configured in /etc/depmod to be > updates and then the kernel built-in directory. > $ cat /etc/depmod.d/ubuntu.conf > search updates ubuntu built-in > > Thus change the placement of openvswitch.ko under updates/ not kernel/updates. > > Signed-off-by: Saurabh Mohan <saurabh@cplanenetworks.com> This appears to be correct, but I'm confused about how this could have not been noticed for years. Did something change recently?
On 14 October 2015 at 15:21, Ben Pfaff <blp@nicira.com> wrote: > On Tue, Oct 06, 2015 at 04:35:32PM -0700, Saurabh Mohan wrote: >> On Ubuntu depmod's search priority is configured in /etc/depmod to be >> updates and then the kernel built-in directory. >> $ cat /etc/depmod.d/ubuntu.conf >> search updates ubuntu built-in >> >> Thus change the placement of openvswitch.ko under updates/ not kernel/updates. >> >> Signed-off-by: Saurabh Mohan <saurabh@cplanenetworks.com> > > This appears to be correct, but I'm confused about how this could have > not been noticed for years. Did something change recently? We recently changed it from kernel/ to kernel/updates (prior to v2.4 release), and the commit message suggests it was previously nondeterministic: commit b519432205c36bda5c7331f77a49eaaa919967ad Author: Ansis Atteka <aatteka@nicira.com> Date: Tue May 26 16:49:49 2015 -0700 debian: install openvswitch kernel module under "updates" directory This patch fixes a bug where "modprobe openvswitch" command on Ubuntu distribution would have sometimes tried to load OVS kernel module that shipped together with Linux Kernel, even though one had also installed OVS datapath debian package created with module-assistant. Because of this issue force-reload-kmod command occasionally malfunctioned and failed to load the right kernel module. This bug happened *occasionally* because the default Ubuntu depmod configuration in /etc/depmod.d/ubuntu.conf is set to look for kernel modules first in "updates" directory, then in "ubuntu" directory and then in other directories. If there were two openvswitch.ko modules in "other directories", then modprobe would have loaded kernel module that was nondeterministically listed first by file system.
On Wed, Oct 14, 2015 at 03:28:24PM -0700, Joe Stringer wrote: > On 14 October 2015 at 15:21, Ben Pfaff <blp@nicira.com> wrote: > > On Tue, Oct 06, 2015 at 04:35:32PM -0700, Saurabh Mohan wrote: > >> On Ubuntu depmod's search priority is configured in /etc/depmod to be > >> updates and then the kernel built-in directory. > >> $ cat /etc/depmod.d/ubuntu.conf > >> search updates ubuntu built-in > >> > >> Thus change the placement of openvswitch.ko under updates/ not kernel/updates. > >> > >> Signed-off-by: Saurabh Mohan <saurabh@cplanenetworks.com> > > > > This appears to be correct, but I'm confused about how this could have > > not been noticed for years. Did something change recently? > > We recently changed it from kernel/ to kernel/updates (prior to v2.4 > release), and the commit message suggests it was previously > nondeterministic: > > commit b519432205c36bda5c7331f77a49eaaa919967ad > Author: Ansis Atteka <aatteka@nicira.com> > Date: Tue May 26 16:49:49 2015 -0700 > > debian: install openvswitch kernel module under "updates" directory > > This patch fixes a bug where "modprobe openvswitch" command on Ubuntu > distribution would have sometimes tried to load OVS kernel module that > shipped together with Linux Kernel, even though one had also installed > OVS datapath debian package created with module-assistant. Because of > this issue force-reload-kmod command occasionally malfunctioned and > failed to load the right kernel module. > > This bug happened *occasionally* because the default Ubuntu depmod > configuration in /etc/depmod.d/ubuntu.conf is set to look for kernel > modules first in "updates" directory, then in "ubuntu" directory and > then in other directories. If there were two openvswitch.ko modules > in "other directories", then modprobe would have loaded kernel > module that was nondeterministically listed first by file system. OK, I understand why it was nondeterministic before, but where does kernel/updates come in then, since it seems to be different from and not as high-priority as "updates"? Does anyone know? Thanks, Ben.
On Wed, Oct 14, 2015 at 4:08 PM, Ben Pfaff <blp@nicira.com> wrote: > On Wed, Oct 14, 2015 at 03:28:24PM -0700, Joe Stringer wrote: >> On 14 October 2015 at 15:21, Ben Pfaff <blp@nicira.com> wrote: >> > On Tue, Oct 06, 2015 at 04:35:32PM -0700, Saurabh Mohan wrote: >> >> On Ubuntu depmod's search priority is configured in /etc/depmod to be >> >> updates and then the kernel built-in directory. >> >> $ cat /etc/depmod.d/ubuntu.conf >> >> search updates ubuntu built-in >> >> >> >> Thus change the placement of openvswitch.ko under updates/ not kernel/updates. >> >> >> >> Signed-off-by: Saurabh Mohan <saurabh@cplanenetworks.com> >> > >> > This appears to be correct, but I'm confused about how this could have >> > not been noticed for years. Did something change recently? >> >> We recently changed it from kernel/ to kernel/updates (prior to v2.4 >> release), and the commit message suggests it was previously >> nondeterministic: >> >> commit b519432205c36bda5c7331f77a49eaaa919967ad >> Author: Ansis Atteka <aatteka@nicira.com> >> Date: Tue May 26 16:49:49 2015 -0700 >> >> debian: install openvswitch kernel module under "updates" directory >> >> This patch fixes a bug where "modprobe openvswitch" command on Ubuntu >> distribution would have sometimes tried to load OVS kernel module that >> shipped together with Linux Kernel, even though one had also installed >> OVS datapath debian package created with module-assistant. Because of >> this issue force-reload-kmod command occasionally malfunctioned and >> failed to load the right kernel module. >> >> This bug happened *occasionally* because the default Ubuntu depmod >> configuration in /etc/depmod.d/ubuntu.conf is set to look for kernel >> modules first in "updates" directory, then in "ubuntu" directory and >> then in other directories. If there were two openvswitch.ko modules >> in "other directories", then modprobe would have loaded kernel >> module that was nondeterministically listed first by file system. > > OK, I understand why it was nondeterministic before, but where does > kernel/updates come in then, since it seems to be different from and not > as high-priority as "updates"? Does anyone know? I am still trying to find the answer in my email history why I ended up using "kernel/updates" over "updates". Saurabh, did you encounter an issue where the wrong kernel module was loaded or is this to achieve conformance? > > Thanks, > > Ben. > _______________________________________________ > dev mailing list > dev@openvswitch.org > http://openvswitch.org/mailman/listinfo/dev
On 10/14/2015 04:58 PM, Ansis Atteka wrote: > On Wed, Oct 14, 2015 at 4:08 PM, Ben Pfaff <blp@nicira.com> wrote: >> On Wed, Oct 14, 2015 at 03:28:24PM -0700, Joe Stringer wrote: >>> On 14 October 2015 at 15:21, Ben Pfaff <blp@nicira.com> wrote: >>>> On Tue, Oct 06, 2015 at 04:35:32PM -0700, Saurabh Mohan wrote: >>>>> On Ubuntu depmod's search priority is configured in /etc/depmod to be >>>>> updates and then the kernel built-in directory. >>>>> $ cat /etc/depmod.d/ubuntu.conf >>>>> search updates ubuntu built-in >>>>> >>>>> Thus change the placement of openvswitch.ko under updates/ not kernel/updates. >>>>> >>>>> Signed-off-by: Saurabh Mohan <saurabh@cplanenetworks.com> >>>> >>>> This appears to be correct, but I'm confused about how this could have >>>> not been noticed for years. Did something change recently? >>> >>> We recently changed it from kernel/ to kernel/updates (prior to v2.4 >>> release), and the commit message suggests it was previously >>> nondeterministic: >>> >>> commit b519432205c36bda5c7331f77a49eaaa919967ad >>> Author: Ansis Atteka <aatteka@nicira.com> >>> Date: Tue May 26 16:49:49 2015 -0700 >>> >>> debian: install openvswitch kernel module under "updates" directory >>> >>> This patch fixes a bug where "modprobe openvswitch" command on Ubuntu >>> distribution would have sometimes tried to load OVS kernel module that >>> shipped together with Linux Kernel, even though one had also installed >>> OVS datapath debian package created with module-assistant. Because of >>> this issue force-reload-kmod command occasionally malfunctioned and >>> failed to load the right kernel module. >>> >>> This bug happened *occasionally* because the default Ubuntu depmod >>> configuration in /etc/depmod.d/ubuntu.conf is set to look for kernel >>> modules first in "updates" directory, then in "ubuntu" directory and >>> then in other directories. If there were two openvswitch.ko modules >>> in "other directories", then modprobe would have loaded kernel >>> module that was nondeterministically listed first by file system. >> >> OK, I understand why it was nondeterministic before, but where does >> kernel/updates come in then, since it seems to be different from and not >> as high-priority as "updates"? Does anyone know? > > I am still trying to find the answer in my email history why I ended > up using "kernel/updates" over "updates". > > Saurabh, did you encounter an issue where the wrong kernel module was > loaded or is this to achieve conformance? > Anis, we tried using this patch but still noticed that the wrong kernel module was getting selected. The only way to fix it was to put the module outside kernel/ directory. >> >> Thanks, >> >> Ben. >> _______________________________________________ >> dev mailing list >> dev@openvswitch.org >> http://openvswitch.org/mailman/listinfo/dev
On 10/14/2015 04:58 PM, Ansis Atteka wrote: > On Wed, Oct 14, 2015 at 4:08 PM, Ben Pfaff <blp@nicira.com> wrote: >> On Wed, Oct 14, 2015 at 03:28:24PM -0700, Joe Stringer wrote: >>> On 14 October 2015 at 15:21, Ben Pfaff <blp@nicira.com> wrote: >>>> On Tue, Oct 06, 2015 at 04:35:32PM -0700, Saurabh Mohan wrote: >>>>> On Ubuntu depmod's search priority is configured in /etc/depmod to be >>>>> updates and then the kernel built-in directory. >>>>> $ cat /etc/depmod.d/ubuntu.conf >>>>> search updates ubuntu built-in >>>>> >>>>> Thus change the placement of openvswitch.ko under updates/ not kernel/updates. >>>>> >>>>> Signed-off-by: Saurabh Mohan <saurabh@cplanenetworks.com> >>>> >>>> This appears to be correct, but I'm confused about how this could have >>>> not been noticed for years. Did something change recently? >>> >>> We recently changed it from kernel/ to kernel/updates (prior to v2.4 >>> release), and the commit message suggests it was previously >>> nondeterministic: >>> >>> commit b519432205c36bda5c7331f77a49eaaa919967ad >>> Author: Ansis Atteka <aatteka@nicira.com> >>> Date: Tue May 26 16:49:49 2015 -0700 >>> >>> debian: install openvswitch kernel module under "updates" directory >>> >>> This patch fixes a bug where "modprobe openvswitch" command on Ubuntu >>> distribution would have sometimes tried to load OVS kernel module that >>> shipped together with Linux Kernel, even though one had also installed >>> OVS datapath debian package created with module-assistant. Because of >>> this issue force-reload-kmod command occasionally malfunctioned and >>> failed to load the right kernel module. >>> >>> This bug happened *occasionally* because the default Ubuntu depmod >>> configuration in /etc/depmod.d/ubuntu.conf is set to look for kernel >>> modules first in "updates" directory, then in "ubuntu" directory and >>> then in other directories. If there were two openvswitch.ko modules >>> in "other directories", then modprobe would have loaded kernel >>> module that was nondeterministically listed first by file system. >> >> OK, I understand why it was nondeterministic before, but where does >> kernel/updates come in then, since it seems to be different from and not >> as high-priority as "updates"? Does anyone know? > > I am still trying to find the answer in my email history why I ended > up using "kernel/updates" over "updates". > > Saurabh, did you encounter an issue where the wrong kernel module was > loaded or is this to achieve conformance? (resending it again as previously i did not include everyone. Sorry) Ansis, we tried using this patch but still noticed that the wrong kernel module was getting selected. The only way to fix it was to put the module outside kernel/ directory. > >> >> Thanks, >> >> Ben. >> _______________________________________________ >> dev mailing list >> dev@openvswitch.org >> http://openvswitch.org/mailman/listinfo/dev
On Wed, Oct 14, 2015 at 5:33 PM, Saurabh Mohan <saurabh@cplanenetworks.com> wrote: > On 10/14/2015 04:58 PM, Ansis Atteka wrote: >> >> On Wed, Oct 14, 2015 at 4:08 PM, Ben Pfaff <blp@nicira.com> wrote: >>> >>> On Wed, Oct 14, 2015 at 03:28:24PM -0700, Joe Stringer wrote: >>>> >>>> On 14 October 2015 at 15:21, Ben Pfaff <blp@nicira.com> wrote: >>>>> >>>>> On Tue, Oct 06, 2015 at 04:35:32PM -0700, Saurabh Mohan wrote: >>>>>> >>>>>> On Ubuntu depmod's search priority is configured in /etc/depmod to be >>>>>> updates and then the kernel built-in directory. >>>>>> $ cat /etc/depmod.d/ubuntu.conf >>>>>> search updates ubuntu built-in >>>>>> >>>>>> Thus change the placement of openvswitch.ko under updates/ not >>>>>> kernel/updates. >>>>>> >>>>>> Signed-off-by: Saurabh Mohan <saurabh@cplanenetworks.com> >>>>> >>>>> >>>>> This appears to be correct, but I'm confused about how this could have >>>>> not been noticed for years. Did something change recently? >>>> >>>> >>>> We recently changed it from kernel/ to kernel/updates (prior to v2.4 >>>> release), and the commit message suggests it was previously >>>> nondeterministic: >>>> >>>> commit b519432205c36bda5c7331f77a49eaaa919967ad >>>> Author: Ansis Atteka <aatteka@nicira.com> >>>> Date: Tue May 26 16:49:49 2015 -0700 >>>> >>>> debian: install openvswitch kernel module under "updates" directory >>>> >>>> This patch fixes a bug where "modprobe openvswitch" command on >>>> Ubuntu >>>> distribution would have sometimes tried to load OVS kernel module >>>> that >>>> shipped together with Linux Kernel, even though one had also >>>> installed >>>> OVS datapath debian package created with module-assistant. Because >>>> of >>>> this issue force-reload-kmod command occasionally malfunctioned and >>>> failed to load the right kernel module. >>>> >>>> This bug happened *occasionally* because the default Ubuntu depmod >>>> configuration in /etc/depmod.d/ubuntu.conf is set to look for >>>> kernel >>>> modules first in "updates" directory, then in "ubuntu" directory >>>> and >>>> then in other directories. If there were two openvswitch.ko >>>> modules >>>> in "other directories", then modprobe would have loaded kernel >>>> module that was nondeterministically listed first by file system. >>> >>> >>> OK, I understand why it was nondeterministic before, but where does >>> kernel/updates come in then, since it seems to be different from and not >>> as high-priority as "updates"? Does anyone know? >> >> >> I am still trying to find the answer in my email history why I ended >> up using "kernel/updates" over "updates". >> >> Saurabh, did you encounter an issue where the wrong kernel module was >> loaded or is this to achieve conformance? >> > > Anis, we tried using this patch but still noticed that the wrong kernel > module was getting selected. The only way to fix it was > to put the module outside kernel/ directory. Can you give me locations of all ovs kernel modules (dkms, module-assistant and the one that came with linux) present in /lib/modules/<linux_version>/ on your system that was having trouble? I am just wondering why my patch prioritized module-assistant created kernel module (in kernel/updates) over the one that comes with linux kernel itself. I guess this is the problem you are seeing here again, right?
On 10/15/2015 01:55 PM, Ansis Atteka wrote: > On Wed, Oct 14, 2015 at 5:33 PM, Saurabh Mohan > <saurabh@cplanenetworks.com> wrote: >> On 10/14/2015 04:58 PM, Ansis Atteka wrote: >>> >>> On Wed, Oct 14, 2015 at 4:08 PM, Ben Pfaff <blp@nicira.com> wrote: >>>> >>>> On Wed, Oct 14, 2015 at 03:28:24PM -0700, Joe Stringer wrote: >>>>> >>>>> On 14 October 2015 at 15:21, Ben Pfaff <blp@nicira.com> wrote: >>>>>> >>>>>> On Tue, Oct 06, 2015 at 04:35:32PM -0700, Saurabh Mohan wrote: >>>>>>> >>>>>>> On Ubuntu depmod's search priority is configured in /etc/depmod to be >>>>>>> updates and then the kernel built-in directory. >>>>>>> $ cat /etc/depmod.d/ubuntu.conf >>>>>>> search updates ubuntu built-in >>>>>>> >>>>>>> Thus change the placement of openvswitch.ko under updates/ not >>>>>>> kernel/updates. >>>>>>> >>>>>>> Signed-off-by: Saurabh Mohan <saurabh@cplanenetworks.com> >>>>>> >>>>>> >>>>>> This appears to be correct, but I'm confused about how this could have >>>>>> not been noticed for years. Did something change recently? >>>>> >>>>> >>>>> We recently changed it from kernel/ to kernel/updates (prior to v2.4 >>>>> release), and the commit message suggests it was previously >>>>> nondeterministic: >>>>> >>>>> commit b519432205c36bda5c7331f77a49eaaa919967ad >>>>> Author: Ansis Atteka <aatteka@nicira.com> >>>>> Date: Tue May 26 16:49:49 2015 -0700 >>>>> >>>>> debian: install openvswitch kernel module under "updates" directory >>>>> >>>>> This patch fixes a bug where "modprobe openvswitch" command on >>>>> Ubuntu >>>>> distribution would have sometimes tried to load OVS kernel module >>>>> that >>>>> shipped together with Linux Kernel, even though one had also >>>>> installed >>>>> OVS datapath debian package created with module-assistant. Because >>>>> of >>>>> this issue force-reload-kmod command occasionally malfunctioned and >>>>> failed to load the right kernel module. >>>>> >>>>> This bug happened *occasionally* because the default Ubuntu depmod >>>>> configuration in /etc/depmod.d/ubuntu.conf is set to look for >>>>> kernel >>>>> modules first in "updates" directory, then in "ubuntu" directory >>>>> and >>>>> then in other directories. If there were two openvswitch.ko >>>>> modules >>>>> in "other directories", then modprobe would have loaded kernel >>>>> module that was nondeterministically listed first by file system. >>>> >>>> >>>> OK, I understand why it was nondeterministic before, but where does >>>> kernel/updates come in then, since it seems to be different from and not >>>> as high-priority as "updates"? Does anyone know? >>> >>> >>> I am still trying to find the answer in my email history why I ended >>> up using "kernel/updates" over "updates". >>> >>> Saurabh, did you encounter an issue where the wrong kernel module was >>> loaded or is this to achieve conformance? >>> >> >> Anis, we tried using this patch but still noticed that the wrong kernel >> module was getting selected. The only way to fix it was >> to put the module outside kernel/ directory. > > > Can you give me locations of all ovs kernel modules (dkms, > module-assistant and the one that came with linux) present in > /lib/modules/<linux_version>/ on your system that was having trouble? > root@test01-1:/lib/modules# find . -name openvswitch.ko -print ./3.13.0-32-generic/kernel/net/openvswitch/openvswitch.ko ./3.13.0-32-generic/updates/openvswitch.ko > I am just wondering why my patch prioritized module-assistant created > kernel module (in kernel/updates) over the one that comes with linux > kernel itself. I guess this is the problem you are seeing here again, > right? yes, we were seeing the same problem. my observation was that if we put the module under directory <kernel>/updates then depmod would still select <kernel>/net/openvswitch/openvswitch.ko >
On Thu, Oct 15, 2015 at 4:52 PM, Saurabh Mohan <saurabh@cplanenetworks.com> wrote: > On 10/15/2015 01:55 PM, Ansis Atteka wrote: >> >> On Wed, Oct 14, 2015 at 5:33 PM, Saurabh Mohan >> <saurabh@cplanenetworks.com> wrote: >>> >>> On 10/14/2015 04:58 PM, Ansis Atteka wrote: >>>> >>>> >>>> On Wed, Oct 14, 2015 at 4:08 PM, Ben Pfaff <blp@nicira.com> wrote: >>>>> >>>>> >>>>> On Wed, Oct 14, 2015 at 03:28:24PM -0700, Joe Stringer wrote: >>>>>> >>>>>> >>>>>> On 14 October 2015 at 15:21, Ben Pfaff <blp@nicira.com> wrote: >>>>>>> >>>>>>> >>>>>>> On Tue, Oct 06, 2015 at 04:35:32PM -0700, Saurabh Mohan wrote: >>>>>>>> >>>>>>>> >>>>>>>> On Ubuntu depmod's search priority is configured in /etc/depmod to >>>>>>>> be >>>>>>>> updates and then the kernel built-in directory. >>>>>>>> $ cat /etc/depmod.d/ubuntu.conf >>>>>>>> search updates ubuntu built-in >>>>>>>> >>>>>>>> Thus change the placement of openvswitch.ko under updates/ not >>>>>>>> kernel/updates. >>>>>>>> >>>>>>>> Signed-off-by: Saurabh Mohan <saurabh@cplanenetworks.com> >>>>>>> >>>>>>> >>>>>>> >>>>>>> This appears to be correct, but I'm confused about how this could >>>>>>> have >>>>>>> not been noticed for years. Did something change recently? >>>>>> >>>>>> >>>>>> >>>>>> We recently changed it from kernel/ to kernel/updates (prior to v2.4 >>>>>> release), and the commit message suggests it was previously >>>>>> nondeterministic: >>>>>> >>>>>> commit b519432205c36bda5c7331f77a49eaaa919967ad >>>>>> Author: Ansis Atteka <aatteka@nicira.com> >>>>>> Date: Tue May 26 16:49:49 2015 -0700 >>>>>> >>>>>> debian: install openvswitch kernel module under "updates" >>>>>> directory >>>>>> >>>>>> This patch fixes a bug where "modprobe openvswitch" command on >>>>>> Ubuntu >>>>>> distribution would have sometimes tried to load OVS kernel >>>>>> module >>>>>> that >>>>>> shipped together with Linux Kernel, even though one had also >>>>>> installed >>>>>> OVS datapath debian package created with module-assistant. >>>>>> Because >>>>>> of >>>>>> this issue force-reload-kmod command occasionally malfunctioned >>>>>> and >>>>>> failed to load the right kernel module. >>>>>> >>>>>> This bug happened *occasionally* because the default Ubuntu >>>>>> depmod >>>>>> configuration in /etc/depmod.d/ubuntu.conf is set to look for >>>>>> kernel >>>>>> modules first in "updates" directory, then in "ubuntu" directory >>>>>> and >>>>>> then in other directories. If there were two openvswitch.ko >>>>>> modules >>>>>> in "other directories", then modprobe would have loaded kernel >>>>>> module that was nondeterministically listed first by file >>>>>> system. >>>>> >>>>> >>>>> >>>>> OK, I understand why it was nondeterministic before, but where does >>>>> kernel/updates come in then, since it seems to be different from and >>>>> not >>>>> as high-priority as "updates"? Does anyone know? >>>> >>>> >>>> >>>> I am still trying to find the answer in my email history why I ended >>>> up using "kernel/updates" over "updates". >>>> >>>> Saurabh, did you encounter an issue where the wrong kernel module was >>>> loaded or is this to achieve conformance? >>>> >>> >>> Anis, we tried using this patch but still noticed that the wrong kernel >>> module was getting selected. The only way to fix it was >>> to put the module outside kernel/ directory. >> >> >> >> Can you give me locations of all ovs kernel modules (dkms, >> module-assistant and the one that came with linux) present in >> /lib/modules/<linux_version>/ on your system that was having trouble? >> > root@test01-1:/lib/modules# find . -name openvswitch.ko -print > ./3.13.0-32-generic/kernel/net/openvswitch/openvswitch.ko > ./3.13.0-32-generic/updates/openvswitch.ko > >> I am just wondering why my patch prioritized module-assistant created >> kernel module (in kernel/updates) over the one that comes with linux >> kernel itself. I guess this is the problem you are seeing here again, >> right? > > yes, we were seeing the same problem. > my observation was that if we put the module under directory > <kernel>/updates then depmod would still select > <kernel>/net/openvswitch/openvswitch.ko Ok, give me some time to test this on Debian as well, because IIRC Debian does not have /etc/depmod.conf file and I am not sure what is the exact depmod logic there. Assuming it will work there I will push your patch. Also, what Ubuntu distribution are you on? I remember that my patch allegedly helped to solve the undeterministic kernel loading behavior on Ubuntu 14.04. Thanks, for reporting and proposing solution to this problem!
On 10/16/2015 11:32 AM, Ansis Atteka wrote: > On Thu, Oct 15, 2015 at 4:52 PM, Saurabh Mohan > <saurabh@cplanenetworks.com> wrote: >> On 10/15/2015 01:55 PM, Ansis Atteka wrote: >>> >>> On Wed, Oct 14, 2015 at 5:33 PM, Saurabh Mohan >>> <saurabh@cplanenetworks.com> wrote: >>>> >>>> On 10/14/2015 04:58 PM, Ansis Atteka wrote: >>>>> >>>>> >>>>> On Wed, Oct 14, 2015 at 4:08 PM, Ben Pfaff <blp@nicira.com> wrote: >>>>>> >>>>>> >>>>>> On Wed, Oct 14, 2015 at 03:28:24PM -0700, Joe Stringer wrote: >>>>>>> >>>>>>> >>>>>>> On 14 October 2015 at 15:21, Ben Pfaff <blp@nicira.com> wrote: >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Oct 06, 2015 at 04:35:32PM -0700, Saurabh Mohan wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> On Ubuntu depmod's search priority is configured in /etc/depmod to >>>>>>>>> be >>>>>>>>> updates and then the kernel built-in directory. >>>>>>>>> $ cat /etc/depmod.d/ubuntu.conf >>>>>>>>> search updates ubuntu built-in >>>>>>>>> >>>>>>>>> Thus change the placement of openvswitch.ko under updates/ not >>>>>>>>> kernel/updates. >>>>>>>>> >>>>>>>>> Signed-off-by: Saurabh Mohan <saurabh@cplanenetworks.com> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> This appears to be correct, but I'm confused about how this could >>>>>>>> have >>>>>>>> not been noticed for years. Did something change recently? >>>>>>> >>>>>>> >>>>>>> >>>>>>> We recently changed it from kernel/ to kernel/updates (prior to v2.4 >>>>>>> release), and the commit message suggests it was previously >>>>>>> nondeterministic: >>>>>>> >>>>>>> commit b519432205c36bda5c7331f77a49eaaa919967ad >>>>>>> Author: Ansis Atteka <aatteka@nicira.com> >>>>>>> Date: Tue May 26 16:49:49 2015 -0700 >>>>>>> >>>>>>> debian: install openvswitch kernel module under "updates" >>>>>>> directory >>>>>>> >>>>>>> This patch fixes a bug where "modprobe openvswitch" command on >>>>>>> Ubuntu >>>>>>> distribution would have sometimes tried to load OVS kernel >>>>>>> module >>>>>>> that >>>>>>> shipped together with Linux Kernel, even though one had also >>>>>>> installed >>>>>>> OVS datapath debian package created with module-assistant. >>>>>>> Because >>>>>>> of >>>>>>> this issue force-reload-kmod command occasionally malfunctioned >>>>>>> and >>>>>>> failed to load the right kernel module. >>>>>>> >>>>>>> This bug happened *occasionally* because the default Ubuntu >>>>>>> depmod >>>>>>> configuration in /etc/depmod.d/ubuntu.conf is set to look for >>>>>>> kernel >>>>>>> modules first in "updates" directory, then in "ubuntu" directory >>>>>>> and >>>>>>> then in other directories. If there were two openvswitch.ko >>>>>>> modules >>>>>>> in "other directories", then modprobe would have loaded kernel >>>>>>> module that was nondeterministically listed first by file >>>>>>> system. >>>>>> >>>>>> >>>>>> >>>>>> OK, I understand why it was nondeterministic before, but where does >>>>>> kernel/updates come in then, since it seems to be different from and >>>>>> not >>>>>> as high-priority as "updates"? Does anyone know? >>>>> >>>>> >>>>> >>>>> I am still trying to find the answer in my email history why I ended >>>>> up using "kernel/updates" over "updates". >>>>> >>>>> Saurabh, did you encounter an issue where the wrong kernel module was >>>>> loaded or is this to achieve conformance? >>>>> >>>> >>>> Anis, we tried using this patch but still noticed that the wrong kernel >>>> module was getting selected. The only way to fix it was >>>> to put the module outside kernel/ directory. >>> >>> >>> >>> Can you give me locations of all ovs kernel modules (dkms, >>> module-assistant and the one that came with linux) present in >>> /lib/modules/<linux_version>/ on your system that was having trouble? >>> >> root@test01-1:/lib/modules# find . -name openvswitch.ko -print >> ./3.13.0-32-generic/kernel/net/openvswitch/openvswitch.ko >> ./3.13.0-32-generic/updates/openvswitch.ko >> >>> I am just wondering why my patch prioritized module-assistant created >>> kernel module (in kernel/updates) over the one that comes with linux >>> kernel itself. I guess this is the problem you are seeing here again, >>> right? >> >> yes, we were seeing the same problem. >> my observation was that if we put the module under directory >> <kernel>/updates then depmod would still select >> <kernel>/net/openvswitch/openvswitch.ko > > Ok, give me some time to test this on Debian as well, because IIRC > Debian does not have /etc/depmod.conf file and I am not sure what is > the exact depmod logic there. Assuming it will work there I will push > your patch. > thanks. > Also, what Ubuntu distribution are you on? I remember that my patch > allegedly helped to solve the undeterministic kernel loading behavior > on Ubuntu 14.04. I am on Ubuntu 14.04 LTS also. > > Thanks, for reporting and proposing solution to this problem! >
On 10/16/2015 01:55 PM, Saurabh Mohan wrote: > On 10/16/2015 11:32 AM, Ansis Atteka wrote: >> On Thu, Oct 15, 2015 at 4:52 PM, Saurabh Mohan >> <saurabh@cplanenetworks.com> wrote: >>> On 10/15/2015 01:55 PM, Ansis Atteka wrote: >>>> >>>> On Wed, Oct 14, 2015 at 5:33 PM, Saurabh Mohan >>>> <saurabh@cplanenetworks.com> wrote: >>>>> >>>>> On 10/14/2015 04:58 PM, Ansis Atteka wrote: >>>>>> >>>>>> >>>>>> On Wed, Oct 14, 2015 at 4:08 PM, Ben Pfaff <blp@nicira.com> wrote: >>>>>>> >>>>>>> >>>>>>> On Wed, Oct 14, 2015 at 03:28:24PM -0700, Joe Stringer wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 14 October 2015 at 15:21, Ben Pfaff <blp@nicira.com> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Oct 06, 2015 at 04:35:32PM -0700, Saurabh Mohan wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Ubuntu depmod's search priority is configured in /etc/depmod to >>>>>>>>>> be >>>>>>>>>> updates and then the kernel built-in directory. >>>>>>>>>> $ cat /etc/depmod.d/ubuntu.conf >>>>>>>>>> search updates ubuntu built-in >>>>>>>>>> >>>>>>>>>> Thus change the placement of openvswitch.ko under updates/ not >>>>>>>>>> kernel/updates. >>>>>>>>>> >>>>>>>>>> Signed-off-by: Saurabh Mohan <saurabh@cplanenetworks.com> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> This appears to be correct, but I'm confused about how this could >>>>>>>>> have >>>>>>>>> not been noticed for years. Did something change recently? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> We recently changed it from kernel/ to kernel/updates (prior to v2.4 >>>>>>>> release), and the commit message suggests it was previously >>>>>>>> nondeterministic: >>>>>>>> >>>>>>>> commit b519432205c36bda5c7331f77a49eaaa919967ad >>>>>>>> Author: Ansis Atteka <aatteka@nicira.com> >>>>>>>> Date: Tue May 26 16:49:49 2015 -0700 >>>>>>>> >>>>>>>> debian: install openvswitch kernel module under "updates" >>>>>>>> directory >>>>>>>> >>>>>>>> This patch fixes a bug where "modprobe openvswitch" command on >>>>>>>> Ubuntu >>>>>>>> distribution would have sometimes tried to load OVS kernel >>>>>>>> module >>>>>>>> that >>>>>>>> shipped together with Linux Kernel, even though one had also >>>>>>>> installed >>>>>>>> OVS datapath debian package created with module-assistant. >>>>>>>> Because >>>>>>>> of >>>>>>>> this issue force-reload-kmod command occasionally malfunctioned >>>>>>>> and >>>>>>>> failed to load the right kernel module. >>>>>>>> >>>>>>>> This bug happened *occasionally* because the default Ubuntu >>>>>>>> depmod >>>>>>>> configuration in /etc/depmod.d/ubuntu.conf is set to look for >>>>>>>> kernel >>>>>>>> modules first in "updates" directory, then in "ubuntu" directory >>>>>>>> and >>>>>>>> then in other directories. If there were two openvswitch.ko >>>>>>>> modules >>>>>>>> in "other directories", then modprobe would have loaded kernel >>>>>>>> module that was nondeterministically listed first by file >>>>>>>> system. >>>>>>> >>>>>>> >>>>>>> >>>>>>> OK, I understand why it was nondeterministic before, but where does >>>>>>> kernel/updates come in then, since it seems to be different from and >>>>>>> not >>>>>>> as high-priority as "updates"? Does anyone know? >>>>>> >>>>>> >>>>>> >>>>>> I am still trying to find the answer in my email history why I ended >>>>>> up using "kernel/updates" over "updates". >>>>>> >>>>>> Saurabh, did you encounter an issue where the wrong kernel module was >>>>>> loaded or is this to achieve conformance? >>>>>> >>>>> >>>>> Anis, we tried using this patch but still noticed that the wrong kernel >>>>> module was getting selected. The only way to fix it was >>>>> to put the module outside kernel/ directory. >>>> >>>> >>>> >>>> Can you give me locations of all ovs kernel modules (dkms, >>>> module-assistant and the one that came with linux) present in >>>> /lib/modules/<linux_version>/ on your system that was having trouble? >>>> >>> root@test01-1:/lib/modules# find . -name openvswitch.ko -print >>> ./3.13.0-32-generic/kernel/net/openvswitch/openvswitch.ko >>> ./3.13.0-32-generic/updates/openvswitch.ko >>> >>>> I am just wondering why my patch prioritized module-assistant created >>>> kernel module (in kernel/updates) over the one that comes with linux >>>> kernel itself. I guess this is the problem you are seeing here again, >>>> right? >>> >>> yes, we were seeing the same problem. >>> my observation was that if we put the module under directory >>> <kernel>/updates then depmod would still select >>> <kernel>/net/openvswitch/openvswitch.ko >> >> Ok, give me some time to test this on Debian as well, because IIRC >> Debian does not have /etc/depmod.conf file and I am not sure what is >> the exact depmod logic there. Assuming it will work there I will push >> your patch. >> > thanks. i tried debian jessie, uname -r == 3.16.0-4. it does not have /etc/depmod.d/*conf. I was able to confirm that depmod -a does select /lib/modules/<ker-ver>/updates/openvswitch.ko over /lib/moduels/<ker-ver>/kernel/net/openvswitch/openvswitch.ko it looks like the patch would work for both debian and ubuntu. thanks. > >> Also, what Ubuntu distribution are you on? I remember that my patch >> allegedly helped to solve the undeterministic kernel loading behavior >> on Ubuntu 14.04. > > I am on Ubuntu 14.04 LTS also. > >> >> Thanks, for reporting and proposing solution to this problem! >>
On 2 November 2015 at 13:08, Saurabh Mohan <saurabh@cplanenetworks.com> wrote: > On 10/16/2015 01:55 PM, Saurabh Mohan wrote: > >> On 10/16/2015 11:32 AM, Ansis Atteka wrote: >> >>> On Thu, Oct 15, 2015 at 4:52 PM, Saurabh Mohan >>> <saurabh@cplanenetworks.com> wrote: >>> >>>> On 10/15/2015 01:55 PM, Ansis Atteka wrote: >>>> >>>>> >>>>> On Wed, Oct 14, 2015 at 5:33 PM, Saurabh Mohan >>>>> <saurabh@cplanenetworks.com> wrote: >>>>> >>>>>> >>>>>> On 10/14/2015 04:58 PM, Ansis Atteka wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Oct 14, 2015 at 4:08 PM, Ben Pfaff <blp@nicira.com> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Oct 14, 2015 at 03:28:24PM -0700, Joe Stringer wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 14 October 2015 at 15:21, Ben Pfaff <blp@nicira.com> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Oct 06, 2015 at 04:35:32PM -0700, Saurabh Mohan wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Ubuntu depmod's search priority is configured in /etc/depmod >>>>>>>>>>> to >>>>>>>>>>> be >>>>>>>>>>> updates and then the kernel built-in directory. >>>>>>>>>>> $ cat /etc/depmod.d/ubuntu.conf >>>>>>>>>>> search updates ubuntu built-in >>>>>>>>>>> >>>>>>>>>>> Thus change the placement of openvswitch.ko under updates/ not >>>>>>>>>>> kernel/updates. >>>>>>>>>>> >>>>>>>>>>> Signed-off-by: Saurabh Mohan <saurabh@cplanenetworks.com> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> This appears to be correct, but I'm confused about how this could >>>>>>>>>> have >>>>>>>>>> not been noticed for years. Did something change recently? >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> We recently changed it from kernel/ to kernel/updates (prior to >>>>>>>>> v2.4 >>>>>>>>> release), and the commit message suggests it was previously >>>>>>>>> nondeterministic: >>>>>>>>> >>>>>>>>> commit b519432205c36bda5c7331f77a49eaaa919967ad >>>>>>>>> Author: Ansis Atteka <aatteka@nicira.com> >>>>>>>>> Date: Tue May 26 16:49:49 2015 -0700 >>>>>>>>> >>>>>>>>> debian: install openvswitch kernel module under "updates" >>>>>>>>> directory >>>>>>>>> >>>>>>>>> This patch fixes a bug where "modprobe openvswitch" >>>>>>>>> command on >>>>>>>>> Ubuntu >>>>>>>>> distribution would have sometimes tried to load OVS kernel >>>>>>>>> module >>>>>>>>> that >>>>>>>>> shipped together with Linux Kernel, even though one had >>>>>>>>> also >>>>>>>>> installed >>>>>>>>> OVS datapath debian package created with module-assistant. >>>>>>>>> Because >>>>>>>>> of >>>>>>>>> this issue force-reload-kmod command occasionally >>>>>>>>> malfunctioned >>>>>>>>> and >>>>>>>>> failed to load the right kernel module. >>>>>>>>> >>>>>>>>> This bug happened *occasionally* because the default Ubuntu >>>>>>>>> depmod >>>>>>>>> configuration in /etc/depmod.d/ubuntu.conf is set to look >>>>>>>>> for >>>>>>>>> kernel >>>>>>>>> modules first in "updates" directory, then in "ubuntu" >>>>>>>>> directory >>>>>>>>> and >>>>>>>>> then in other directories. If there were two >>>>>>>>> openvswitch.ko >>>>>>>>> modules >>>>>>>>> in "other directories", then modprobe would have loaded >>>>>>>>> kernel >>>>>>>>> module that was nondeterministically listed first by file >>>>>>>>> system. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> OK, I understand why it was nondeterministic before, but where does >>>>>>>> kernel/updates come in then, since it seems to be different from and >>>>>>>> not >>>>>>>> as high-priority as "updates"? Does anyone know? >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> I am still trying to find the answer in my email history why I ended >>>>>>> up using "kernel/updates" over "updates". >>>>>>> >>>>>>> Saurabh, did you encounter an issue where the wrong kernel module was >>>>>>> loaded or is this to achieve conformance? >>>>>>> >>>>>>> >>>>>> Anis, we tried using this patch but still noticed that the wrong >>>>>> kernel >>>>>> module was getting selected. The only way to fix it was >>>>>> to put the module outside kernel/ directory. >>>>>> >>>>> >>>>> >>>>> >>>>> Can you give me locations of all ovs kernel modules (dkms, >>>>> module-assistant and the one that came with linux) present in >>>>> /lib/modules/<linux_version>/ on your system that was having trouble? >>>>> >>>>> root@test01-1:/lib/modules# find . -name openvswitch.ko -print >>>> ./3.13.0-32-generic/kernel/net/openvswitch/openvswitch.ko >>>> ./3.13.0-32-generic/updates/openvswitch.ko >>>> >>>> I am just wondering why my patch prioritized module-assistant created >>>>> kernel module (in kernel/updates) over the one that comes with linux >>>>> kernel itself. I guess this is the problem you are seeing here again, >>>>> right? >>>>> >>>> >>>> yes, we were seeing the same problem. >>>> my observation was that if we put the module under directory >>>> <kernel>/updates then depmod would still select >>>> <kernel>/net/openvswitch/openvswitch.ko >>>> >>> >>> Ok, give me some time to test this on Debian as well, because IIRC >>> Debian does not have /etc/depmod.conf file and I am not sure what is >>> the exact depmod logic there. Assuming it will work there I will push >>> your patch. >>> >>> thanks. >> > > i tried debian jessie, uname -r == 3.16.0-4. it does not have > /etc/depmod.d/*conf. > I was able to confirm that depmod -a does select > /lib/modules/<ker-ver>/updates/openvswitch.ko > over /lib/moduels/<ker-ver>/kernel/net/openvswitch/openvswitch.ko > > it looks like the patch would work for both debian and ubuntu. > > thanks. > Thanks for contributing. I added you to AUTHORS file and pushed your patch back to branch 2.1
diff --git a/debian/rules.modules b/debian/rules.modules index 6571683..7ea8b1b 100755 --- a/debian/rules.modules +++ b/debian/rules.modules @@ -17,7 +17,7 @@ kdist_clean: kdist_config: prep-deb-files .PHONY: binary-modules -binary-modules: DSTDIR = $(CURDIR)/debian/$(PKGNAME)/lib/modules/$(KVERS)/kernel/updates +binary-modules: DSTDIR = $(CURDIR)/debian/$(PKGNAME)/lib/modules/$(KVERS)/updates binary-modules: prep-deb-files dh_testdir dh_testroot
On Ubuntu depmod's search priority is configured in /etc/depmod to be updates and then the kernel built-in directory. $ cat /etc/depmod.d/ubuntu.conf search updates ubuntu built-in Thus change the placement of openvswitch.ko under updates/ not kernel/updates. Signed-off-by: Saurabh Mohan <saurabh@cplanenetworks.com> --- debian/rules.modules | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)