| Message ID | 20190923090249.127984-1-brendanhiggins@google.com |
|---|---|
| Headers | show |
| Series | kunit: introduce KUnit, the Linux kernel unit testing framework | expand |
On Mon, Sep 23, 2019 at 02:02:30AM -0700, Brendan Higgins wrote: > ## TL;DR > > This revision addresses comments from Linus[1] and Randy[2], by moving > top level `kunit/` directory to `lib/kunit/` and likewise moves top > level Kconfig entry under lib/Kconfig.debug, so the KUnit submenu now > shows up under the "Kernel Hacking" menu. This question is primarily directed at Shuah and Linus.... What's the current status of the kunit series now that Brendan has moved it out of the top-level kunit directory as Linus has requested? There doesn't appear to have been many comments or changes since since September 23rd, and I was very much hoping they could land before -rc2, since I've been hoping to add unit tests for ext4. Is kunit likely to be able to be landed in Linus's tree during this development cycle? Many thanks! - Ted
On Fri, Oct 4, 2019 at 2:39 PM Theodore Y. Ts'o <tytso@mit.edu> wrote: > > This question is primarily directed at Shuah and Linus.... > > What's the current status of the kunit series now that Brendan has > moved it out of the top-level kunit directory as Linus has requested? We seemed to decide to just wait for 5.5, but there is nothing that looks to block that. And I encouraged Shuah to find more kunit cases for when it _does_ get merged. So if the kunit branch is stable, and people want to start using it for their unit tests, then I think that would be a good idea, and then during the 5.5 merge window we'll not just get the infrastructure, we'll get a few more users too and not just examples. Linus
On 10/4/19 3:42 PM, Linus Torvalds wrote: > On Fri, Oct 4, 2019 at 2:39 PM Theodore Y. Ts'o <tytso@mit.edu> wrote: >> >> This question is primarily directed at Shuah and Linus.... >> >> What's the current status of the kunit series now that Brendan has >> moved it out of the top-level kunit directory as Linus has requested? > The move happened smack in the middle of merge window and landed in linux-next towards the end of the merge window. > We seemed to decide to just wait for 5.5, but there is nothing that > looks to block that. And I encouraged Shuah to find more kunit cases > for when it _does_ get merged. > Right. I communicated that to Brendan that we could work on adding more kunit based tests which would help get more mileage on the kunit. > So if the kunit branch is stable, and people want to start using it > for their unit tests, then I think that would be a good idea, and then > during the 5.5 merge window we'll not just get the infrastructure, > we'll get a few more users too and not just examples. > thanks, -- Shuah
On Fri, Oct 04, 2019 at 03:59:10PM -0600, shuah wrote: > On 10/4/19 3:42 PM, Linus Torvalds wrote: > > On Fri, Oct 4, 2019 at 2:39 PM Theodore Y. Ts'o <tytso@mit.edu> wrote: > > > > > > This question is primarily directed at Shuah and Linus.... > > > > > > What's the current status of the kunit series now that Brendan has > > > moved it out of the top-level kunit directory as Linus has requested? > > > > The move happened smack in the middle of merge window and landed in > linux-next towards the end of the merge window. > > > We seemed to decide to just wait for 5.5, but there is nothing that > > looks to block that. And I encouraged Shuah to find more kunit cases > > for when it _does_ get merged. > > > > Right. I communicated that to Brendan that we could work on adding more > kunit based tests which would help get more mileage on the kunit. > > > So if the kunit branch is stable, and people want to start using it > > for their unit tests, then I think that would be a good idea, and then > > during the 5.5 merge window we'll not just get the infrastructure, > > we'll get a few more users too and not just examples. I was planning on holding off on accepting more tests/changes until KUnit is in torvalds/master. As much as I would like to go around promoting it, I don't really want to promote too much complexity in a non-upstream branch before getting it upstream because I don't want to risk adding something that might cause it to get rejected again. To be clear, I can understand from your perspective why getting more tests/usage before accepting it is a good thing. The more people that play around with it, the more likely that someone will find an issue with it, and more likely that what is accepted into torvalds/master is of high quality. However, if I encourage arbitrary tests/improvements into my KUnit branch, it further diverges away from torvalds/master, and is more likely that there will be a merge conflict or issue that is not related to the core KUnit changes that will cause the whole thing to be rejected again in v5.5. I don't know. I guess we could maybe address that situation by splitting up the pull request into features and tests when we go to send it in, but that seems to invite a lot of unnecessary complexity. I actually already had some other tests/changes ready to send for review, but was holding off until the initial set of patches mad it in. Looking forward to hearing other people's thoughts.
On 10/4/19 4:27 PM, Brendan Higgins wrote: > On Fri, Oct 04, 2019 at 03:59:10PM -0600, shuah wrote: >> On 10/4/19 3:42 PM, Linus Torvalds wrote: >>> On Fri, Oct 4, 2019 at 2:39 PM Theodore Y. Ts'o <tytso@mit.edu> wrote: >>>> >>>> This question is primarily directed at Shuah and Linus.... >>>> >>>> What's the current status of the kunit series now that Brendan has >>>> moved it out of the top-level kunit directory as Linus has requested? >>> >> >> The move happened smack in the middle of merge window and landed in >> linux-next towards the end of the merge window. >> >>> We seemed to decide to just wait for 5.5, but there is nothing that >>> looks to block that. And I encouraged Shuah to find more kunit cases >>> for when it _does_ get merged. >>> >> >> Right. I communicated that to Brendan that we could work on adding more >> kunit based tests which would help get more mileage on the kunit. >> >>> So if the kunit branch is stable, and people want to start using it >>> for their unit tests, then I think that would be a good idea, and then >>> during the 5.5 merge window we'll not just get the infrastructure, >>> we'll get a few more users too and not just examples. > > I was planning on holding off on accepting more tests/changes until > KUnit is in torvalds/master. As much as I would like to go around > promoting it, I don't really want to promote too much complexity in a > non-upstream branch before getting it upstream because I don't want to > risk adding something that might cause it to get rejected again. > > To be clear, I can understand from your perspective why getting more > tests/usage before accepting it is a good thing. The more people that > play around with it, the more likely that someone will find an issue > with it, and more likely that what is accepted into torvalds/master is > of high quality. > > However, if I encourage arbitrary tests/improvements into my KUnit > branch, it further diverges away from torvalds/master, and is more > likely that there will be a merge conflict or issue that is not related > to the core KUnit changes that will cause the whole thing to be > rejected again in v5.5. > The idea is that the new development will happen based on kunit in linux-kselftest next. It will work just fine. As we accepts patches, they will go on top of kunit that is in linux-next now. thanks, -- Shuah
On Fri, Oct 4, 2019 at 3:47 PM shuah <shuah@kernel.org> wrote: > > On 10/4/19 4:27 PM, Brendan Higgins wrote: > > On Fri, Oct 04, 2019 at 03:59:10PM -0600, shuah wrote: > >> On 10/4/19 3:42 PM, Linus Torvalds wrote: > >>> On Fri, Oct 4, 2019 at 2:39 PM Theodore Y. Ts'o <tytso@mit.edu> wrote: > >>>> > >>>> This question is primarily directed at Shuah and Linus.... > >>>> > >>>> What's the current status of the kunit series now that Brendan has > >>>> moved it out of the top-level kunit directory as Linus has requested? > >>> > >> > >> The move happened smack in the middle of merge window and landed in > >> linux-next towards the end of the merge window. > >> > >>> We seemed to decide to just wait for 5.5, but there is nothing that > >>> looks to block that. And I encouraged Shuah to find more kunit cases > >>> for when it _does_ get merged. > >>> > >> > >> Right. I communicated that to Brendan that we could work on adding more > >> kunit based tests which would help get more mileage on the kunit. > >> > >>> So if the kunit branch is stable, and people want to start using it > >>> for their unit tests, then I think that would be a good idea, and then > >>> during the 5.5 merge window we'll not just get the infrastructure, > >>> we'll get a few more users too and not just examples. > > > > I was planning on holding off on accepting more tests/changes until > > KUnit is in torvalds/master. As much as I would like to go around > > promoting it, I don't really want to promote too much complexity in a > > non-upstream branch before getting it upstream because I don't want to > > risk adding something that might cause it to get rejected again. > > > > To be clear, I can understand from your perspective why getting more > > tests/usage before accepting it is a good thing. The more people that > > play around with it, the more likely that someone will find an issue > > with it, and more likely that what is accepted into torvalds/master is > > of high quality. > > > > However, if I encourage arbitrary tests/improvements into my KUnit > > branch, it further diverges away from torvalds/master, and is more > > likely that there will be a merge conflict or issue that is not related > > to the core KUnit changes that will cause the whole thing to be > > rejected again in v5.5. > > > > The idea is that the new development will happen based on kunit in > linux-kselftest next. It will work just fine. As we accepts patches, > they will go on top of kunit that is in linux-next now. But then wouldn't we want to limit what KUnit changes are going into linux-kselftest next for v5.5? For example, we probably don't want to do anymore feature development on it until it is in v5.5, since the goal is to make it more stable, right? I am guessing that it will probably be fine, but it still sounds like we need to establish some ground rules, and play it *very* safe.
On 10/4/19 5:10 PM, Brendan Higgins wrote: > On Fri, Oct 4, 2019 at 3:47 PM shuah <shuah@kernel.org> wrote: >> >> On 10/4/19 4:27 PM, Brendan Higgins wrote: >>> On Fri, Oct 04, 2019 at 03:59:10PM -0600, shuah wrote: >>>> On 10/4/19 3:42 PM, Linus Torvalds wrote: >>>>> On Fri, Oct 4, 2019 at 2:39 PM Theodore Y. Ts'o <tytso@mit.edu> wrote: >>>>>> >>>>>> This question is primarily directed at Shuah and Linus.... >>>>>> >>>>>> What's the current status of the kunit series now that Brendan has >>>>>> moved it out of the top-level kunit directory as Linus has requested? >>>>> >>>> >>>> The move happened smack in the middle of merge window and landed in >>>> linux-next towards the end of the merge window. >>>> >>>>> We seemed to decide to just wait for 5.5, but there is nothing that >>>>> looks to block that. And I encouraged Shuah to find more kunit cases >>>>> for when it _does_ get merged. >>>>> >>>> >>>> Right. I communicated that to Brendan that we could work on adding more >>>> kunit based tests which would help get more mileage on the kunit. >>>> >>>>> So if the kunit branch is stable, and people want to start using it >>>>> for their unit tests, then I think that would be a good idea, and then >>>>> during the 5.5 merge window we'll not just get the infrastructure, >>>>> we'll get a few more users too and not just examples. >>> >>> I was planning on holding off on accepting more tests/changes until >>> KUnit is in torvalds/master. As much as I would like to go around >>> promoting it, I don't really want to promote too much complexity in a >>> non-upstream branch before getting it upstream because I don't want to >>> risk adding something that might cause it to get rejected again. >>> >>> To be clear, I can understand from your perspective why getting more >>> tests/usage before accepting it is a good thing. The more people that >>> play around with it, the more likely that someone will find an issue >>> with it, and more likely that what is accepted into torvalds/master is >>> of high quality. >>> >>> However, if I encourage arbitrary tests/improvements into my KUnit >>> branch, it further diverges away from torvalds/master, and is more >>> likely that there will be a merge conflict or issue that is not related >>> to the core KUnit changes that will cause the whole thing to be >>> rejected again in v5.5. >>> >> >> The idea is that the new development will happen based on kunit in >> linux-kselftest next. It will work just fine. As we accepts patches, >> they will go on top of kunit that is in linux-next now. > > But then wouldn't we want to limit what KUnit changes are going into > linux-kselftest next for v5.5? For example, we probably don't want to > do anymore feature development on it until it is in v5.5, since the > goal is to make it more stable, right? > > I am guessing that it will probably be fine, but it still sounds like > we need to establish some ground rules, and play it *very* safe. > How about we identify a small number tests that can add value and focus on them. I am thinking a number between 2 and 5. This way we get a feel for the API, if it changes for the better great, if it doesn't have to, then you know you already did a great job. Does that sound reasonable to you? thanks, -- Shuah
On Fri, Oct 04, 2019 at 04:47:09PM -0600, shuah wrote: > > However, if I encourage arbitrary tests/improvements into my KUnit > > branch, it further diverges away from torvalds/master, and is more > > likely that there will be a merge conflict or issue that is not related > > to the core KUnit changes that will cause the whole thing to be > > rejected again in v5.5. > > The idea is that the new development will happen based on kunit in > linux-kselftest next. It will work just fine. As we accepts patches, > they will go on top of kunit that is in linux-next now. I don't see how this would work. If I add unit tests to ext4, they would be in fs/ext4. And to the extent that I need to add test mocks to allow the unit tests to work, they will involve changes to existing files in fs/ext4. I can't put them in the ext4.git tree without pulling in the kunit changes into the ext4 git tree. And if they ext4 unit tests land in the kunit tree, it would be a receipe for large numbers of merge conflicts. - Ted
On Fri, Oct 4, 2019 at 4:30 PM Theodore Y. Ts'o <tytso@mit.edu> wrote: > > On Fri, Oct 04, 2019 at 04:47:09PM -0600, shuah wrote: > > > However, if I encourage arbitrary tests/improvements into my KUnit > > > branch, it further diverges away from torvalds/master, and is more > > > likely that there will be a merge conflict or issue that is not related > > > to the core KUnit changes that will cause the whole thing to be > > > rejected again in v5.5. > > > > The idea is that the new development will happen based on kunit in > > linux-kselftest next. It will work just fine. As we accepts patches, > > they will go on top of kunit that is in linux-next now. > > I don't see how this would work. If I add unit tests to ext4, they > would be in fs/ext4. And to the extent that I need to add test mocks > to allow the unit tests to work, they will involve changes to existing > files in fs/ext4. I can't put them in the ext4.git tree without > pulling in the kunit changes into the ext4 git tree. And if they ext4 > unit tests land in the kunit tree, it would be a receipe for large > numbers of merge conflicts. That's where I was originally coming from. So here's a dumb idea: what if we merged KUnit through the ext4 tree? I imagine that could potentially get very confusing when we go back to sending changes in through the kselftest tree, but at least it means that ext4 can use it in the meantime, which means that it at least gets to be useful to one group of people. Also, since Ted seems pretty keen on using this, I imagine it is much more likely to produce real world, immediately useful tests not written by me (I'm not being lazy, I just think it is better to get other people's experiences other than my own). Thoughts?
On 10/4/19 5:52 PM, Brendan Higgins wrote: > On Fri, Oct 4, 2019 at 4:30 PM Theodore Y. Ts'o <tytso@mit.edu> wrote: >> >> On Fri, Oct 04, 2019 at 04:47:09PM -0600, shuah wrote: >>>> However, if I encourage arbitrary tests/improvements into my KUnit >>>> branch, it further diverges away from torvalds/master, and is more >>>> likely that there will be a merge conflict or issue that is not related >>>> to the core KUnit changes that will cause the whole thing to be >>>> rejected again in v5.5. >>> >>> The idea is that the new development will happen based on kunit in >>> linux-kselftest next. It will work just fine. As we accepts patches, >>> they will go on top of kunit that is in linux-next now. >> >> I don't see how this would work. If I add unit tests to ext4, they >> would be in fs/ext4. And to the extent that I need to add test mocks >> to allow the unit tests to work, they will involve changes to existing >> files in fs/ext4. I can't put them in the ext4.git tree without >> pulling in the kunit changes into the ext4 git tree. And if they ext4 >> unit tests land in the kunit tree, it would be a receipe for large >> numbers of merge conflicts. > > That's where I was originally coming from. > > So here's a dumb idea: what if we merged KUnit through the ext4 tree? > I imagine that could potentially get very confusing when we go back to > sending changes in through the kselftest tree, but at least it means > that ext4 can use it in the meantime, which means that it at least > gets to be useful to one group of people. Also, since Ted seems pretty > keen on using this, I imagine it is much more likely to produce real > world, immediately useful tests not written by me (I'm not being lazy, > I just think it is better to get other people's experiences other than > my own). > That doesn't make sense does it? The tests might not be limited to fs/ext4. We might have other sub-systems that add tests. So, we will have to work to make linux-next as the base for new development and limit the number of tests to where it will be easier work in this mode for 5.5. We can stage the pull requests so that kunit lands first followed by tests. We have a similar situation with kselftest as well. Sub-systems send tests that depend on their tress separately. thanks, -- Shuah
On Fri, Oct 4, 2019 at 4:57 PM shuah <shuah@kernel.org> wrote: > > On 10/4/19 5:52 PM, Brendan Higgins wrote: > > On Fri, Oct 4, 2019 at 4:30 PM Theodore Y. Ts'o <tytso@mit.edu> wrote: > >> > >> On Fri, Oct 04, 2019 at 04:47:09PM -0600, shuah wrote: > >>>> However, if I encourage arbitrary tests/improvements into my KUnit > >>>> branch, it further diverges away from torvalds/master, and is more > >>>> likely that there will be a merge conflict or issue that is not related > >>>> to the core KUnit changes that will cause the whole thing to be > >>>> rejected again in v5.5. > >>> > >>> The idea is that the new development will happen based on kunit in > >>> linux-kselftest next. It will work just fine. As we accepts patches, > >>> they will go on top of kunit that is in linux-next now. > >> > >> I don't see how this would work. If I add unit tests to ext4, they > >> would be in fs/ext4. And to the extent that I need to add test mocks > >> to allow the unit tests to work, they will involve changes to existing > >> files in fs/ext4. I can't put them in the ext4.git tree without > >> pulling in the kunit changes into the ext4 git tree. And if they ext4 > >> unit tests land in the kunit tree, it would be a receipe for large > >> numbers of merge conflicts. > > > > That's where I was originally coming from. > > > > So here's a dumb idea: what if we merged KUnit through the ext4 tree? > > I imagine that could potentially get very confusing when we go back to > > sending changes in through the kselftest tree, but at least it means > > that ext4 can use it in the meantime, which means that it at least > > gets to be useful to one group of people. Also, since Ted seems pretty > > keen on using this, I imagine it is much more likely to produce real > > world, immediately useful tests not written by me (I'm not being lazy, > > I just think it is better to get other people's experiences other than > > my own). > > > > That doesn't make sense does it? The tests might not be limited to > fs/ext4. We might have other sub-systems that add tests. Well, I have some small isolated examples that I think would probably work no matter what, so we can get some usage outside of ext4. Also, if we want to limit the number of tests, then we don't expect to get much usage outside of ext4 before v5.5 anyway. I just figure, it's better to make it work for one person than no one, right? In any case, I admit it is not a great idea. I just thought it had some interesting advantages over going in through linux-kselftest that were worth exploring. > So, we will have to work to make linux-next as the base for new > development and limit the number of tests to where it will be > easier work in this mode for 5.5. We can stage the pull requests > so that kunit lands first followed by tests. So we are going to encourage maintainers to allow tests in their tree based on KUnit on the assumption that KUnit will get merged before there changes? That sounds like a huge burden, not just on us, but on other maintainers and users. I think if we are going to allow tests before KUnit is merged, we should have the tests come in through the same tree as KUnit. > We have a similar situation with kselftest as well. Sub-systems > send tests that depend on their tress separately. Well it is different if the maintainer wants to send the test in through their tree, right? Otherwise, it won't matter what the state of linux-next is and it won't matter when linux-kselftest gets merged. Or am I not understanding you?
On 10/4/19 6:33 PM, Brendan Higgins wrote: > On Fri, Oct 4, 2019 at 4:57 PM shuah <shuah@kernel.org> wrote: >> >> On 10/4/19 5:52 PM, Brendan Higgins wrote: >>> On Fri, Oct 4, 2019 at 4:30 PM Theodore Y. Ts'o <tytso@mit.edu> wrote: >>>> >>>> On Fri, Oct 04, 2019 at 04:47:09PM -0600, shuah wrote: >>>>>> However, if I encourage arbitrary tests/improvements into my KUnit >>>>>> branch, it further diverges away from torvalds/master, and is more >>>>>> likely that there will be a merge conflict or issue that is not related >>>>>> to the core KUnit changes that will cause the whole thing to be >>>>>> rejected again in v5.5. >>>>> >>>>> The idea is that the new development will happen based on kunit in >>>>> linux-kselftest next. It will work just fine. As we accepts patches, >>>>> they will go on top of kunit that is in linux-next now. >>>> >>>> I don't see how this would work. If I add unit tests to ext4, they >>>> would be in fs/ext4. And to the extent that I need to add test mocks >>>> to allow the unit tests to work, they will involve changes to existing >>>> files in fs/ext4. I can't put them in the ext4.git tree without >>>> pulling in the kunit changes into the ext4 git tree. And if they ext4 >>>> unit tests land in the kunit tree, it would be a receipe for large >>>> numbers of merge conflicts. >>> >>> That's where I was originally coming from. >>> >>> So here's a dumb idea: what if we merged KUnit through the ext4 tree? >>> I imagine that could potentially get very confusing when we go back to >>> sending changes in through the kselftest tree, but at least it means >>> that ext4 can use it in the meantime, which means that it at least >>> gets to be useful to one group of people. Also, since Ted seems pretty >>> keen on using this, I imagine it is much more likely to produce real >>> world, immediately useful tests not written by me (I'm not being lazy, >>> I just think it is better to get other people's experiences other than >>> my own). >>> >> >> That doesn't make sense does it? The tests might not be limited to >> fs/ext4. We might have other sub-systems that add tests. > > Well, I have some small isolated examples that I think would probably > work no matter what, so we can get some usage outside of ext4. Also, > if we want to limit the number of tests, then we don't expect to get > much usage outside of ext4 before v5.5 anyway. I just figure, it's > better to make it work for one person than no one, right? > > In any case, I admit it is not a great idea. I just thought it had > some interesting advantages over going in through linux-kselftest that > were worth exploring. > >> So, we will have to work to make linux-next as the base for new >> development and limit the number of tests to where it will be >> easier work in this mode for 5.5. We can stage the pull requests >> so that kunit lands first followed by tests. > > So we are going to encourage maintainers to allow tests in their tree > based on KUnit on the assumption that KUnit will get merged before > there changes? That sounds like a huge burden, not just on us, but on > other maintainers and users. > > I think if we are going to allow tests before KUnit is merged, we > should have the tests come in through the same tree as KUnit. > >> We have a similar situation with kselftest as well. Sub-systems >> send tests that depend on their tress separately. > > Well it is different if the maintainer wants to send the test in > through their tree, right? Otherwise, it won't matter what the state > of linux-next is and it won't matter when linux-kselftest gets merged. > Or am I not understanding you? > Let's talk about current state. Right now kunit is in linux-next and we want to add a few more tests. We will have to coordinate the effort. Once kunit get into mainline, then the need for this coordination goes down. Let's focus on the next few weeks first so we can get this into mainline in 5.5. The two of us can chat next week and come up with a plan. thanks, -- Shuah
On Fri, Oct 4, 2019 at 5:49 PM shuah <shuah@kernel.org> wrote: > > On 10/4/19 6:33 PM, Brendan Higgins wrote: > > On Fri, Oct 4, 2019 at 4:57 PM shuah <shuah@kernel.org> wrote: > >> > >> On 10/4/19 5:52 PM, Brendan Higgins wrote: > >>> On Fri, Oct 4, 2019 at 4:30 PM Theodore Y. Ts'o <tytso@mit.edu> wrote: > >>>> > >>>> On Fri, Oct 04, 2019 at 04:47:09PM -0600, shuah wrote: > >>>>>> However, if I encourage arbitrary tests/improvements into my KUnit > >>>>>> branch, it further diverges away from torvalds/master, and is more > >>>>>> likely that there will be a merge conflict or issue that is not related > >>>>>> to the core KUnit changes that will cause the whole thing to be > >>>>>> rejected again in v5.5. > >>>>> > >>>>> The idea is that the new development will happen based on kunit in > >>>>> linux-kselftest next. It will work just fine. As we accepts patches, > >>>>> they will go on top of kunit that is in linux-next now. > >>>> > >>>> I don't see how this would work. If I add unit tests to ext4, they > >>>> would be in fs/ext4. And to the extent that I need to add test mocks > >>>> to allow the unit tests to work, they will involve changes to existing > >>>> files in fs/ext4. I can't put them in the ext4.git tree without > >>>> pulling in the kunit changes into the ext4 git tree. And if they ext4 > >>>> unit tests land in the kunit tree, it would be a receipe for large > >>>> numbers of merge conflicts. > >>> > >>> That's where I was originally coming from. > >>> > >>> So here's a dumb idea: what if we merged KUnit through the ext4 tree? > >>> I imagine that could potentially get very confusing when we go back to > >>> sending changes in through the kselftest tree, but at least it means > >>> that ext4 can use it in the meantime, which means that it at least > >>> gets to be useful to one group of people. Also, since Ted seems pretty > >>> keen on using this, I imagine it is much more likely to produce real > >>> world, immediately useful tests not written by me (I'm not being lazy, > >>> I just think it is better to get other people's experiences other than > >>> my own). > >>> > >> > >> That doesn't make sense does it? The tests might not be limited to > >> fs/ext4. We might have other sub-systems that add tests. > > > > Well, I have some small isolated examples that I think would probably > > work no matter what, so we can get some usage outside of ext4. Also, > > if we want to limit the number of tests, then we don't expect to get > > much usage outside of ext4 before v5.5 anyway. I just figure, it's > > better to make it work for one person than no one, right? > > > > In any case, I admit it is not a great idea. I just thought it had > > some interesting advantages over going in through linux-kselftest that > > were worth exploring. > > > >> So, we will have to work to make linux-next as the base for new > >> development and limit the number of tests to where it will be > >> easier work in this mode for 5.5. We can stage the pull requests > >> so that kunit lands first followed by tests. > > > > So we are going to encourage maintainers to allow tests in their tree > > based on KUnit on the assumption that KUnit will get merged before > > there changes? That sounds like a huge burden, not just on us, but on > > other maintainers and users. > > > > I think if we are going to allow tests before KUnit is merged, we > > should have the tests come in through the same tree as KUnit. > > > >> We have a similar situation with kselftest as well. Sub-systems > >> send tests that depend on their tress separately. > > > > Well it is different if the maintainer wants to send the test in > > through their tree, right? Otherwise, it won't matter what the state > > of linux-next is and it won't matter when linux-kselftest gets merged. > > Or am I not understanding you? > > > > Let's talk about current state. Right now kunit is in linux-next and > we want to add a few more tests. We will have to coordinate the effort. > Once kunit get into mainline, then the need for this coordination goes > down. Sure, I was just thinking that getting other people to write the tests would be better. Since not only is it then useful to someone else, it provides the best possible exercise of KUnit. Nevertheless, it would probably just be easier to get a handful of example tests, and it is less likely to result in any issues for v5.5, so that's probably better. (I think that is what you are hinting at here. ;-) ) Hey Ted, do you know if that ext4 timestamp test can go in through linux-kselftest? It seemed fairly self-contained. Or is that what you were saying wouldn't work for you? > Let's focus on the next few weeks first so we can get this into mainline > in 5.5. I agree. That is the most important thing. > The two of us can chat next week and come up with a plan. Sure. Cheers!
On Fri, Oct 04, 2019 at 06:18:04PM -0700, Brendan Higgins wrote: > > Let's talk about current state. Right now kunit is in linux-next and > > we want to add a few more tests. We will have to coordinate the effort. > > Once kunit get into mainline, then the need for this coordination goes > > down. > > Sure, I was just thinking that getting other people to write the tests > would be better. Since not only is it then useful to someone else, it > provides the best possible exercise of KUnit. Well, one thing we *can* do is if (a) if we can create a kselftest branch which we know is stable and won't change, and (b) we can get assurances that Linus *will* accept that branch during the next merge window, those subsystems which want to use kself test can simply pull it into their tree. We've done this before in the file system world, when there has been some common set of changes needed to improve, say, Direct I/O, where the changes are put into a standalone branch, say, in the xfs tree, and those file systems which need it as a building block can pull it into their tree, and then add the changes needed to use those changes into their file system git tree. These changes are generally not terribly controversial, and we've not had to worry about people want to bikeshed the changes. There is a risk with doing this of course, which is that if the branch *is* controversial, or gets bike-shedded for some reason, then Linus gets upset and any branches which depended on said branch will get rejected at the next merge window. Which is the requirement for (a) and (b) above. Presumably, the fact that people were unwilling to let Kunit land during this merge window might will *because* we think more changes might be pending? The other thing I suppose I can do is to let the ext4 kunit tests land in ext4 tree, but with the necessary #ifdef's around things like "#include <kunit/test.h>" so that the build won't blow up w/o kunit changes being in the tree that I'm building. It means I won't be able to run the tests without creating a test integration branch and merging in kunit by hand, which will super-annoying, of course. And if some of the bike-shedding is in Kunit's interfaces, then that becomes problematic as well, since any tests that are in ext4.git tree might change if people want to rename Kunit's publically exported functions (for example). > Hey Ted, do you know if that ext4 timestamp test can go in through > linux-kselftest? It seemed fairly self-contained. Or is that what you > were saying wouldn't work for you? Well, I was hoping that we might start creating more tests beyond just the ext4 timestamp tests.... - Ted
On Sun, Oct 6, 2019 at 9:55 AM Theodore Y. Ts'o <tytso@mit.edu> wrote: > > Well, one thing we *can* do is if (a) if we can create a kselftest > branch which we know is stable and won't change, and (b) we can get > assurances that Linus *will* accept that branch during the next merge > window, those subsystems which want to use kself test can simply pull > it into their tree. Yes. At the same time, I don't think it needs to be even that fancy. Even if it's not a stable branch that gets shared between different developers, it would be good to just have people do a "let's try this" throw-away branch to use the kunit functionality and verify that "yeah, this is fairly convenient for ext4". It doesn't have to be merged in that form, but just confirmation that the infrastructure is helpful before it gets merged would be good. Linus
On Sun, Oct 6, 2019 at 9:55 AM Theodore Y. Ts'o <tytso@mit.edu> wrote: > > On Fri, Oct 04, 2019 at 06:18:04PM -0700, Brendan Higgins wrote: > > > Let's talk about current state. Right now kunit is in linux-next and > > > we want to add a few more tests. We will have to coordinate the effort. > > > Once kunit get into mainline, then the need for this coordination goes > > > down. > > > > Sure, I was just thinking that getting other people to write the tests > > would be better. Since not only is it then useful to someone else, it > > provides the best possible exercise of KUnit. > > Well, one thing we *can* do is if (a) if we can create a kselftest > branch which we know is stable and won't change, and (b) we can get > assurances that Linus *will* accept that branch during the next merge > window, those subsystems which want to use kself test can simply pull > it into their tree. Yeah, I can't think of any reason that you haven't outlined already why that might not work, but that seems kind of like circumventing Linus. > We've done this before in the file system world, when there has been > some common set of changes needed to improve, say, Direct I/O, where > the changes are put into a standalone branch, say, in the xfs tree, > and those file systems which need it as a building block can pull it > into their tree, and then add the changes needed to use those changes > into their file system git tree. These changes are generally not > terribly controversial, and we've not had to worry about people want > to bikeshed the changes. > > There is a risk with doing this of course, which is that if the branch > *is* controversial, or gets bike-shedded for some reason, then Linus > gets upset and any branches which depended on said branch will get > rejected at the next merge window. Which is the requirement for (a) > and (b) above. Presumably, the fact that people were unwilling to let > Kunit land during this merge window might will *because* we think more > changes might be pending? My understanding, based on what I have been told, is that we were simply unlucky with the timing when Linus pulled the branch in the first week of the 5.4 merge window (Friday), such that once I fixed the directory naming issue, the updated changes didn't spend enough time in linux-next. And now with this issue fixed and KUnit back in linux-next, if nothing interesting happens between now and 5.5, it will be accepted in the 5.5 merge window. I do not think that anyone is expecting anymore changes before merging. Shuah, Linus, is my understanding correct? > The other thing I suppose I can do is to let the ext4 kunit tests land > in ext4 tree, but with the necessary #ifdef's around things like > "#include <kunit/test.h>" so that the build won't blow up w/o kunit > changes being in the tree that I'm building. It means I won't be able > to run the tests without creating a test integration branch and > merging in kunit by hand, which will super-annoying, of course. And > if some of the bike-shedding is in Kunit's interfaces, then that > becomes problematic as well, since any tests that are in ext4.git tree > might change if people want to rename Kunit's publically exported > functions (for example). Yeah, that seems even worse. I'm sorry to have caused this frustration. > > Hey Ted, do you know if that ext4 timestamp test can go in through > > linux-kselftest? It seemed fairly self-contained. Or is that what you > > were saying wouldn't work for you? > > Well, I was hoping that we might start creating more tests beyond just > the ext4 timestamp tests.... Okay, that's what I thought (and what I hoped) you were saying :-) I hope we can figure out something that will work for you. Or otherwise that you won't mind waiting until 5.5. Sorry
On Sun, Oct 6, 2019 at 10:18 AM Linus Torvalds <torvalds@linux-foundation.org> wrote: > > On Sun, Oct 6, 2019 at 9:55 AM Theodore Y. Ts'o <tytso@mit.edu> wrote: > > > > Well, one thing we *can* do is if (a) if we can create a kselftest > > branch which we know is stable and won't change, and (b) we can get > > assurances that Linus *will* accept that branch during the next merge > > window, those subsystems which want to use kself test can simply pull > > it into their tree. > > Yes. > > At the same time, I don't think it needs to be even that fancy. Even > if it's not a stable branch that gets shared between different > developers, it would be good to just have people do a "let's try this" > throw-away branch to use the kunit functionality and verify that > "yeah, this is fairly convenient for ext4". > > It doesn't have to be merged in that form, but just confirmation that > the infrastructure is helpful before it gets merged would be good. I thought we already had done this satisfactorily. We have one proof-of-concept test in the branch in the kselftest repo (proc sysctl test) that went out in the pull request, and we also had some other tests that were not in the pull request (there is the ext4 timestamp stuff mentioned above, and we also had one against the list data structure), which we were planning on sending out for review once Shuah's pull request was accepted. I know the apparmor people also wrote some tests that they said were useful; however, I have not coordinated with them on upstreaming their tests. I know of some other people who are using it, but I don't think the tests are as far along for upstreaming. The point is: I thought we had plenty of signal that KUnit would be useful to have merged into the mainline kernel. I thought the only reason it was rejected for 5.4 was due to the directory name issue combined with bad timing. Please correct me if I missed anything. Thanks!
On Sun, 6 Oct 2019 10:18:11 -0700 Linus Torvalds <torvalds@linux-foundation.org> wrote: > On Sun, Oct 6, 2019 at 9:55 AM Theodore Y. Ts'o <tytso@mit.edu> wrote: > > > > Well, one thing we *can* do is if (a) if we can create a kselftest > > branch which we know is stable and won't change, and (b) we can get > > assurances that Linus *will* accept that branch during the next merge > > window, those subsystems which want to use kself test can simply pull > > it into their tree. > > Yes. > > At the same time, I don't think it needs to be even that fancy. Even > if it's not a stable branch that gets shared between different > developers, it would be good to just have people do a "let's try this" > throw-away branch to use the kunit functionality and verify that > "yeah, this is fairly convenient for ext4". > > It doesn't have to be merged in that form, but just confirmation that > the infrastructure is helpful before it gets merged would be good. Can't you just create an ext4 branch that has the kselftest-next branch in it, that you build upon. And push that after the kunit test is merged? In the past I've had to rely on other branches in next, and would just hold two branches myself. One with everything not dependent on the other developer's branch, and one with the work that was. At the merge window, I would either merge the two or just send two pull requests with the two branches. -- Steve
On 10/7/19 2:40 AM, Brendan Higgins wrote: > On Sun, Oct 6, 2019 at 10:18 AM Linus Torvalds > <torvalds@linux-foundation.org> wrote: >> >> On Sun, Oct 6, 2019 at 9:55 AM Theodore Y. Ts'o <tytso@mit.edu> wrote: >>> >>> Well, one thing we *can* do is if (a) if we can create a kselftest >>> branch which we know is stable and won't change, and (b) we can get >>> assurances that Linus *will* accept that branch during the next merge >>> window, those subsystems which want to use kself test can simply pull >>> it into their tree. >> >> Yes. >> >> At the same time, I don't think it needs to be even that fancy. Even >> if it's not a stable branch that gets shared between different >> developers, it would be good to just have people do a "let's try this" >> throw-away branch to use the kunit functionality and verify that >> "yeah, this is fairly convenient for ext4". >> >> It doesn't have to be merged in that form, but just confirmation that >> the infrastructure is helpful before it gets merged would be good. > > I thought we already had done this satisfactorily. > Adding a couple more tests will only help in the long run. The idea is to see can this help > We have one proof-of-concept test in the branch in the kselftest repo > (proc sysctl test) that went out in the pull request, and we also had > some other tests that were not in the pull request (there is the ext4 > timestamp stuff mentioned above, and we also had one against the list > data structure), which we were planning on sending out for review once > Shuah's pull request was accepted. I know the apparmor people also > wrote some tests that they said were useful; however, I have not > coordinated with them on upstreaming their tests. I know of some other > people who are using it, but I don't think the tests are as far along > for upstreaming. > Maybe that is a good start. To get the tests that are already in use and get them in shape for upstream. > The point is: I thought we had plenty of signal that KUnit would be > useful to have merged into the mainline kernel. I thought the only > reason it was rejected for 5.4 was due to the directory name issue > combined with bad timing. > That is probably the initial thought. However, it makes perfect sense to add a couple of tests in. We have a few weeks anyway and it gives us more confidence on kunit. I already have a branch that is in linux-next and it just has kunit in it and I will rebase it to 5.4-rc1. https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/log/?h=test Let's use that for kunit work for 5.5. I won't add any kselftest patches to it and keep it dedicated for kunit work. When tests are ready for upstream, I can keep adding them to this branch. thanks, -- Shuah
On 10/7/19 8:40 AM, Steven Rostedt wrote: > On Sun, 6 Oct 2019 10:18:11 -0700 > Linus Torvalds <torvalds@linux-foundation.org> wrote: > >> On Sun, Oct 6, 2019 at 9:55 AM Theodore Y. Ts'o <tytso@mit.edu> wrote: >>> >>> Well, one thing we *can* do is if (a) if we can create a kselftest >>> branch which we know is stable and won't change, and (b) we can get >>> assurances that Linus *will* accept that branch during the next merge >>> window, those subsystems which want to use kself test can simply pull >>> it into their tree. >> >> Yes. >> >> At the same time, I don't think it needs to be even that fancy. Even >> if it's not a stable branch that gets shared between different >> developers, it would be good to just have people do a "let's try this" >> throw-away branch to use the kunit functionality and verify that >> "yeah, this is fairly convenient for ext4". >> >> It doesn't have to be merged in that form, but just confirmation that >> the infrastructure is helpful before it gets merged would be good. > > Can't you just create an ext4 branch that has the kselftest-next branch > in it, that you build upon. And push that after the kunit test is > merged? > > In the past I've had to rely on other branches in next, and would just > hold two branches myself. One with everything not dependent on the other > developer's branch, and one with the work that was. At the merge > window, I would either merge the two or just send two pull requests > with the two branches. > I do something similar when I am working on top of a branch that isn't already in the mainline. In any case, repeating myself Let's work on top of - it is rebased to 5.4-rc1 and ready for use. https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/log/?h=test Let's use that for kunit work for 5.5. I won't add any kselftest patches to it and keep it dedicated for kunit work. When tests are ready for upstream, I can keep adding them to this branch. thanks, -- Shuah