mbox series

[dev-5.2,0/2] i2c: aspeed: Add H/W timeout support

Message ID 20190904200758.5420-1-jae.hyun.yoo@linux.intel.com
Headers show
Series i2c: aspeed: Add H/W timeout support | expand

Message

Jae Hyun Yoo Sept. 4, 2019, 8:07 p.m. UTC
In case of multi-master environment, if a peer master incorrectly handles
a bus in the middle of a transaction, I2C hardware hangs in slave state
and it can't escape from the slave state, so this commit adds slave
inactive timeout support to recover the bus in the case.

By applying this change, SDA data-low and SCL clock-low timeout feature
also could be enabled which was disabled previously.

Jae Hyun Yoo (2):
  dt-bindings: i2c: aspeed: add hardware timeout support
  i2c: aspeed: add slave inactive timeout support

 .../devicetree/bindings/i2c/i2c-aspeed.txt    |  2 +
 drivers/i2c/busses/i2c-aspeed.c               | 79 +++++++++++++++++--
 2 files changed, 75 insertions(+), 6 deletions(-)

Comments

Joel Stanley Sept. 4, 2019, 10:54 p.m. UTC | #1
Hi Jae,

On Wed, 4 Sep 2019 at 20:08, Jae Hyun Yoo <jae.hyun.yoo@linux.intel.com> wrote:
>
> In case of multi-master environment, if a peer master incorrectly handles
> a bus in the middle of a transaction, I2C hardware hangs in slave state
> and it can't escape from the slave state, so this commit adds slave
> inactive timeout support to recover the bus in the case.
>
> By applying this change, SDA data-low and SCL clock-low timeout feature
> also could be enabled which was disabled previously.

Please consider sending your RFC patches to the upstream list. You
have a big backlog of patches now.

Cheers,

Joel

>
> Jae Hyun Yoo (2):
>   dt-bindings: i2c: aspeed: add hardware timeout support
>   i2c: aspeed: add slave inactive timeout support
>
>  .../devicetree/bindings/i2c/i2c-aspeed.txt    |  2 +
>  drivers/i2c/busses/i2c-aspeed.c               | 79 +++++++++++++++++--
>  2 files changed, 75 insertions(+), 6 deletions(-)
>
> --
> 2.23.0
>
Jae Hyun Yoo Sept. 4, 2019, 11:01 p.m. UTC | #2
Hi Joel,

On 9/4/2019 3:54 PM, Joel Stanley wrote:
> Hi Jae,
> 
> On Wed, 4 Sep 2019 at 20:08, Jae Hyun Yoo <jae.hyun.yoo@linux.intel.com> wrote:
>>
>> In case of multi-master environment, if a peer master incorrectly handles
>> a bus in the middle of a transaction, I2C hardware hangs in slave state
>> and it can't escape from the slave state, so this commit adds slave
>> inactive timeout support to recover the bus in the case.
>>
>> By applying this change, SDA data-low and SCL clock-low timeout feature
>> also could be enabled which was disabled previously.
> 
> Please consider sending your RFC patches to the upstream list. You
> have a big backlog of patches now.

Thanks for the reminding. I can't send the RFC patches yet because QEMU
H/W model isn't ready yet. I'm still waiting for the fix from Cedric.

Thanks,
Jae

> 
> Cheers,
> 
> Joel
> 
>>
>> Jae Hyun Yoo (2):
>>    dt-bindings: i2c: aspeed: add hardware timeout support
>>    i2c: aspeed: add slave inactive timeout support
>>
>>   .../devicetree/bindings/i2c/i2c-aspeed.txt    |  2 +
>>   drivers/i2c/busses/i2c-aspeed.c               | 79 +++++++++++++++++--
>>   2 files changed, 75 insertions(+), 6 deletions(-)
>>
>> --
>> 2.23.0
>>
Andrew Jeffery Sept. 4, 2019, 11:12 p.m. UTC | #3
On Thu, 5 Sep 2019, at 08:31, Jae Hyun Yoo wrote:
> Hi Joel,
> 
> On 9/4/2019 3:54 PM, Joel Stanley wrote:
> > Hi Jae,
> > 
> > On Wed, 4 Sep 2019 at 20:08, Jae Hyun Yoo <jae.hyun.yoo@linux.intel.com> wrote:
> >>
> >> In case of multi-master environment, if a peer master incorrectly handles
> >> a bus in the middle of a transaction, I2C hardware hangs in slave state
> >> and it can't escape from the slave state, so this commit adds slave
> >> inactive timeout support to recover the bus in the case.
> >>
> >> By applying this change, SDA data-low and SCL clock-low timeout feature
> >> also could be enabled which was disabled previously.
> > 
> > Please consider sending your RFC patches to the upstream list. You
> > have a big backlog of patches now.
> 
> Thanks for the reminding. I can't send the RFC patches yet because QEMU
> H/W model isn't ready yet. I'm still waiting for the fix from Cedric.

QEMU shouldn't be preventing you from sending patches upstream, rather
it prevents us from enabling the buffer mode support in the OpenBMC
kernel tree. You should be sending all patches upstream as early as possible.

Cheers,

Andrew
Jae Hyun Yoo Sept. 4, 2019, 11:40 p.m. UTC | #4
Hi Andrew,

On 9/4/2019 4:12 PM, Andrew Jeffery wrote:
> On Thu, 5 Sep 2019, at 08:31, Jae Hyun Yoo wrote:
>> Hi Joel,
>>
>> On 9/4/2019 3:54 PM, Joel Stanley wrote:
>>> Hi Jae,
>>>
>>> On Wed, 4 Sep 2019 at 20:08, Jae Hyun Yoo <jae.hyun.yoo@linux.intel.com> wrote:
>>>>
>>>> In case of multi-master environment, if a peer master incorrectly handles
>>>> a bus in the middle of a transaction, I2C hardware hangs in slave state
>>>> and it can't escape from the slave state, so this commit adds slave
>>>> inactive timeout support to recover the bus in the case.
>>>>
>>>> By applying this change, SDA data-low and SCL clock-low timeout feature
>>>> also could be enabled which was disabled previously.
>>>
>>> Please consider sending your RFC patches to the upstream list. You
>>> have a big backlog of patches now.
>>
>> Thanks for the reminding. I can't send the RFC patches yet because QEMU
>> H/W model isn't ready yet. I'm still waiting for the fix from Cedric.
> 
> QEMU shouldn't be preventing you from sending patches upstream, rather
> it prevents us from enabling the buffer mode support in the OpenBMC
> kernel tree. You should be sending all patches upstream as early as possible.

I met a QEMU issue when I was upstreaming a patch set last year:
https://lists.ozlabs.org/pipermail/linux-aspeed/2018-September/000750.html

If OpenBMC community accepts the QEMU issue, I can submit the RFC
patches to upstream. Will submit the patch set soon to linux tree.

Thanks,
Jae
Brendan Higgins Sept. 4, 2019, 11:50 p.m. UTC | #5
On Wed, Sep 4, 2019 at 3:55 PM Joel Stanley <joel@jms.id.au> wrote:
>
> Hi Jae,
>
> On Wed, 4 Sep 2019 at 20:08, Jae Hyun Yoo <jae.hyun.yoo@linux.intel.com> wrote:
> >
> > In case of multi-master environment, if a peer master incorrectly handles
> > a bus in the middle of a transaction, I2C hardware hangs in slave state
> > and it can't escape from the slave state, so this commit adds slave
> > inactive timeout support to recover the bus in the case.
> >
> > By applying this change, SDA data-low and SCL clock-low timeout feature
> > also could be enabled which was disabled previously.
>
> Please consider sending your RFC patches to the upstream list. You
> have a big backlog of patches now.

Joel, thanks for keeping track of this!

Sorry, for not keeping up with my emails, I don't think I have time to
continue to maintain this.

However, I don't want to hand this off in a bad state. I will try to
get up to date on all the emails in the coming weeks.

Jae, can you start sending things upstream as Joel suggested?
Andrew Jeffery Sept. 5, 2019, 12:10 a.m. UTC | #6
On Thu, 5 Sep 2019, at 09:10, Jae Hyun Yoo wrote:
> Hi Andrew,
> 
> On 9/4/2019 4:12 PM, Andrew Jeffery wrote:
> > On Thu, 5 Sep 2019, at 08:31, Jae Hyun Yoo wrote:
> >> Hi Joel,
> >>
> >> On 9/4/2019 3:54 PM, Joel Stanley wrote:
> >>> Hi Jae,
> >>>
> >>> On Wed, 4 Sep 2019 at 20:08, Jae Hyun Yoo <jae.hyun.yoo@linux.intel.com> wrote:
> >>>>
> >>>> In case of multi-master environment, if a peer master incorrectly handles
> >>>> a bus in the middle of a transaction, I2C hardware hangs in slave state
> >>>> and it can't escape from the slave state, so this commit adds slave
> >>>> inactive timeout support to recover the bus in the case.
> >>>>
> >>>> By applying this change, SDA data-low and SCL clock-low timeout feature
> >>>> also could be enabled which was disabled previously.
> >>>
> >>> Please consider sending your RFC patches to the upstream list. You
> >>> have a big backlog of patches now.
> >>
> >> Thanks for the reminding. I can't send the RFC patches yet because QEMU
> >> H/W model isn't ready yet. I'm still waiting for the fix from Cedric.
> > 
> > QEMU shouldn't be preventing you from sending patches upstream, rather
> > it prevents us from enabling the buffer mode support in the OpenBMC
> > kernel tree. You should be sending all patches upstream as early as possible.
> 
> I met a QEMU issue when I was upstreaming a patch set last year:
> https://lists.ozlabs.org/pipermail/linux-aspeed/2018-September/000750.html
> 
> If OpenBMC community accepts the QEMU issue, I can submit the RFC
> patches to upstream. Will submit the patch set soon to linux tree.

Ah, didn't realise it was Guenter that ran into it. We have some changes[1] in
Cedric's aspeed-4.2 qemu tree - do you mind testing it out? If those patches
resolve the issue Maybe we could point Guenter at that tree, though really we
should get the fixes upstream so this isn't an issue.

[1] https://github.com/legoater/qemu/compare/59dda66ab756e52e6a9c1ef89660d30b3769f63c...aspeed-4.2

Cheers,

Andrew
Jae Hyun Yoo Sept. 5, 2019, 12:54 a.m. UTC | #7
On 9/4/2019 5:10 PM, Andrew Jeffery wrote:
> 
> 
> On Thu, 5 Sep 2019, at 09:10, Jae Hyun Yoo wrote:
>> Hi Andrew,
>>
>> On 9/4/2019 4:12 PM, Andrew Jeffery wrote:
>>> On Thu, 5 Sep 2019, at 08:31, Jae Hyun Yoo wrote:
>>>> Hi Joel,
>>>>
>>>> On 9/4/2019 3:54 PM, Joel Stanley wrote:
>>>>> Hi Jae,
>>>>>
>>>>> On Wed, 4 Sep 2019 at 20:08, Jae Hyun Yoo <jae.hyun.yoo@linux.intel.com> wrote:
>>>>>>
>>>>>> In case of multi-master environment, if a peer master incorrectly handles
>>>>>> a bus in the middle of a transaction, I2C hardware hangs in slave state
>>>>>> and it can't escape from the slave state, so this commit adds slave
>>>>>> inactive timeout support to recover the bus in the case.
>>>>>>
>>>>>> By applying this change, SDA data-low and SCL clock-low timeout feature
>>>>>> also could be enabled which was disabled previously.
>>>>>
>>>>> Please consider sending your RFC patches to the upstream list. You
>>>>> have a big backlog of patches now.
>>>>
>>>> Thanks for the reminding. I can't send the RFC patches yet because QEMU
>>>> H/W model isn't ready yet. I'm still waiting for the fix from Cedric.
>>>
>>> QEMU shouldn't be preventing you from sending patches upstream, rather
>>> it prevents us from enabling the buffer mode support in the OpenBMC
>>> kernel tree. You should be sending all patches upstream as early as possible.
>>
>> I met a QEMU issue when I was upstreaming a patch set last year:
>> https://lists.ozlabs.org/pipermail/linux-aspeed/2018-September/000750.html
>>
>> If OpenBMC community accepts the QEMU issue, I can submit the RFC
>> patches to upstream. Will submit the patch set soon to linux tree.
> 
> Ah, didn't realise it was Guenter that ran into it. We have some changes[1] in
> Cedric's aspeed-4.2 qemu tree - do you mind testing it out? If those patches
> resolve the issue Maybe we could point Guenter at that tree, though really we
> should get the fixes upstream so this isn't an issue.
> 
> [1] https://github.com/legoater/qemu/compare/59dda66ab756e52e6a9c1ef89660d30b3769f63c...aspeed-4.2
> 

Okay. I'll give it a try.

Thanks,
Jae
Jae Hyun Yoo Sept. 5, 2019, 12:56 a.m. UTC | #8
On 9/4/2019 4:50 PM, Brendan Higgins wrote:
> On Wed, Sep 4, 2019 at 3:55 PM Joel Stanley <joel@jms.id.au> wrote:
>>
>> Hi Jae,
>>
>> On Wed, 4 Sep 2019 at 20:08, Jae Hyun Yoo <jae.hyun.yoo@linux.intel.com> wrote:
>>>
>>> In case of multi-master environment, if a peer master incorrectly handles
>>> a bus in the middle of a transaction, I2C hardware hangs in slave state
>>> and it can't escape from the slave state, so this commit adds slave
>>> inactive timeout support to recover the bus in the case.
>>>
>>> By applying this change, SDA data-low and SCL clock-low timeout feature
>>> also could be enabled which was disabled previously.
>>
>> Please consider sending your RFC patches to the upstream list. You
>> have a big backlog of patches now.
> 
> Joel, thanks for keeping track of this!
> 
> Sorry, for not keeping up with my emails, I don't think I have time to
> continue to maintain this.
> 
> However, I don't want to hand this off in a bad state. I will try to
> get up to date on all the emails in the coming weeks.
> 
> Jae, can you start sending things upstream as Joel suggested?

Sure, I'll start upstreaming after taking some tests on Cedric's QEMU
patch.

Thanks,
Jae
Jae Hyun Yoo Sept. 11, 2019, 6:56 p.m. UTC | #9
Hi Andrew,

On 9/4/2019 5:54 PM, Jae Hyun Yoo wrote:
> On 9/4/2019 5:10 PM, Andrew Jeffery wrote:
>>
>>
>> On Thu, 5 Sep 2019, at 09:10, Jae Hyun Yoo wrote:
>>> Hi Andrew,
>>>
>>> On 9/4/2019 4:12 PM, Andrew Jeffery wrote:
>>>> On Thu, 5 Sep 2019, at 08:31, Jae Hyun Yoo wrote:
>>>>> Hi Joel,
>>>>>
>>>>> On 9/4/2019 3:54 PM, Joel Stanley wrote:
>>>>>> Hi Jae,
>>>>>>
>>>>>> On Wed, 4 Sep 2019 at 20:08, Jae Hyun Yoo 
>>>>>> <jae.hyun.yoo@linux.intel.com> wrote:
>>>>>>>
>>>>>>> In case of multi-master environment, if a peer master incorrectly 
>>>>>>> handles
>>>>>>> a bus in the middle of a transaction, I2C hardware hangs in slave 
>>>>>>> state
>>>>>>> and it can't escape from the slave state, so this commit adds slave
>>>>>>> inactive timeout support to recover the bus in the case.
>>>>>>>
>>>>>>> By applying this change, SDA data-low and SCL clock-low timeout 
>>>>>>> feature
>>>>>>> also could be enabled which was disabled previously.
>>>>>>
>>>>>> Please consider sending your RFC patches to the upstream list. You
>>>>>> have a big backlog of patches now.
>>>>>
>>>>> Thanks for the reminding. I can't send the RFC patches yet because 
>>>>> QEMU
>>>>> H/W model isn't ready yet. I'm still waiting for the fix from Cedric.
>>>>
>>>> QEMU shouldn't be preventing you from sending patches upstream, rather
>>>> it prevents us from enabling the buffer mode support in the OpenBMC
>>>> kernel tree. You should be sending all patches upstream as early as 
>>>> possible.
>>>
>>> I met a QEMU issue when I was upstreaming a patch set last year:
>>> https://lists.ozlabs.org/pipermail/linux-aspeed/2018-September/000750.html 
>>>
>>>
>>> If OpenBMC community accepts the QEMU issue, I can submit the RFC
>>> patches to upstream. Will submit the patch set soon to linux tree.
>>
>> Ah, didn't realise it was Guenter that ran into it. We have some 
>> changes[1] in
>> Cedric's aspeed-4.2 qemu tree - do you mind testing it out? If those 
>> patches
>> resolve the issue Maybe we could point Guenter at that tree, though 
>> really we
>> should get the fixes upstream so this isn't an issue.
>>
>> [1] 
>> https://github.com/legoater/qemu/compare/59dda66ab756e52e6a9c1ef89660d30b3769f63c...aspeed-4.2 
>>
>>
> 
> Okay. I'll give it a try.

I've tested I2C buffer mode support in QEMU using:
git://github.com/legoater/qemu.git
SRCBRANCH = "aspeed-4.2"
SRCREV = "1b31d645c448858eb7d11d463a4cb77df0ee7923"

Checked that I2C buffer mode works on the latest QEMU H/W model.
Thanks Cedric, Eddie for the H/W model implementation.

I'll submit all I2C backlog patches into linux upstream.

Cheers,
Jae
Andrew Jeffery Sept. 12, 2019, 1:22 a.m. UTC | #10
On Thu, 12 Sep 2019, at 04:26, Jae Hyun Yoo wrote:
> Hi Andrew,
> 
> On 9/4/2019 5:54 PM, Jae Hyun Yoo wrote:
> > On 9/4/2019 5:10 PM, Andrew Jeffery wrote:
> >>
> >>
> >> On Thu, 5 Sep 2019, at 09:10, Jae Hyun Yoo wrote:
> >>> Hi Andrew,
> >>>
> >>> On 9/4/2019 4:12 PM, Andrew Jeffery wrote:
> >>>> On Thu, 5 Sep 2019, at 08:31, Jae Hyun Yoo wrote:
> >>>>> Hi Joel,
> >>>>>
> >>>>> On 9/4/2019 3:54 PM, Joel Stanley wrote:
> >>>>>> Hi Jae,
> >>>>>>
> >>>>>> On Wed, 4 Sep 2019 at 20:08, Jae Hyun Yoo 
> >>>>>> <jae.hyun.yoo@linux.intel.com> wrote:
> >>>>>>>
> >>>>>>> In case of multi-master environment, if a peer master incorrectly 
> >>>>>>> handles
> >>>>>>> a bus in the middle of a transaction, I2C hardware hangs in slave 
> >>>>>>> state
> >>>>>>> and it can't escape from the slave state, so this commit adds slave
> >>>>>>> inactive timeout support to recover the bus in the case.
> >>>>>>>
> >>>>>>> By applying this change, SDA data-low and SCL clock-low timeout 
> >>>>>>> feature
> >>>>>>> also could be enabled which was disabled previously.
> >>>>>>
> >>>>>> Please consider sending your RFC patches to the upstream list. You
> >>>>>> have a big backlog of patches now.
> >>>>>
> >>>>> Thanks for the reminding. I can't send the RFC patches yet because 
> >>>>> QEMU
> >>>>> H/W model isn't ready yet. I'm still waiting for the fix from Cedric.
> >>>>
> >>>> QEMU shouldn't be preventing you from sending patches upstream, rather
> >>>> it prevents us from enabling the buffer mode support in the OpenBMC
> >>>> kernel tree. You should be sending all patches upstream as early as 
> >>>> possible.
> >>>
> >>> I met a QEMU issue when I was upstreaming a patch set last year:
> >>> https://lists.ozlabs.org/pipermail/linux-aspeed/2018-September/000750.html 
> >>>
> >>>
> >>> If OpenBMC community accepts the QEMU issue, I can submit the RFC
> >>> patches to upstream. Will submit the patch set soon to linux tree.
> >>
> >> Ah, didn't realise it was Guenter that ran into it. We have some 
> >> changes[1] in
> >> Cedric's aspeed-4.2 qemu tree - do you mind testing it out? If those 
> >> patches
> >> resolve the issue Maybe we could point Guenter at that tree, though 
> >> really we
> >> should get the fixes upstream so this isn't an issue.
> >>
> >> [1] 
> >> https://github.com/legoater/qemu/compare/59dda66ab756e52e6a9c1ef89660d30b3769f63c...aspeed-4.2 
> >>
> >>
> > 
> > Okay. I'll give it a try.
> 
> I've tested I2C buffer mode support in QEMU using:
> git://github.com/legoater/qemu.git
> SRCBRANCH = "aspeed-4.2"
> SRCREV = "1b31d645c448858eb7d11d463a4cb77df0ee7923"

This looks like changes to the qemu bitbake recipe? Have you
integrated openbmc/qemu into Yocto's runqemu infrastructure,
or were you just using bitbake to build qemu? I'd love to see
patches if you've done the former :)

Andrew
Jae Hyun Yoo Sept. 12, 2019, 1:26 a.m. UTC | #11
On 9/11/2019 6:22 PM, Andrew Jeffery wrote:
> 
> 
> On Thu, 12 Sep 2019, at 04:26, Jae Hyun Yoo wrote:
>> Hi Andrew,
>>
>> On 9/4/2019 5:54 PM, Jae Hyun Yoo wrote:
>>> On 9/4/2019 5:10 PM, Andrew Jeffery wrote:
>>>>
>>>>
>>>> On Thu, 5 Sep 2019, at 09:10, Jae Hyun Yoo wrote:
>>>>> Hi Andrew,
>>>>>
>>>>> On 9/4/2019 4:12 PM, Andrew Jeffery wrote:
>>>>>> On Thu, 5 Sep 2019, at 08:31, Jae Hyun Yoo wrote:
>>>>>>> Hi Joel,
>>>>>>>
>>>>>>> On 9/4/2019 3:54 PM, Joel Stanley wrote:
>>>>>>>> Hi Jae,
>>>>>>>>
>>>>>>>> On Wed, 4 Sep 2019 at 20:08, Jae Hyun Yoo
>>>>>>>> <jae.hyun.yoo@linux.intel.com> wrote:
>>>>>>>>>
>>>>>>>>> In case of multi-master environment, if a peer master incorrectly
>>>>>>>>> handles
>>>>>>>>> a bus in the middle of a transaction, I2C hardware hangs in slave
>>>>>>>>> state
>>>>>>>>> and it can't escape from the slave state, so this commit adds slave
>>>>>>>>> inactive timeout support to recover the bus in the case.
>>>>>>>>>
>>>>>>>>> By applying this change, SDA data-low and SCL clock-low timeout
>>>>>>>>> feature
>>>>>>>>> also could be enabled which was disabled previously.
>>>>>>>>
>>>>>>>> Please consider sending your RFC patches to the upstream list. You
>>>>>>>> have a big backlog of patches now.
>>>>>>>
>>>>>>> Thanks for the reminding. I can't send the RFC patches yet because
>>>>>>> QEMU
>>>>>>> H/W model isn't ready yet. I'm still waiting for the fix from Cedric.
>>>>>>
>>>>>> QEMU shouldn't be preventing you from sending patches upstream, rather
>>>>>> it prevents us from enabling the buffer mode support in the OpenBMC
>>>>>> kernel tree. You should be sending all patches upstream as early as
>>>>>> possible.
>>>>>
>>>>> I met a QEMU issue when I was upstreaming a patch set last year:
>>>>> https://lists.ozlabs.org/pipermail/linux-aspeed/2018-September/000750.html
>>>>>
>>>>>
>>>>> If OpenBMC community accepts the QEMU issue, I can submit the RFC
>>>>> patches to upstream. Will submit the patch set soon to linux tree.
>>>>
>>>> Ah, didn't realise it was Guenter that ran into it. We have some
>>>> changes[1] in
>>>> Cedric's aspeed-4.2 qemu tree - do you mind testing it out? If those
>>>> patches
>>>> resolve the issue Maybe we could point Guenter at that tree, though
>>>> really we
>>>> should get the fixes upstream so this isn't an issue.
>>>>
>>>> [1]
>>>> https://github.com/legoater/qemu/compare/59dda66ab756e52e6a9c1ef89660d30b3769f63c...aspeed-4.2
>>>>
>>>>
>>>
>>> Okay. I'll give it a try.
>>
>> I've tested I2C buffer mode support in QEMU using:
>> git://github.com/legoater/qemu.git
>> SRCBRANCH = "aspeed-4.2"
>> SRCREV = "1b31d645c448858eb7d11d463a4cb77df0ee7923"
> 
> This looks like changes to the qemu bitbake recipe? Have you
> integrated openbmc/qemu into Yocto's runqemu infrastructure,
> or were you just using bitbake to build qemu? I'd love to see
> patches if you've done the former :)
> 
> Andrew
> 

No, I just made this change using a bbappend in my local yocto tree.
The source revision on the branch worked well.

Jae