diff mbox

[dm-devel,RFC] training mpath to discern between SCSI errors

Message ID 4C7BC5B4.3010707@suse.de
State Not Applicable
Delegated to: David Miller
Headers show

Commit Message

Hannes Reinecke Aug. 30, 2010, 2:52 p.m. UTC 
Hannes Reinecke wrote:
> Sergei Shtylyov wrote:
>> Hello.
>>
>> Hannes Reinecke wrote:
>>
>>> Actually, I think we have two separate issues here:
>>> 1) The need of having more detailed I/O errors even in the fs layer. This
>>>    we've already discussed at the LSF, consensus here is to allow other
>>>    errors than just 'EIO'.
>>>    Instead of Mike's approach I would rather use existing error codes
>>> here;
>>>    this will make the transition somewhat easier.
>>>    Initially I would propose to return 'ENOLINK' for a transport failure,
>>>    'EIO' for a non-retryable failure on the target, and 'ENODEV' for a
>>>    retryable failure on the target.
>>    Are you sure it's not vice versa: EIO for retryable and ENODEV for
>> non-retryable failures. ENODEV looks more like permanent condition to me.
>>
> Ok, can do.
> And looking a the error numbers again, maybe we should be using 'EREMOTEIO'
> for non-retryable failures.
> 
> So we would be ending with:
> 
> ENOLINK: transport failure
> EIO: retryable remote failure
> EREMOTEIO: non-retryable remote failure
> 
And here is the corresponding patch.
Compile tested only; just to give an idea of the possible implementation.

I have decided to pass the I/O failure information in-line:
- scsi_check_sense() might now return 'TARGET_ERROR' to signal
  a permanent error
- scsi_decide_disposition() sets the driver byte of the result
  field to 'DID_TARGET_FAILURE' if a return code of 'TARGET_ERROR'
  is encountered.
- scsi_io_completion() sets the error to ENOLINK for DID_TRANSPORT_FAILFAST,
  EREMOTEIO for DID_TARGET_FAILURE, and EIO for any other error. It also
  resets DID_TARGET_FAILURE back to DID_OK once the error code is set.

I'm not 100% happy with this patch; DID_TARGET_FAILURE is really just
a communication vehicle to signal the permanent target failure.
I looked at passing this information directly via an explicit argument
to scsi_finish_command(), but this would include changing
scsi_io_completion(), too. As both of them are exported / public
interfaces I didn't like modifying them.

Another possibility would be to re-use / redefine the 'DRIVER_'
bits; they don't seem to be used a the moment. Eg 'DRIVER_HARD'
for permanent errors, DRIVER_SOFT for link failures.

Opinions welcome.

Cheers,

Hannes

Comments

Jun'ichi Nomura (NEC) Oct. 18, 2010, 8:09 a.m. UTC | #1 
Hi Hannes,

Thank you for working on this issue and sorry for very late reply...

(08/30/10 23:52), Hannes Reinecke wrote:
> From: Hannes Reinecke <hare@suse.de>
> Date: Mon, 30 Aug 2010 16:21:10 +0200
> Subject: [RFC][PATCH] scsi: Detailed I/O errors
> 
> Instead of just passing 'EIO' for any I/O errors we should be
> notifying the upper layers with some more details about the cause
> of this error.
> This patch updates the possible I/O errors to:
> 
> - ENOLINK: Link failure between host and target
> - EIO: Retryable I/O error
> - EREMOTEIO: Non-retryable I/O error
> 
> 'Retryable' in this context means that an I/O error _might_ be
> restricted to the I_T_L nexus (vulgo: path), so retrying on another
> nexus / path might succeed.

Does 'retryable' of EIO mean retryable in multipath layer?
If so, what is the difference between EIO and ENOLINK?

I've heard of a case where just retrying within path-group is
preferred to (relatively costly) switching group.
So, if EIO (or other error code) can be used to indicate such type
of errors, it's nice.


Also (although this might be a bit off topic from your patch),
can we expand such a distinction to what should be logged?
Currently, it's difficult to distinguish important SCSI/block errors
and less important ones in kernel log.
For example, when I get a link failure on sda, kernel prints something
like below, regardless of whether the I/O is recovered by multipathing or not:
  end_request: I/O error, dev sda, sector XXXXX

Setting REQ_QUIET in dm-multipath could mask the message
but also other important ones in SCSI.


Thanks,
Hannes Reinecke Oct. 18, 2010, 11:55 a.m. UTC | #2 
On 10/18/2010 10:09 AM, Jun'ichi Nomura wrote:
> Hi Hannes,
> 
> Thank you for working on this issue and sorry for very late reply...
> 
> (08/30/10 23:52), Hannes Reinecke wrote:
>> From: Hannes Reinecke <hare@suse.de>
>> Date: Mon, 30 Aug 2010 16:21:10 +0200
>> Subject: [RFC][PATCH] scsi: Detailed I/O errors
>>
>> Instead of just passing 'EIO' for any I/O errors we should be
>> notifying the upper layers with some more details about the cause
>> of this error.
>> This patch updates the possible I/O errors to:
>>
>> - ENOLINK: Link failure between host and target
>> - EIO: Retryable I/O error
>> - EREMOTEIO: Non-retryable I/O error
>>
>> 'Retryable' in this context means that an I/O error _might_ be
>> restricted to the I_T_L nexus (vulgo: path), so retrying on another
>> nexus / path might succeed.
> 
> Does 'retryable' of EIO mean retryable in multipath layer?
> If so, what is the difference between EIO and ENOLINK?
> 
Yes, EIO is intended for errors which should be retried at the
multipath layer. This does _not_ include transport errors, which are
signalled by ENOLINK.

Basically, ENOLINK is a transport error, and EIO just means
something is wrong and we weren't able to classify it properly.
If we were, it'd be either ENOLINK or EREMOTEIO.

> I've heard of a case where just retrying within path-group is
> preferred to (relatively costly) switching group.
> So, if EIO (or other error code) can be used to indicate such type
> of errors, it's nice.
> 
Yes, that was one of the intention.

> 
> Also (although this might be a bit off topic from your patch),
> can we expand such a distinction to what should be logged?
> Currently, it's difficult to distinguish important SCSI/block errors
> and less important ones in kernel log.
> For example, when I get a link failure on sda, kernel prints something
> like below, regardless of whether the I/O is recovered by multipathing or not:
>   end_request: I/O error, dev sda, sector XXXXX
> 
Indeed, when using the above we could be modifying the above
message, eg by

end_request: transport error, dev sda, sector XXXXX

or

end_request: target error, dev sda, sector XXXXX

which would improve the output noticeable.

> Setting REQ_QUIET in dm-multipath could mask the message
> but also other important ones in SCSI.
> 
Hmm. Not sure about that, but I think the above modifications will
be useful already.

I'll be sending an updated patch.

Cheers,

Hannes
Jun'ichi Nomura (NEC) Oct. 19, 2010, 4:03 a.m. UTC | #3 
Hi Hannes,

(10/18/10 20:55), Hannes Reinecke wrote:
>> Does 'retryable' of EIO mean retryable in multipath layer?
>> If so, what is the difference between EIO and ENOLINK?
>>
> Yes, EIO is intended for errors which should be retried at the
> multipath layer. This does _not_ include transport errors, which are
> signalled by ENOLINK.
> 
> Basically, ENOLINK is a transport error, and EIO just means
> something is wrong and we weren't able to classify it properly.
> If we were, it'd be either ENOLINK or EREMOTEIO.
> 
>> I've heard of a case where just retrying within path-group is
>> preferred to (relatively costly) switching group.
>> So, if EIO (or other error code) can be used to indicate such type
>> of errors, it's nice.
>>
> Yes, that was one of the intention.

Great to hear that.

And when it comes to retrying, the next problem is who controls it.
I don't think it's good to duplicate retry logic in multipath and
underlying device like SCSI (i.e. sd retries 5 times).
So perhaps we need a way to disable (or limit) retries in underlying
device at least.

>> Also (although this might be a bit off topic from your patch),
>> can we expand such a distinction to what should be logged?
>> Currently, it's difficult to distinguish important SCSI/block errors
>> and less important ones in kernel log.
>> For example, when I get a link failure on sda, kernel prints something
>> like below, regardless of whether the I/O is recovered by multipathing or not:
>>   end_request: I/O error, dev sda, sector XXXXX
>>
> Indeed, when using the above we could be modifying the above
> message, eg by
> 
> end_request: transport error, dev sda, sector XXXXX
> 
> or
> 
> end_request: target error, dev sda, sector XXXXX
> 
> which would improve the output noticeable.

It improves but still they look like critical errors
even if multipath saves them.

When I see this:

  end_request: target error, dev sda, sector XXXXX

I can't tell whether it's a real error visible to user space
or it's just recoverred by multipath retry/failover afterwards.


>> Setting REQ_QUIET in dm-multipath could mask the message
>> but also other important ones in SCSI.
>>
> Hmm. Not sure about that, but I think the above modifications will
> be useful already.
> 
> I'll be sending an updated patch.

Thank you. I'm looking for that.
malahal@us.ibm.com Nov. 19, 2010, 3:11 a.m. UTC | #4 
Hannes Reinecke [hare@suse.de] wrote:
> > Also (although this might be a bit off topic from your patch),
> > can we expand such a distinction to what should be logged?
> > Currently, it's difficult to distinguish important SCSI/block errors
> > and less important ones in kernel log.
> > For example, when I get a link failure on sda, kernel prints something
> > like below, regardless of whether the I/O is recovered by multipathing or not:
> >   end_request: I/O error, dev sda, sector XXXXX
> > 
> Indeed, when using the above we could be modifying the above
> message, eg by
> 
> end_request: transport error, dev sda, sector XXXXX
> 
> or
> 
> end_request: target error, dev sda, sector XXXXX
> 
> which would improve the output noticeable.
> 
> > Setting REQ_QUIET in dm-multipath could mask the message
> > but also other important ones in SCSI.
> > 
> Hmm. Not sure about that, but I think the above modifications will
> be useful already.
> 
> I'll be sending an updated patch.

Hannes, is there an updated version of this patch? It applied fine with
Linus git tree with a minor reject! I would like to test an updated
version if you have one (the update seems to refer to better logging
only, right?).

Thanks, Malahal.
--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Mike Snitzer Nov. 30, 2010, 10:59 p.m. UTC | #5 
On Thu, Nov 18 2010 at 10:11pm -0500,
Malahal Naineni <malahal@us.ibm.com> wrote:

> Hannes Reinecke [hare@suse.de] wrote:
> > > Also (although this might be a bit off topic from your patch),
> > > can we expand such a distinction to what should be logged?
> > > Currently, it's difficult to distinguish important SCSI/block errors
> > > and less important ones in kernel log.
> > > For example, when I get a link failure on sda, kernel prints something
> > > like below, regardless of whether the I/O is recovered by multipathing or not:
> > >   end_request: I/O error, dev sda, sector XXXXX
> > > 
> > Indeed, when using the above we could be modifying the above
> > message, eg by
> > 
> > end_request: transport error, dev sda, sector XXXXX
> > 
> > or
> > 
> > end_request: target error, dev sda, sector XXXXX
> > 
> > which would improve the output noticeable.
> > 
> > > Setting REQ_QUIET in dm-multipath could mask the message
> > > but also other important ones in SCSI.
> > > 
> > Hmm. Not sure about that, but I think the above modifications will
> > be useful already.
> > 
> > I'll be sending an updated patch.
> 
> Hannes, is there an updated version of this patch? It applied fine with
> Linus git tree with a minor reject! I would like to test an updated
> version if you have one (the update seems to refer to better logging
> only, right?).

Hannes,

Any chance you've had time to fold your proposed logging changes in and
rebase this patch?  Could you post that updated patch?

I'd like to help see this patch through to inclussion when 2.6.38 merge
window opens.  I can help with further review, testing and development.

Please advise, thanks.
Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Mike Snitzer Dec. 7, 2010, 11:16 p.m. UTC | #6 
Refreshed Hannes' initial "scsi: Detailed I/O errors" patch against
v2.6.37-rc5.  v2 introduces __scsi_error_from_host_byte to avoid
the duplicate switch statement.  Also a few whitespace and comment
changes.

Split DM mpath change out to separate v2 patch; failed discard is now
retryable in the face of a non-target IO error.

Added improved block layer's I/O error message (based on the finer
grained I/O error returns afforded by SCSI).

Comments/suggestions are welcome.

Thanks,
Mike

Hannes Reinecke (1):
  scsi: Detailed I/O errors

Mike Snitzer (2):
  dm mpath: propagate target I/O errors immediately
  block: improve detail in I/O error messages

 block/blk-core.c          |   12 +++++++++---
 drivers/md/dm-mpath.c     |   11 +----------
 drivers/scsi/scsi_error.c |   24 +++++++++++++++++-------
 drivers/scsi/scsi_lib.c   |   24 ++++++++++++++++++++++--
 include/scsi/scsi.h       |    3 +++
 5 files changed, 52 insertions(+), 22 deletions(-)
malahal@us.ibm.com Dec. 10, 2010, 11:40 p.m. UTC | #7 
Mike Snitzer [snitzer@redhat.com] wrote:
> Refreshed Hannes' initial "scsi: Detailed I/O errors" patch against
> v2.6.37-rc5.  v2 introduces __scsi_error_from_host_byte to avoid
> the duplicate switch statement.  Also a few whitespace and comment
> changes.
> 
> Split DM mpath change out to separate v2 patch; failed discard is now
> retryable in the face of a non-target IO error.
> 
> Added improved block layer's I/O error message (based on the finer
> grained I/O error returns afforded by SCSI).
> 
> Comments/suggestions are welcome.

I did test the Hannes original patch with the latest Linus' git tree! I
used scsi_debug to simulate path failures as well as 'Media' failures
and it did work as expected. I will test your patches soon.

Thanks, Malahal.
--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Hannes Reinecke Dec. 17, 2010, 9:47 a.m. UTC | #8 
On 11/30/2010 11:59 PM, Mike Snitzer wrote:
> On Thu, Nov 18 2010 at 10:11pm -0500,
> Malahal Naineni <malahal@us.ibm.com> wrote:
> 
>> Hannes Reinecke [hare@suse.de] wrote:
>>>> Also (although this might be a bit off topic from your patch),
>>>> can we expand such a distinction to what should be logged?
>>>> Currently, it's difficult to distinguish important SCSI/block errors
>>>> and less important ones in kernel log.
>>>> For example, when I get a link failure on sda, kernel prints something
>>>> like below, regardless of whether the I/O is recovered by multipathing or not:
>>>>   end_request: I/O error, dev sda, sector XXXXX
>>>>
>>> Indeed, when using the above we could be modifying the above
>>> message, eg by
>>>
>>> end_request: transport error, dev sda, sector XXXXX
>>>
>>> or
>>>
>>> end_request: target error, dev sda, sector XXXXX
>>>
>>> which would improve the output noticeable.
>>>
>>>> Setting REQ_QUIET in dm-multipath could mask the message
>>>> but also other important ones in SCSI.
>>>>
>>> Hmm. Not sure about that, but I think the above modifications will
>>> be useful already.
>>>
>>> I'll be sending an updated patch.
>>
>> Hannes, is there an updated version of this patch? It applied fine with
>> Linus git tree with a minor reject! I would like to test an updated
>> version if you have one (the update seems to refer to better logging
>> only, right?).
> 
> Hannes,
> 
> Any chance you've had time to fold your proposed logging changes in and
> rebase this patch?  Could you post that updated patch?
> 
yes, will be following shortly.

> I'd like to help see this patch through to inclussion when 2.6.38 merge
> window opens.  I can help with further review, testing and development.
> 
Ok, thanks.

Cheers,

Hannes
Mike Snitzer Dec. 17, 2010, 2:06 p.m. UTC | #9 
On Fri, Dec 17 2010 at  4:47am -0500,
Hannes Reinecke <hare@suse.de> wrote:

> On 11/30/2010 11:59 PM, Mike Snitzer wrote:
> > Hannes,
> > 
> > Any chance you've had time to fold your proposed logging changes in and
> > rebase this patch?  Could you post that updated patch?
> > 
> yes, will be following shortly.
> 
> > I'd like to help see this patch through to inclussion when 2.6.38 merge
> > window opens.  I can help with further review, testing and development.
> > 
> Ok, thanks.

I took some steps at furthering your work.  Here is the cover letter to
the patches I resently sent to dm-devel:
https://www.redhat.com/archives/dm-devel/2010-December/msg00090.html

And here are the patches:
https://patchwork.kernel.org/patch/384612/
https://patchwork.kernel.org/patch/384602/
https://patchwork.kernel.org/patch/390882/

Please feel free to change these how ever you see fit but your feedback
is really appreciated.

Thanks,
Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Mike Snitzer Jan. 14, 2011, 1:09 a.m. UTC | #10 
On Fri, Dec 17 2010 at  9:06am -0500,
Mike Snitzer <snitzer@redhat.com> wrote:

> On Fri, Dec 17 2010 at  4:47am -0500,
> Hannes Reinecke <hare@suse.de> wrote:
> 
> > On 11/30/2010 11:59 PM, Mike Snitzer wrote:
> > > Hannes,
> > > 
> > > Any chance you've had time to fold your proposed logging changes in and
> > > rebase this patch?  Could you post that updated patch?
> > > 
> > yes, will be following shortly.
> > 
> > > I'd like to help see this patch through to inclussion when 2.6.38 merge
> > > window opens.  I can help with further review, testing and development.
> > > 
> > Ok, thanks.
> 
> I took some steps at furthering your work.  Here is the cover letter to
> the patches I resently sent to dm-devel:
> https://www.redhat.com/archives/dm-devel/2010-December/msg00090.html
> 
> And here are the patches:
> https://patchwork.kernel.org/patch/384612/
> https://patchwork.kernel.org/patch/384602/
> https://patchwork.kernel.org/patch/390882/
> 
> Please feel free to change these how ever you see fit but your feedback
> is really appreciated.

Hannes,

Any update?  I'd really like to see this work get upstream ASAP.  I'm
doubtful that is possible for 2.6.38 given the merge window is likely to
close shortly.

Regardless, if we could get consensus on this work now and then stage it
with James that would be great.

Thanks,
Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Mike Snitzer Jan. 14, 2011, 1:15 a.m. UTC | #11 
On Fri, Dec 10 2010 at  6:40pm -0500,
Malahal Naineni <malahal@us.ibm.com> wrote:

> Mike Snitzer [snitzer@redhat.com] wrote:
> > Refreshed Hannes' initial "scsi: Detailed I/O errors" patch against
> > v2.6.37-rc5.  v2 introduces __scsi_error_from_host_byte to avoid
> > the duplicate switch statement.  Also a few whitespace and comment
> > changes.
> > 
> > Split DM mpath change out to separate v2 patch; failed discard is now
> > retryable in the face of a non-target IO error.
> > 
> > Added improved block layer's I/O error message (based on the finer
> > grained I/O error returns afforded by SCSI).
> > 
> > Comments/suggestions are welcome.
> 
> I did test the Hannes original patch with the latest Linus' git tree! I
> used scsi_debug to simulate path failures as well as 'Media' failures
> and it did work as expected. I will test your patches soon.

Hi Malahal,

I was wondering if you had any feedback (testing or otherwise) for these
patches:

https://patchwork.kernel.org/patch/384612/
https://patchwork.kernel.org/patch/384602/
https://patchwork.kernel.org/patch/390882/

We haven't heard from Hannes in a bit but I was hoping we could at least
understand that the few changes I made are agreeable and working as
expected.

Thanks,
Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Hannes Reinecke Jan. 14, 2011, 7:45 a.m. UTC | #12 
On 01/14/2011 02:09 AM, Mike Snitzer wrote:
> On Fri, Dec 17 2010 at  9:06am -0500,
> Mike Snitzer <snitzer@redhat.com> wrote:
> 
>> On Fri, Dec 17 2010 at  4:47am -0500,
>> Hannes Reinecke <hare@suse.de> wrote:
>>
>>> On 11/30/2010 11:59 PM, Mike Snitzer wrote:
>>>> Hannes,
>>>>
>>>> Any chance you've had time to fold your proposed logging changes in and
>>>> rebase this patch?  Could you post that updated patch?
>>>>
>>> yes, will be following shortly.
>>>
>>>> I'd like to help see this patch through to inclussion when 2.6.38 merge
>>>> window opens.  I can help with further review, testing and development.
>>>>
>>> Ok, thanks.
>>
>> I took some steps at furthering your work.  Here is the cover letter to
>> the patches I resently sent to dm-devel:
>> https://www.redhat.com/archives/dm-devel/2010-December/msg00090.html
>>
>> And here are the patches:
>> https://patchwork.kernel.org/patch/384612/
>> https://patchwork.kernel.org/patch/384602/
>> https://patchwork.kernel.org/patch/390882/
>>
>> Please feel free to change these how ever you see fit but your feedback
>> is really appreciated.
> 
> Hannes,
> 
> Any update?  I'd really like to see this work get upstream ASAP.  I'm
> doubtful that is possible for 2.6.38 given the merge window is likely to
> close shortly.
> 
Sorry for the late answer; the above patches somehow got lost in my
various mail folders :-(

But yes, the patchset looks okay.
Feel free to add my Acked-by: to last two.

> Regardless, if we could get consensus on this work now and then stage it
> with James that would be great.
> 
Indeed. Will you resend it to linux-scsi or shall I do it?

Cheers,

Hannes
Mike Snitzer Jan. 14, 2011, 1:59 p.m. UTC | #13 
On Fri, Jan 14 2011 at  2:45am -0500,
Hannes Reinecke <hare@suse.de> wrote:

> On 01/14/2011 02:09 AM, Mike Snitzer wrote:
> > On Fri, Dec 17 2010 at  9:06am -0500,
> > Mike Snitzer <snitzer@redhat.com> wrote:
> > 
> >> On Fri, Dec 17 2010 at  4:47am -0500,
> >> Hannes Reinecke <hare@suse.de> wrote:
> >>
> >>> On 11/30/2010 11:59 PM, Mike Snitzer wrote:
> >>>> Hannes,
> >>>>
> >>>> Any chance you've had time to fold your proposed logging changes in and
> >>>> rebase this patch?  Could you post that updated patch?
> >>>>
> >>> yes, will be following shortly.
> >>>
> >>>> I'd like to help see this patch through to inclussion when 2.6.38 merge
> >>>> window opens.  I can help with further review, testing and development.
> >>>>
> >>> Ok, thanks.
> >>
> >> I took some steps at furthering your work.  Here is the cover letter to
> >> the patches I resently sent to dm-devel:
> >> https://www.redhat.com/archives/dm-devel/2010-December/msg00090.html
> >>
> >> And here are the patches:
> >> https://patchwork.kernel.org/patch/384612/
> >> https://patchwork.kernel.org/patch/384602/
> >> https://patchwork.kernel.org/patch/390882/
> >>
> >> Please feel free to change these how ever you see fit but your feedback
> >> is really appreciated.
> > 
> > Hannes,
> > 
> > Any update?  I'd really like to see this work get upstream ASAP.  I'm
> > doubtful that is possible for 2.6.38 given the merge window is likely to
> > close shortly.
> > 
> Sorry for the late answer; the above patches somehow got lost in my
> various mail folders :-(
> 
> But yes, the patchset looks okay.
> Feel free to add my Acked-by: to last two.

OK.

> > Regardless, if we could get consensus on this work now and then stage it
> > with James that would be great.
> > 
> Indeed. Will you resend it to linux-scsi or shall I do it?

I can resend, will do so shortly.

Thanks!
Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
diff mbox

Patch

diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
index 487ecda..d49b375 100644
--- a/drivers/md/dm-mpath.c
+++ b/drivers/md/dm-mpath.c
@@ -1270,7 +1270,7 @@  static int do_end_io(struct multipath *m, struct request *clone,
 	if (!error && !clone->errors)
 		return 0;	/* I/O complete */
 
-	if (error == -EOPNOTSUPP)
+	if (error == -EOPNOTSUPP || error == -EREMOTEIO)
 		return error;
 
 	if (clone->cmd_flags & REQ_DISCARD)
diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c
index ce089df..5da040b 100644
--- a/drivers/scsi/scsi_error.c
+++ b/drivers/scsi/scsi_error.c
@@ -223,7 +223,7 @@  static inline void scsi_eh_prt_fail_stats(struct Scsi_Host *shost,
  * @scmd:	Cmd to have sense checked.
  *
  * Return value:
- * 	SUCCESS or FAILED or NEEDS_RETRY
+ *	SUCCESS or FAILED or NEEDS_RETRY or TARGET_ERROR
  *
  * Notes:
  *	When a deferred error is detected the current command has
@@ -338,25 +338,25 @@  static int scsi_check_sense(struct scsi_cmnd *scmd)
 	case COPY_ABORTED:
 	case VOLUME_OVERFLOW:
 	case MISCOMPARE:
-		return SUCCESS;
+	case DATA_PROTECT:
+	case BLANK_CHECK:
+		return TARGET_ERROR;
 
 	case MEDIUM_ERROR:
 		if (sshdr.asc == 0x11 || /* UNRECOVERED READ ERR */
 		    sshdr.asc == 0x13 || /* AMNF DATA FIELD */
 		    sshdr.asc == 0x14) { /* RECORD NOT FOUND */
-			return SUCCESS;
+			return TARGET_ERROR;
 		}
 		return NEEDS_RETRY;
 
 	case HARDWARE_ERROR:
 		if (scmd->device->retry_hwerror)
 			return ADD_TO_MLQUEUE;
-		else
-			return SUCCESS;
-
+		else {
+			return TARGET_ERROR;
+		}
 	case ILLEGAL_REQUEST:
-	case BLANK_CHECK:
-	case DATA_PROTECT:
 	default:
 		return SUCCESS;
 	}
@@ -819,6 +819,7 @@  static int scsi_send_eh_cmnd(struct scsi_cmnd *scmd, unsigned char *cmnd,
 		case SUCCESS:
 		case NEEDS_RETRY:
 		case FAILED:
+		case TARGET_ERROR:
 			break;
 		case ADD_TO_MLQUEUE:
 			rtn = NEEDS_RETRY;
@@ -1512,6 +1513,12 @@  int scsi_decide_disposition(struct scsi_cmnd *scmd)
 		rtn = scsi_check_sense(scmd);
 		if (rtn == NEEDS_RETRY)
 			goto maybe_retry;
+		if (rtn == TARGET_ERROR) {
+			/* Need to modify host byte to signal a
+			 * permanent target failure */
+			scmd->result |= (DID_TARGET_FAILURE << 16);
+			rtn = SUCCESS;
+		}
 		/* if rtn == FAILED, we have no sense information;
 		 * returning FAILED will wake the error handler thread
 		 * to collect the sense and redo the decide
@@ -1529,6 +1536,7 @@  int scsi_decide_disposition(struct scsi_cmnd *scmd)
 	case RESERVATION_CONFLICT:
 		SCSI_LOG_ERROR_RECOVERY(3, sdev_printk(KERN_INFO, scmd->device,
 						"reservation conflict\n"));
+		scmd->result |= (DID_TARGET_FAILURE << 16);
 		return SUCCESS; /* causes immediate i/o error */
 	default:
 		return FAILED;
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 9ade720..fb841e3 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -736,8 +736,20 @@  void scsi_io_completion(struct scsi_cmnd *cmd, unsigned int good_bytes)
 				memcpy(req->sense, cmd->sense_buffer,  len);
 				req->sense_len = len;
 			}
-			if (!sense_deferred)
-				error = -EIO;
+			if (!sense_deferred) {
+				switch(host_byte(result)) {
+				case DID_TRANSPORT_FAILFAST:
+					error = -ENOLINK;
+					break;
+				case DID_TARGET_FAILURE:
+					cmd->result |= (DID_OK << 16);
+					error = -EREMOTEIO;
+					break;
+				default:
+					error = -EIO;
+					break;
+				}
+			}
 		}
 
 		req->resid_len = scsi_get_resid(cmd);
@@ -796,7 +808,18 @@  void scsi_io_completion(struct scsi_cmnd *cmd, unsigned int good_bytes)
 	if (scsi_end_request(cmd, error, good_bytes, result == 0) == NULL)
 		return;
 
-	error = -EIO;
+	switch (host_byte(result)) {
+	case DID_TRANSPORT_FAILFAST:
+		error = -ENOLINK;
+		break;
+	case DID_TARGET_FAILURE:
+		cmd->result |= (DID_OK << 16);
+		error = -EREMOTEIO;
+		break;
+	default:
+		error = -EIO;
+		break;
+	}
 
 	if (host_byte(result) == DID_RESET) {
 		/* Third party bus reset or reset for error recovery
@@ -1418,7 +1441,6 @@  static void scsi_softirq_done(struct request *rq)
 			    wait_for/HZ);
 		disposition = SUCCESS;
 	}
-			
 	scsi_log_completion(cmd, disposition);
 
 	switch (disposition) {
diff --git a/include/scsi/scsi.h b/include/scsi/scsi.h
index 8fcb6e0..abfee76 100644
--- a/include/scsi/scsi.h
+++ b/include/scsi/scsi.h
@@ -397,6 +397,8 @@  static inline int scsi_is_wlun(unsigned int lun)
 				      * recover the link. Transport class will
 				      * retry or fail IO */
 #define DID_TRANSPORT_FAILFAST	0x0f /* Transport class fastfailed the io */
+#define DID_TARGET_FAILURE 0x10 /* Permanent target failure, do not retry on
+				 * other paths */
 #define DRIVER_OK       0x00	/* Driver status                           */
 
 /*
@@ -426,6 +428,7 @@  static inline int scsi_is_wlun(unsigned int lun)
 #define TIMEOUT_ERROR   0x2007
 #define SCSI_RETURN_NOT_HANDLED   0x2008
 #define FAST_IO_FAIL	0x2009
+#define TARGET_ERROR    0x200A
 
 /*
  * Midlevel queue return values.