diff mbox series

[RFC,bpf-next,v2,4/4] error-injection: Support fault injection framework

Message ID 151427450538.32561.2776225740675148782.stgit@devbox
State Changes Requested, archived
Delegated to: BPF Maintainers
Headers show
Series Separate error injection table from kprobes | expand

Commit Message

Masami Hiramatsu (Google) Dec. 26, 2017, 7:48 a.m. UTC
Support in-kernel fault-injection framework via debugfs.
This allows you to inject a conditional error to specified
function using debugfs interfaces.

Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
 Documentation/fault-injection/fault-injection.txt |    5 +
 kernel/Makefile                                   |    1 
 kernel/fail_function.c                            |  169 +++++++++++++++++++++
 lib/Kconfig.debug                                 |   10 +
 4 files changed, 185 insertions(+)
 create mode 100644 kernel/fail_function.c

Comments

Alexei Starovoitov Dec. 27, 2017, 2:12 a.m. UTC | #1
On Tue, Dec 26, 2017 at 04:48:25PM +0900, Masami Hiramatsu wrote:
> Support in-kernel fault-injection framework via debugfs.
> This allows you to inject a conditional error to specified
> function using debugfs interfaces.
> 
> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
> ---
>  Documentation/fault-injection/fault-injection.txt |    5 +
>  kernel/Makefile                                   |    1 
>  kernel/fail_function.c                            |  169 +++++++++++++++++++++
>  lib/Kconfig.debug                                 |   10 +
>  4 files changed, 185 insertions(+)
>  create mode 100644 kernel/fail_function.c
> 
> diff --git a/Documentation/fault-injection/fault-injection.txt b/Documentation/fault-injection/fault-injection.txt
> index 918972babcd8..6243a588dd71 100644
> --- a/Documentation/fault-injection/fault-injection.txt
> +++ b/Documentation/fault-injection/fault-injection.txt
> @@ -30,6 +30,11 @@ o fail_mmc_request
>    injects MMC data errors on devices permitted by setting
>    debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
>  
> +o fail_function
> +
> +  injects error return on specific functions by setting debugfs entries
> +  under /sys/kernel/debug/fail_function. No boot option supported.

I like it.
Could you document it a bit better?
In particular retval is configurable, but without an example no one
will be able to figure out how to use it.

I think you can drop RFC tag from the next version of these patches.
Thanks!
Masami Hiramatsu (Google) Dec. 27, 2017, 8:09 a.m. UTC | #2
On Tue, 26 Dec 2017 18:12:56 -0800
Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:

> On Tue, Dec 26, 2017 at 04:48:25PM +0900, Masami Hiramatsu wrote:
> > Support in-kernel fault-injection framework via debugfs.
> > This allows you to inject a conditional error to specified
> > function using debugfs interfaces.
> > 
> > Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
> > ---
> >  Documentation/fault-injection/fault-injection.txt |    5 +
> >  kernel/Makefile                                   |    1 
> >  kernel/fail_function.c                            |  169 +++++++++++++++++++++
> >  lib/Kconfig.debug                                 |   10 +
> >  4 files changed, 185 insertions(+)
> >  create mode 100644 kernel/fail_function.c
> > 
> > diff --git a/Documentation/fault-injection/fault-injection.txt b/Documentation/fault-injection/fault-injection.txt
> > index 918972babcd8..6243a588dd71 100644
> > --- a/Documentation/fault-injection/fault-injection.txt
> > +++ b/Documentation/fault-injection/fault-injection.txt
> > @@ -30,6 +30,11 @@ o fail_mmc_request
> >    injects MMC data errors on devices permitted by setting
> >    debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
> >  
> > +o fail_function
> > +
> > +  injects error return on specific functions by setting debugfs entries
> > +  under /sys/kernel/debug/fail_function. No boot option supported.
> 
> I like it.
> Could you document it a bit better?

Yes, I will do in next series.

> In particular retval is configurable, but without an example no one
> will be able to figure out how to use it.

Ah, right. BTW, as I pointed in the covermail, should we store the
expected error value range into the injectable list? e.g.

ALLOW_ERROR_INJECTION(open_ctree, -1, -MAX_ERRNO)

And provide APIs to check/get it.

const struct error_range *ei_get_error_range(unsigned long addr);


> 
> I think you can drop RFC tag from the next version of these patches.
> Thanks!

Thank you, I'll fix some errors came from configurations, and resend it.


Thanks!
Alexei Starovoitov Dec. 27, 2017, 10:49 p.m. UTC | #3
On 12/27/17 12:09 AM, Masami Hiramatsu wrote:
> On Tue, 26 Dec 2017 18:12:56 -0800
> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>
>> On Tue, Dec 26, 2017 at 04:48:25PM +0900, Masami Hiramatsu wrote:
>>> Support in-kernel fault-injection framework via debugfs.
>>> This allows you to inject a conditional error to specified
>>> function using debugfs interfaces.
>>>
>>> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
>>> ---
>>>  Documentation/fault-injection/fault-injection.txt |    5 +
>>>  kernel/Makefile                                   |    1
>>>  kernel/fail_function.c                            |  169 +++++++++++++++++++++
>>>  lib/Kconfig.debug                                 |   10 +
>>>  4 files changed, 185 insertions(+)
>>>  create mode 100644 kernel/fail_function.c
>>>
>>> diff --git a/Documentation/fault-injection/fault-injection.txt b/Documentation/fault-injection/fault-injection.txt
>>> index 918972babcd8..6243a588dd71 100644
>>> --- a/Documentation/fault-injection/fault-injection.txt
>>> +++ b/Documentation/fault-injection/fault-injection.txt
>>> @@ -30,6 +30,11 @@ o fail_mmc_request
>>>    injects MMC data errors on devices permitted by setting
>>>    debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
>>>
>>> +o fail_function
>>> +
>>> +  injects error return on specific functions by setting debugfs entries
>>> +  under /sys/kernel/debug/fail_function. No boot option supported.
>>
>> I like it.
>> Could you document it a bit better?
>
> Yes, I will do in next series.
>
>> In particular retval is configurable, but without an example no one
>> will be able to figure out how to use it.
>
> Ah, right. BTW, as I pointed in the covermail, should we store the
> expected error value range into the injectable list? e.g.
>
> ALLOW_ERROR_INJECTION(open_ctree, -1, -MAX_ERRNO)
>
> And provide APIs to check/get it.

I'm afraid such check would be too costly.
Right now we have only two functions marked but I expect hundreds more
will be added in the near future as soon as developers realize the
potential of such error injection.
All of ALLOW_ERROR_INJECTION marks add 8 byte overhead each to .data.
Multiple by 1k and we have 8k of data spent on marks.
If we add max/min range marks that doubles it for very little use.
I think marking function only is enough.
Masami Hiramatsu (Google) Dec. 28, 2017, 1:38 a.m. UTC | #4
On Wed, 27 Dec 2017 14:49:46 -0800
Alexei Starovoitov <ast@fb.com> wrote:

> On 12/27/17 12:09 AM, Masami Hiramatsu wrote:
> > On Tue, 26 Dec 2017 18:12:56 -0800
> > Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
> >
> >> On Tue, Dec 26, 2017 at 04:48:25PM +0900, Masami Hiramatsu wrote:
> >>> Support in-kernel fault-injection framework via debugfs.
> >>> This allows you to inject a conditional error to specified
> >>> function using debugfs interfaces.
> >>>
> >>> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
> >>> ---
> >>>  Documentation/fault-injection/fault-injection.txt |    5 +
> >>>  kernel/Makefile                                   |    1
> >>>  kernel/fail_function.c                            |  169 +++++++++++++++++++++
> >>>  lib/Kconfig.debug                                 |   10 +
> >>>  4 files changed, 185 insertions(+)
> >>>  create mode 100644 kernel/fail_function.c
> >>>
> >>> diff --git a/Documentation/fault-injection/fault-injection.txt b/Documentation/fault-injection/fault-injection.txt
> >>> index 918972babcd8..6243a588dd71 100644
> >>> --- a/Documentation/fault-injection/fault-injection.txt
> >>> +++ b/Documentation/fault-injection/fault-injection.txt
> >>> @@ -30,6 +30,11 @@ o fail_mmc_request
> >>>    injects MMC data errors on devices permitted by setting
> >>>    debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
> >>>
> >>> +o fail_function
> >>> +
> >>> +  injects error return on specific functions by setting debugfs entries
> >>> +  under /sys/kernel/debug/fail_function. No boot option supported.
> >>
> >> I like it.
> >> Could you document it a bit better?
> >
> > Yes, I will do in next series.
> >
> >> In particular retval is configurable, but without an example no one
> >> will be able to figure out how to use it.
> >
> > Ah, right. BTW, as I pointed in the covermail, should we store the
> > expected error value range into the injectable list? e.g.
> >
> > ALLOW_ERROR_INJECTION(open_ctree, -1, -MAX_ERRNO)
> >
> > And provide APIs to check/get it.
> 
> I'm afraid such check would be too costly.
> Right now we have only two functions marked but I expect hundreds more
> will be added in the near future as soon as developers realize the
> potential of such error injection.
> All of ALLOW_ERROR_INJECTION marks add 8 byte overhead each to .data.
> Multiple by 1k and we have 8k of data spent on marks.
> If we add max/min range marks that doubles it for very little use.
> I think marking function only is enough.

Sorry, I don't think so.
Even if it takes 16 bytes more for each points, I don't think it is
any overhead for machines in these days. Even if so, we can provide
a kconfig to reduce it.
I mean, we are living in GB-order memory are, and it will be bigger
in the future. Why we have to worry about hundreds of 16bytes memory
pieces? It will take a few KB, and even if we mark thousands of
functions, it never reaches 1MB, in GB memory pool. :)

Of course, for many small-footprint embedded devices (like having
less than 128MB memory), this feature can be a overhead. But they
can cut off the table by kconfig.

Thank you,
Alexei Starovoitov Dec. 28, 2017, 3:49 a.m. UTC | #5
On 12/27/17 5:38 PM, Masami Hiramatsu wrote:
> On Wed, 27 Dec 2017 14:49:46 -0800
> Alexei Starovoitov <ast@fb.com> wrote:
>
>> On 12/27/17 12:09 AM, Masami Hiramatsu wrote:
>>> On Tue, 26 Dec 2017 18:12:56 -0800
>>> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>>>
>>>> On Tue, Dec 26, 2017 at 04:48:25PM +0900, Masami Hiramatsu wrote:
>>>>> Support in-kernel fault-injection framework via debugfs.
>>>>> This allows you to inject a conditional error to specified
>>>>> function using debugfs interfaces.
>>>>>
>>>>> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
>>>>> ---
>>>>>  Documentation/fault-injection/fault-injection.txt |    5 +
>>>>>  kernel/Makefile                                   |    1
>>>>>  kernel/fail_function.c                            |  169 +++++++++++++++++++++
>>>>>  lib/Kconfig.debug                                 |   10 +
>>>>>  4 files changed, 185 insertions(+)
>>>>>  create mode 100644 kernel/fail_function.c
>>>>>
>>>>> diff --git a/Documentation/fault-injection/fault-injection.txt b/Documentation/fault-injection/fault-injection.txt
>>>>> index 918972babcd8..6243a588dd71 100644
>>>>> --- a/Documentation/fault-injection/fault-injection.txt
>>>>> +++ b/Documentation/fault-injection/fault-injection.txt
>>>>> @@ -30,6 +30,11 @@ o fail_mmc_request
>>>>>    injects MMC data errors on devices permitted by setting
>>>>>    debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
>>>>>
>>>>> +o fail_function
>>>>> +
>>>>> +  injects error return on specific functions by setting debugfs entries
>>>>> +  under /sys/kernel/debug/fail_function. No boot option supported.
>>>>
>>>> I like it.
>>>> Could you document it a bit better?
>>>
>>> Yes, I will do in next series.
>>>
>>>> In particular retval is configurable, but without an example no one
>>>> will be able to figure out how to use it.
>>>
>>> Ah, right. BTW, as I pointed in the covermail, should we store the
>>> expected error value range into the injectable list? e.g.
>>>
>>> ALLOW_ERROR_INJECTION(open_ctree, -1, -MAX_ERRNO)
>>>
>>> And provide APIs to check/get it.
>>
>> I'm afraid such check would be too costly.
>> Right now we have only two functions marked but I expect hundreds more
>> will be added in the near future as soon as developers realize the
>> potential of such error injection.
>> All of ALLOW_ERROR_INJECTION marks add 8 byte overhead each to .data.
>> Multiple by 1k and we have 8k of data spent on marks.
>> If we add max/min range marks that doubles it for very little use.
>> I think marking function only is enough.
>
> Sorry, I don't think so.
> Even if it takes 16 bytes more for each points, I don't think it is
> any overhead for machines in these days. Even if so, we can provide
> a kconfig to reduce it.
> I mean, we are living in GB-order memory are, and it will be bigger
> in the future. Why we have to worry about hundreds of 16bytes memory
> pieces? It will take a few KB, and even if we mark thousands of
> functions, it never reaches 1MB, in GB memory pool. :)
>
> Of course, for many small-footprint embedded devices (like having
> less than 128MB memory), this feature can be a overhead. But they
> can cut off the table by kconfig.

I still disagree on wasting 16-byte * num_of_funcs of .data here.
The trade-off of usability vs memory just not worth it. Sorry.
Please focus on testing your changes instead.
Masami Hiramatsu (Google) Dec. 28, 2017, 7:51 a.m. UTC | #6
On Wed, 27 Dec 2017 19:49:28 -0800
Alexei Starovoitov <ast@fb.com> wrote:

> On 12/27/17 5:38 PM, Masami Hiramatsu wrote:
> > On Wed, 27 Dec 2017 14:49:46 -0800
> > Alexei Starovoitov <ast@fb.com> wrote:
> >
> >> On 12/27/17 12:09 AM, Masami Hiramatsu wrote:
> >>> On Tue, 26 Dec 2017 18:12:56 -0800
> >>> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
> >>>
> >>>> On Tue, Dec 26, 2017 at 04:48:25PM +0900, Masami Hiramatsu wrote:
> >>>>> Support in-kernel fault-injection framework via debugfs.
> >>>>> This allows you to inject a conditional error to specified
> >>>>> function using debugfs interfaces.
> >>>>>
> >>>>> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
> >>>>> ---
> >>>>>  Documentation/fault-injection/fault-injection.txt |    5 +
> >>>>>  kernel/Makefile                                   |    1
> >>>>>  kernel/fail_function.c                            |  169 +++++++++++++++++++++
> >>>>>  lib/Kconfig.debug                                 |   10 +
> >>>>>  4 files changed, 185 insertions(+)
> >>>>>  create mode 100644 kernel/fail_function.c
> >>>>>
> >>>>> diff --git a/Documentation/fault-injection/fault-injection.txt b/Documentation/fault-injection/fault-injection.txt
> >>>>> index 918972babcd8..6243a588dd71 100644
> >>>>> --- a/Documentation/fault-injection/fault-injection.txt
> >>>>> +++ b/Documentation/fault-injection/fault-injection.txt
> >>>>> @@ -30,6 +30,11 @@ o fail_mmc_request
> >>>>>    injects MMC data errors on devices permitted by setting
> >>>>>    debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
> >>>>>
> >>>>> +o fail_function
> >>>>> +
> >>>>> +  injects error return on specific functions by setting debugfs entries
> >>>>> +  under /sys/kernel/debug/fail_function. No boot option supported.
> >>>>
> >>>> I like it.
> >>>> Could you document it a bit better?
> >>>
> >>> Yes, I will do in next series.
> >>>
> >>>> In particular retval is configurable, but without an example no one
> >>>> will be able to figure out how to use it.
> >>>
> >>> Ah, right. BTW, as I pointed in the covermail, should we store the
> >>> expected error value range into the injectable list? e.g.
> >>>
> >>> ALLOW_ERROR_INJECTION(open_ctree, -1, -MAX_ERRNO)
> >>>
> >>> And provide APIs to check/get it.
> >>
> >> I'm afraid such check would be too costly.
> >> Right now we have only two functions marked but I expect hundreds more
> >> will be added in the near future as soon as developers realize the
> >> potential of such error injection.
> >> All of ALLOW_ERROR_INJECTION marks add 8 byte overhead each to .data.
> >> Multiple by 1k and we have 8k of data spent on marks.
> >> If we add max/min range marks that doubles it for very little use.
> >> I think marking function only is enough.
> >
> > Sorry, I don't think so.
> > Even if it takes 16 bytes more for each points, I don't think it is
> > any overhead for machines in these days. Even if so, we can provide
> > a kconfig to reduce it.
> > I mean, we are living in GB-order memory are, and it will be bigger
> > in the future. Why we have to worry about hundreds of 16bytes memory
> > pieces? It will take a few KB, and even if we mark thousands of
> > functions, it never reaches 1MB, in GB memory pool. :)
> >
> > Of course, for many small-footprint embedded devices (like having
> > less than 128MB memory), this feature can be a overhead. But they
> > can cut off the table by kconfig.
> 
> I still disagree on wasting 16-byte * num_of_funcs of .data here.
> The trade-off of usability vs memory just not worth it. Sorry.
> Please focus on testing your changes instead.

Then what happen if the user set invalid retval to those functions?
even if we limit the injectable functions, it can cause a problem,

for example, 

 obj = func_return_object();
 if (!obj) {
    handling_error...;
 }
 obj->field = x;

In this case, obviously func_return_object() must return NULL if there is
an error, not -ENOMEM. But without the correct retval information, how would
you check the BPF code doesn't cause a trouble?
Currently it seems you are expecting only the functions which return error code.

 ret = func_return_state();
 if (ret < 0) {
    handling_error...;
 }

But how we can distinguish those?

If we have the error range for each function, we can ensure what is
*correct* error code, NULL or errno, or any other error numbers. :)

At least fail_function needs this feature because it can check
return value when setting it up.

Thank you,
Alexei Starovoitov Dec. 29, 2017, 1:11 a.m. UTC | #7
On 12/27/17 11:51 PM, Masami Hiramatsu wrote:
>
> Then what happen if the user set invalid retval to those functions?
> even if we limit the injectable functions, it can cause a problem,
>
> for example,
>
>  obj = func_return_object();
>  if (!obj) {
>     handling_error...;
>  }
>  obj->field = x;
>
> In this case, obviously func_return_object() must return NULL if there is
> an error, not -ENOMEM. But without the correct retval information, how would
> you check the BPF code doesn't cause a trouble?
> Currently it seems you are expecting only the functions which return error code.
>
>  ret = func_return_state();
>  if (ret < 0) {
>     handling_error...;
>  }
>
> But how we can distinguish those?
>
> If we have the error range for each function, we can ensure what is
> *correct* error code, NULL or errno, or any other error numbers. :)

messing up return values may cause problems and range check is
not going to magically help.
The caller may handle only a certain set of errors or interpret
some of them like EBUSY as a signal to retry.
It's plain impossible to make sure that kernel will be functional
after error injection has been made.
Like kmalloc() unconditionally returning NULL will be deadly
for the kernel, hence this patch 4/4 has very limited practical
use. The bpf program need to make intelligent decisions when
to return an error and what kind of error to return.
Doing blank range check adds a false sense of additional safety.
More so it wastes kilobytes of memory to do this check, hence nack.
Masami Hiramatsu (Google) Dec. 29, 2017, 7:34 a.m. UTC | #8
On Thu, 28 Dec 2017 17:11:31 -0800
Alexei Starovoitov <ast@fb.com> wrote:

> On 12/27/17 11:51 PM, Masami Hiramatsu wrote:
> >
> > Then what happen if the user set invalid retval to those functions?
> > even if we limit the injectable functions, it can cause a problem,
> >
> > for example,
> >
> >  obj = func_return_object();
> >  if (!obj) {
> >     handling_error...;
> >  }
> >  obj->field = x;
> >
> > In this case, obviously func_return_object() must return NULL if there is
> > an error, not -ENOMEM. But without the correct retval information, how would
> > you check the BPF code doesn't cause a trouble?
> > Currently it seems you are expecting only the functions which return error code.
> >
> >  ret = func_return_state();
> >  if (ret < 0) {
> >     handling_error...;
> >  }
> >
> > But how we can distinguish those?
> >
> > If we have the error range for each function, we can ensure what is
> > *correct* error code, NULL or errno, or any other error numbers. :)
> 
> messing up return values may cause problems and range check is
> not going to magically help.
> The caller may handle only a certain set of errors or interpret
> some of them like EBUSY as a signal to retry.
> It's plain impossible to make sure that kernel will be functional
> after error injection has been made.

Hmm, if so, why we need this injectable table?
If we can not make sure the safeness of the error injection (of course, yes)
why we need to limit the error injection on such limited functions?
I think we don't need it anymore. Any function can be injectable, and no
need to make sure the safeness.

Thank you,

> Like kmalloc() unconditionally returning NULL will be deadly
> for the kernel, hence this patch 4/4 has very limited practical
> use. The bpf program need to make intelligent decisions when
> to return an error and what kind of error to return.
> Doing blank range check adds a false sense of additional safety.
> More so it wastes kilobytes of memory to do this check, hence nack.
>
diff mbox series

Patch

diff --git a/Documentation/fault-injection/fault-injection.txt b/Documentation/fault-injection/fault-injection.txt
index 918972babcd8..6243a588dd71 100644
--- a/Documentation/fault-injection/fault-injection.txt
+++ b/Documentation/fault-injection/fault-injection.txt
@@ -30,6 +30,11 @@  o fail_mmc_request
   injects MMC data errors on devices permitted by setting
   debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
 
+o fail_function
+
+  injects error return on specific functions by setting debugfs entries
+  under /sys/kernel/debug/fail_function. No boot option supported.
+
 Configure fault-injection capabilities behavior
 -----------------------------------------------
 
diff --git a/kernel/Makefile b/kernel/Makefile
index 172d151d429c..f85ae5dfa474 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -81,6 +81,7 @@  obj-$(CONFIG_AUDIT_TREE) += audit_tree.o
 obj-$(CONFIG_GCOV_KERNEL) += gcov/
 obj-$(CONFIG_KCOV) += kcov.o
 obj-$(CONFIG_KPROBES) += kprobes.o
+obj-$(CONFIG_FAIL_FUNCTION) += fail_function.o
 obj-$(CONFIG_KGDB) += debug/
 obj-$(CONFIG_DETECT_HUNG_TASK) += hung_task.o
 obj-$(CONFIG_LOCKUP_DETECTOR) += watchdog.o
diff --git a/kernel/fail_function.c b/kernel/fail_function.c
new file mode 100644
index 000000000000..203d3582487a
--- /dev/null
+++ b/kernel/fail_function.c
@@ -0,0 +1,169 @@ 
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * fail_function.c: Function-based error injection
+ */
+#include <linux/error-injection.h>
+#include <linux/debugfs.h>
+#include <linux/fault-inject.h>
+#include <linux/kallsyms.h>
+#include <linux/kprobes.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/slab.h>
+#include <linux/uaccess.h>
+
+static int fei_kprobe_handler(struct kprobe *kp, struct pt_regs *regs);
+
+static DEFINE_MUTEX(fei_lock);
+static struct {
+	struct kprobe kp;
+	unsigned long retval;
+	struct fault_attr attr;
+} fei_attr = {
+	.kp = { .pre_handler = fei_kprobe_handler, },
+	.retval = ~0UL,	/* This indicates -1 in long/int return value */
+	.attr = FAULT_ATTR_INITIALIZER,
+};
+
+static int fei_kprobe_handler(struct kprobe *kp, struct pt_regs *regs)
+{
+	if (should_fail(&fei_attr.attr, 1)) {
+		regs_set_return_value(regs, fei_attr.retval);
+		override_function_with_return(regs);
+		/* Kprobe specific fixup */
+		reset_current_kprobe();
+		preempt_enable_no_resched();
+		return 1;
+	}
+
+	return 0;
+}
+NOKPROBE_SYMBOL(fei_kprobe_handler)
+
+static void *fei_seq_start(struct seq_file *m, loff_t *pos)
+{
+	mutex_lock(&fei_lock);
+	return *pos == 0 ? (void *)1 : NULL;
+}
+
+static void fei_seq_stop(struct seq_file *m, void *v)
+{
+	mutex_unlock(&fei_lock);
+}
+
+static void *fei_seq_next(struct seq_file *m, void *v, loff_t *pos)
+{
+	return NULL;
+}
+
+static int fei_seq_show(struct seq_file *m, void *v)
+{
+	if (fei_attr.kp.addr)
+		seq_printf(m, "%pf\n", fei_attr.kp.addr);
+	else
+		seq_puts(m, "# not specified\n");
+	return 0;
+}
+
+static const struct seq_operations fei_seq_ops = {
+	.start	= fei_seq_start,
+	.next	= fei_seq_next,
+	.stop	= fei_seq_stop,
+	.show	= fei_seq_show,
+};
+
+static int fei_open(struct inode *inode, struct file *file)
+{
+	return seq_open(file, &fei_seq_ops);
+}
+
+static ssize_t fei_write(struct file *file, const char __user *buffer,
+			 size_t count, loff_t *ppos)
+{
+	unsigned long addr;
+	char *buf, *sym;
+	int ret;
+
+	/* cut off if it is too long */
+	if (count > KSYM_NAME_LEN)
+		count = KSYM_NAME_LEN;
+	buf = kmalloc(sizeof(char) * (count + 1), GFP_KERNEL);
+	if (!buf)
+		return -ENOMEM;
+
+	if (copy_from_user(buf, buffer, count)) {
+		ret = -EFAULT;
+		goto out;
+	}
+	buf[count] = '\0';
+	sym = strstrip(buf);
+
+	if (strlen(sym) == 0 || sym[0] == '0') {
+		if (fei_attr.kp.addr) {
+			unregister_kprobe(&fei_attr.kp);
+			fei_attr.kp.addr = NULL;
+		}
+		ret = count;
+		goto out;
+	}
+
+	addr = kallsyms_lookup_name(sym);
+	if (!addr) {
+		ret = -EINVAL;
+		goto out;
+	}
+	if (!within_error_injection_list(addr)) {
+		ret = -ERANGE;
+		goto out;
+	}
+
+	if (fei_attr.kp.addr) {
+		unregister_kprobe(&fei_attr.kp);
+		fei_attr.kp.addr = NULL;
+	}
+	fei_attr.kp.addr = (void *)addr;
+	ret = register_kprobe(&fei_attr.kp);
+	if (ret < 0)
+		fei_attr.kp.addr = NULL;
+	else
+		ret = count;
+out:
+	kfree(buf);
+	return ret;
+}
+
+static const struct file_operations fei_ops = {
+	.open =		fei_open,
+	.read =		seq_read,
+	.write =	fei_write,
+	.llseek =	seq_lseek,
+	.release =	seq_release,
+};
+
+static int __init fei_debugfs_init(void)
+{
+	struct dentry *dir;
+
+	dir = fault_create_debugfs_attr("fail_function", NULL,
+					&fei_attr.attr);
+	if (IS_ERR(dir))
+		return PTR_ERR(dir);
+
+	// injectable attribute is just a symlink of error_inject/list
+	if (!debugfs_create_symlink("injectable", dir,
+				    "../error_injection/list"))
+		goto error;
+
+	if (!debugfs_create_file("inject", 0600, dir, NULL, &fei_ops))
+		goto error;
+
+	if (!debugfs_create_ulong("retval", 0600, dir, &fei_attr.retval))
+		goto error;
+
+	return 0;
+error:
+	debugfs_remove_recursive(dir);
+	return -ENOMEM;
+}
+
+late_initcall(fei_debugfs_init);
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index fe88ac0f003c..15c6609a1bb8 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1551,6 +1551,16 @@  config FAIL_FUTEX
 	help
 	  Provide fault-injection capability for futexes.
 
+config FAIL_FUNCTION
+	bool "Fault-injection capability for functions"
+	depends on FAULT_INJECTION_DEBUG_FS && FUNCTION_ERROR_INJECTION
+	help
+	  Provide function-based fault-injection capability.
+	  This will allow you to override a specific function with a return
+	  with given return value. As a result, function caller will see
+	  an error value and have to handle it. This is useful to test the
+	  error handling in various subsystems.
+
 config FAULT_INJECTION_DEBUG_FS
 	bool "Debugfs entries for fault-injection capabilities"
 	depends on FAULT_INJECTION && SYSFS && DEBUG_FS