: Add Network Sysrq Support

Add Network Sysrq Support

In some circumstances, a system can hang/lockup in such a way that the system
is completely unresponsive to keyboard or console input but is still
responsive to ping.  The config option, CONFIG_SYSRQ_PING, builds
net/ipv4/sysrq-ping.ko which allows a root user to configure the system for
a remote sysrq.

To use this do:

mount -t debugfs none /sys/kernel/debug/
echo 1 > /proc/sys/kernel/sysrq
echo <hex digit val> > /sys/kernel/debug/network_sysrq_magic
echo 1 > /sys/kernel/debug/network_sysrq_enable

Then on another system on the network you can do:

ping -c 1 -p <up to 30 hex digit val><hex val of sysrq> <target_system_name>

ex) sysrq-m, m is ascii 0x6d

ping -c 1 p 1623a06f554d46d676d <target_system_name>

Note that the network sysrq automatically disables after the receipt of
the ping, ie) it is single-shot mode.  If you want to use this again, you
must complete the above four steps again.

Signed-off-by: Prarit Bhargava <prarit@redhat.com>

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Tue, 21 Jun 2011 09:00:40 -0400
Prarit Bhargava <prarit@redhat.com> wrote:

> Add Network Sysrq Support
> 
> In some circumstances, a system can hang/lockup in such a way that the system
> is completely unresponsive to keyboard or console input but is still
> responsive to ping.  The config option, CONFIG_SYSRQ_PING, builds
> net/ipv4/sysrq-ping.ko which allows a root user to configure the system for
> a remote sysrq.
> 
> To use this do:
> 
> mount -t debugfs none /sys/kernel/debug/
> echo 1 > /proc/sys/kernel/sysrq
> echo <hex digit val> > /sys/kernel/debug/network_sysrq_magic
> echo 1 > /sys/kernel/debug/network_sysrq_enable
> 
> Then on another system on the network you can do:
> 
> ping -c 1 -p <up to 30 hex digit val><hex val of sysrq> <target_system_name>
> 
> ex) sysrq-m, m is ascii 0x6d
> 
> ping -c 1 p 1623a06f554d46d676d <target_system_name>
> 
> Note that the network sysrq automatically disables after the receipt of
> the ping, ie) it is single-shot mode.  If you want to use this again, you
> must complete the above four steps again.
> 
> Signed-off-by: Prarit Bhargava <prarit@redhat.com>

Isn't same functionality already available with netconsole?
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Tue, Jun 21, 2011 at 10:08:58AM -0700, Stephen Hemminger wrote:
> On Tue, 21 Jun 2011 09:00:40 -0400
> Prarit Bhargava <prarit@redhat.com> wrote:
> 
> > Add Network Sysrq Support
> > 
> > In some circumstances, a system can hang/lockup in such a way that the system
> > is completely unresponsive to keyboard or console input but is still
> > responsive to ping.  The config option, CONFIG_SYSRQ_PING, builds
> > net/ipv4/sysrq-ping.ko which allows a root user to configure the system for
> > a remote sysrq.
> > 
> > To use this do:
> > 
> > mount -t debugfs none /sys/kernel/debug/
> > echo 1 > /proc/sys/kernel/sysrq
> > echo <hex digit val> > /sys/kernel/debug/network_sysrq_magic
> > echo 1 > /sys/kernel/debug/network_sysrq_enable
> > 
> > Then on another system on the network you can do:
> > 
> > ping -c 1 -p <up to 30 hex digit val><hex val of sysrq> <target_system_name>
> > 
> > ex) sysrq-m, m is ascii 0x6d
> > 
> > ping -c 1 p 1623a06f554d46d676d <target_system_name>
> > 
> > Note that the network sysrq automatically disables after the receipt of
> > the ping, ie) it is single-shot mode.  If you want to use this again, you
> > must complete the above four steps again.
> > 
> > Signed-off-by: Prarit Bhargava <prarit@redhat.com>
> 
> Isn't same functionality already available with netconsole?
Negative, netconsole is a unidirectional protocol.  It registers no rx_hook and
so it doesn't receive frames.  It could be added, and we could add this
functionality that way, but that would seem wierd (to me at least):

1) It would mean that netconsole was writeable, which means we would need a
utility on the remote server to do that writing to the proper udp port

2) Said utility in (1) would have some security concerns, unless we implemented
the exact same magic key sequencing that prarit has here.  Otherwise any
attacker could send an unsolicited request to the netconsole port on a system.
I like the idea of keeping this functionality separate from a feature that gets
such wide use in a production environment.

I've hacked this kind of feature together to solve this problem a few times now,
I think it would be good to have it as a well defined feature permanently.
Acked-by: Neil Horman <nhorman@tuxdriver.com> 

> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Tue, 21 Jun 2011 09:00:40 -0400 Prarit Bhargava wrote:

> Add Network Sysrq Support

> Signed-off-by: Prarit Bhargava <prarit@redhat.com>
> 
> diff --git a/Documentation/networking/sysrq-ping.txt b/Documentation/networking/sysrq-ping.txt
> new file mode 100644
> index 0000000..efa8be3
> --- /dev/null
> +++ b/Documentation/networking/sysrq-ping.txt
> @@ -0,0 +1,26 @@
> +In some circumstances, a system can hang/lockup in such a way that the system
> +is completely unresponsive to keyboard or console input but is still
> +responsive to ping.  The config option, CONFIG_SYSRQ_PING, builds
> +net/ipv4/sysrq-ping.ko which allows a root user to configure the system for a

or it can be built-in the kernel image... (i.e., not a loadable module)

> +remote sysrq.
> +
> +To use this do:
> +
> +mount -t debugfs none /sys/kernel/debug/
> +echo 1 > /proc/sys/kernel/sysrq
> +echo <hex digit val> > /sys/kernel/debug/network_sysrq_magic
> +echo 1 > /sys/kernel/debug/network_sysrq_enable

so all of this (insecure) stuff has to be done before you suspect that
you need it .. in case the local keyboard/console is dead.

> +
> +Then on another system you can do:
> +
> +ping -c 1 -p <hex digit val><hex val of sysrq> <target_system_name>

What are the <up to 30 hex digits> for?

> +
> +ex) sysrq-m, m is ascii 0x6d

e.g., sysrq-m (m is ASCII 0x6d)

> +
> +    ping -c 1 p 1623a06f554d46d676d <target_system_name>
> +
> +Note that the network sysrq automatically disables after the receipt of
> +the ping, ie) it is single-shot mode.  If you want to use this again, you

             i.e., it is single-shot mode.

> +must complete the above four steps again.
> +
> +Hint: 'man ascii' ;)
> diff --git a/net/ipv4/Kconfig b/net/ipv4/Kconfig
> index cbb505b..03bb7b1 100644
> --- a/net/ipv4/Kconfig
> +++ b/net/ipv4/Kconfig
> @@ -624,3 +624,11 @@ config TCP_MD5SIG
>  	  on the Internet.
>  
>  	  If unsure, say N.
> +
> +config SYSRQ_PING
> +	tristate

Needs a prompt?

> +	default m

Please do not enable random code to be built.

> +	help
> +	  Allows execution of sysrq-X commands via ping over ipv4.  This is a

No ipv6 support?  future?

> +	  known security hazard and should not be used in unsecure

	                                                  insecure

> +	  environments.


---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Randy Dunlap <rdunlap@xenotime.net> wrote:
> > diff --git a/Documentation/networking/sysrq-ping.txt b/Documentation/networking/sysrq-ping.txt
> > new file mode 100644
> > index 0000000..efa8be3
> > --- /dev/null
> > +++ b/Documentation/networking/sysrq-ping.txt
> > @@ -0,0 +1,26 @@
> > +In some circumstances, a system can hang/lockup in such a way that the system
> > +is completely unresponsive to keyboard or console input but is still
> > +responsive to ping.  The config option, CONFIG_SYSRQ_PING, builds
> > +net/ipv4/sysrq-ping.ko which allows a root user to configure the system for a
> 
> or it can be built-in the kernel image... (i.e., not a loadable module)
> 
> > +remote sysrq.
> > +
> > +To use this do:
> > +
> > +mount -t debugfs none /sys/kernel/debug/
> > +echo 1 > /proc/sys/kernel/sysrq
> > +echo <hex digit val> > /sys/kernel/debug/network_sysrq_magic
> > +echo 1 > /sys/kernel/debug/network_sysrq_enable
> 
> so all of this (insecure) stuff has to be done before you suspect that
> you need it .. in case the local keyboard/console is dead.

There is an xt_SYSREQ module in xtables-addons package (i.e., a
netfilter target), it supports hashed passwords and has some sequence
number scheme to avoid replays.

I think it would make more sense to merge that upstream, simply because
it seems to be a bit more advanced...
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Tue, 21 Jun 2011 22:37:20 +0200 Florian Westphal wrote:

> Randy Dunlap <rdunlap@xenotime.net> wrote:
> > > diff --git a/Documentation/networking/sysrq-ping.txt b/Documentation/networking/sysrq-ping.txt
> > > new file mode 100644
> > > index 0000000..efa8be3
> > > --- /dev/null
> > > +++ b/Documentation/networking/sysrq-ping.txt
> > > @@ -0,0 +1,26 @@
> > > +In some circumstances, a system can hang/lockup in such a way that the system
> > > +is completely unresponsive to keyboard or console input but is still
> > > +responsive to ping.  The config option, CONFIG_SYSRQ_PING, builds
> > > +net/ipv4/sysrq-ping.ko which allows a root user to configure the system for a
> > 
> > or it can be built-in the kernel image... (i.e., not a loadable module)
> > 
> > > +remote sysrq.
> > > +
> > > +To use this do:
> > > +
> > > +mount -t debugfs none /sys/kernel/debug/
> > > +echo 1 > /proc/sys/kernel/sysrq
> > > +echo <hex digit val> > /sys/kernel/debug/network_sysrq_magic
> > > +echo 1 > /sys/kernel/debug/network_sysrq_enable
> > 
> > so all of this (insecure) stuff has to be done before you suspect that
> > you need it .. in case the local keyboard/console is dead.
> 
> There is an xt_SYSREQ module in xtables-addons package (i.e., a
> netfilter target), it supports hashed passwords and has some sequence
> number scheme to avoid replays.
> 
> I think it would make more sense to merge that upstream, simply because
> it seems to be a bit more advanced...

in which case I would prefer this patch instead of a netfilter patch.  :)

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 06/21/2011 10:00 AM, Prarit Bhargava wrote:
> Add Network Sysrq Support
> 
> In some circumstances, a system can hang/lockup in such a way that the system
> is completely unresponsive to keyboard or console input but is still
> responsive to ping.  The config option, CONFIG_SYSRQ_PING, builds
> net/ipv4/sysrq-ping.ko which allows a root user to configure the system for
> a remote sysrq.
> 
> To use this do:
> 
> mount -t debugfs none /sys/kernel/debug/
> echo 1 > /proc/sys/kernel/sysrq
> echo <hex digit val> > /sys/kernel/debug/network_sysrq_magic
> echo 1 > /sys/kernel/debug/network_sysrq_enable
> 
> Then on another system on the network you can do:
> 
> ping -c 1 -p <up to 30 hex digit val><hex val of sysrq> <target_system_name>
> 
> ex) sysrq-m, m is ascii 0x6d
> 
> ping -c 1 p 1623a06f554d46d676d <target_system_name>

missing '-' in front of 'p'


> 
> Note that the network sysrq automatically disables after the receipt of
> the ping, ie) it is single-shot mode.  If you want to use this again, you
> must complete the above four steps again.
> 
> Signed-off-by: Prarit Bhargava <prarit@redhat.com>
> 
> diff --git a/Documentation/networking/sysrq-ping.txt b/Documentation/networking/sysrq-ping.txt
> new file mode 100644
> index 0000000..efa8be3
> --- /dev/null
> +++ b/Documentation/networking/sysrq-ping.txt
> @@ -0,0 +1,26 @@
> +In some circumstances, a system can hang/lockup in such a way that the system
> +is completely unresponsive to keyboard or console input but is still
> +responsive to ping.  The config option, CONFIG_SYSRQ_PING, builds
> +net/ipv4/sysrq-ping.ko which allows a root user to configure the system for a
> +remote sysrq.
> +
> +To use this do:
> +
> +mount -t debugfs none /sys/kernel/debug/
> +echo 1 > /proc/sys/kernel/sysrq
> +echo <hex digit val> > /sys/kernel/debug/network_sysrq_magic
> +echo 1 > /sys/kernel/debug/network_sysrq_enable
> +
> +Then on another system you can do:
> +
> +ping -c 1 -p <hex digit val><hex val of sysrq> <target_system_name>
> +
> +ex) sysrq-m, m is ascii 0x6d
> +
> +    ping -c 1 p 1623a06f554d46d676d <target_system_name>

again -p

> +
> +Note that the network sysrq automatically disables after the receipt of
> +the ping, ie) it is single-shot mode.  If you want to use this again, you
> +must complete the above four steps again.
> +
> +Hint: 'man ascii' ;)
> diff --git a/net/ipv4/Kconfig b/net/ipv4/Kconfig
> index cbb505b..03bb7b1 100644
> --- a/net/ipv4/Kconfig
> +++ b/net/ipv4/Kconfig
> @@ -624,3 +624,11 @@ config TCP_MD5SIG
>  	  on the Internet.
>  
>  	  If unsure, say N.
> +
> +config SYSRQ_PING
> +	tristate
> +	default m

prompt?


> +	help
> +	  Allows execution of sysrq-X commands via ping over ipv4.  This is a
> +	  known security hazard and should not be used in unsecure
> +	  environments.
> diff --git a/net/ipv4/Makefile b/net/ipv4/Makefile
> index f2dc69c..c23c15e 100644
> --- a/net/ipv4/Makefile
> +++ b/net/ipv4/Makefile
> @@ -48,6 +48,7 @@ obj-$(CONFIG_TCP_CONG_LP) += tcp_lp.o
>  obj-$(CONFIG_TCP_CONG_YEAH) += tcp_yeah.o
>  obj-$(CONFIG_TCP_CONG_ILLINOIS) += tcp_illinois.o
>  obj-$(CONFIG_NETLABEL) += cipso_ipv4.o
> +obj-$(CONFIG_SYSRQ_PING) += sysrq-ping.o
>  
>  obj-$(CONFIG_XFRM) += xfrm4_policy.o xfrm4_state.o xfrm4_input.o \
>  		      xfrm4_output.o
> diff --git a/net/ipv4/sysrq-ping.c b/net/ipv4/sysrq-ping.c
> new file mode 100644
> index 0000000..67a6d0e
> --- /dev/null
> +++ b/net/ipv4/sysrq-ping.c
> @@ -0,0 +1,207 @@
> +/*
> + * network_sysrq.c - allow sysrq to be executed over a network via ping
> + *
> + * written by:  Prarit Bhargava <prarit@redhat.com>
> + *		Andy Gospodarek <agospoda@redhat.com>
> + *		Neil Horman <nhorman@redhat.com>
> + *
> + * based on work by:	Larry Woodman <lwoodman@redhat.com>
> + *
> + * To use this do:
> + *
> + *	mount -t debugfs none /sys/kernel/debug/
> + *	echo 1 > /proc/sys/kernel/sysrq
> + *	echo <hex digit val> > /sys/kernel/debug/network_sysrq_magic
> + *	echo 1 > /sys/kernel/debug/network_sysrq_enable
> + *
> + * Then on another system you can do:
> + *
> + *	ping -c 1 -p <hex digit val><hex val of sysrq> <target_system_name>
> + *
> + *	ex) sysrq-m, m is 0x6d
> + *
> + *	    ping -c 1 p 1623a06f554d46d676d <target_system_name>

again -p

> + *
> + * Note that the network sysrq automatically disables after the receipt of
> + * *ANY* ping.  If you want to use this again, you must complete the
> + * above four steps again.

why not leave it running? perhaps disable it for some time to
filter any ping retransmission, for instance netsysrq_reenable=10 seconds. 
and then do if (time_after(jiffies, last_ping + netsysrq_reenable))...

I have an USB KVM console, so often when testing workload, the USB takes
ages to register the keyboard and I'd like to shot some sysrq+t or w,
so if it disables after the first one, I can't get more traces because
I simply can't enable it again.


> + *
> + */
> +
> +#include <linux/debugfs.h>
> +#include <linux/icmp.h>
> +#include <linux/init.h>
> +#include <linux/ip.h>
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +#include <linux/netdevice.h>
> +#include <linux/sysrq.h>
> +
> +#include <net/xfrm.h>
> +
> +static u8 network_sysrq_enable; /* set in debugfs network_sysrq_enable */
> +static u16 network_sysrq_magic[16]; /* 15 bytes leaves 1 feature byte */
> +static int network_sysrq_magic_len;
> +
> +static int to_hex(int val)
> +{
> +	if ((val >= '0') && (val <= '9'))
> +		return val - 0x30;
> +
> +	if ((val >= 'a') && (val <= 'f'))
> +		return val - 0x37;
> +
> +	if ((val >= 'A') && (val <= 'F'))
> +		return val - 0x57;
> +
> +	return -1;
> +}
> +
> +static bool network_sysrq_armed(void)
> +{
> +	int i;
> +
> +	if (!network_sysrq_enable)
> +		return false;
> +	if (!network_sysrq_magic_len)
> +		return false;
> +	for (i = 0; i < 16; i++)
> +		if (network_sysrq_magic[i] != 0)
> +			return true;
> +	return false;
> +}
> +
> +static void network_sysrq_disable(void)
> +{
> +	network_sysrq_enable = 0;
> +	memset(network_sysrq_magic, 0, 32);
> +	network_sysrq_magic_len = 0;
> +}
> +
> +static ssize_t network_sysrq_seq_write(struct file *file,
> +				       const char __user *ubuf,
> +				       size_t count, loff_t *ppos)
> +{
> +	int i, j, hi, lo;
> +	char buf[33];
> +	memset(buf, 0, sizeof(buf));
> +
> +	if (count >= 33)
> +		return -EINVAL;
> +
> +	if (copy_from_user(&buf, ubuf, min_t(size_t, sizeof(buf) - 1, count)))
> +		return -EFAULT;
> +
> +	for (i = 0, j = 0; i < count - 2 ; i += 2, j++) {
> +		hi = to_hex(buf[i]);
> +		lo = to_hex(buf[i+1]) & 0x0f;
> +		if ((hi == -1) || (lo == -1)) {
> +			network_sysrq_disable();
> +			return -EINVAL;
> +		}
> +		network_sysrq_magic[j] = (u16)(hi << 4) + lo;
> +	}
> +	network_sysrq_magic_len = j;
> +
> +	return count;
> +}
> +
> +static int network_sysrq_seq_show(struct seq_file *m, void *p)
> +{
> +	int i;
> +
> +	for (i = 0; i < network_sysrq_magic_len; i++)
> +		seq_printf(m, "%02x", network_sysrq_magic[i]);
> +	seq_printf(m, "\n");
> +	return 0;
> +}
> +
> +static int network_sysrq_fops_open(struct inode *inode, struct file *file)
> +{
> +	return single_open(file, network_sysrq_seq_show, inode->i_private);
> +}
> +
> +static const struct file_operations xnetwork_sysrq_fops = {
> +	.open		= network_sysrq_fops_open,
> +	.write		= network_sysrq_seq_write,
> +	.read		= seq_read,
> +	.llseek		= seq_lseek,
> +};
> +
> +static int network_sysrq_func(struct sk_buff *skb, struct net_device *dev,
> +			      struct packet_type *pt,
> +			      struct net_device *orig_dev)
> +{
> +	struct icmphdr *icmph;
> +	char *found;
> +
> +	if (ip_hdr(skb)->protocol != IPPROTO_ICMP)
> +		goto end;
> +
> +	if (!skb_pull(skb, sizeof(struct iphdr)))
> +		goto end;
> +
> +	skb_reset_transport_header(skb);
> +	icmph = icmp_hdr(skb);
> +
> +	if (!skb_pull(skb, sizeof(*icmph)))
> +		goto end;
> +
> +	/* is this a ping? */
> +	if (icmph->type != ICMP_ECHO)
> +		goto end;


What about a whitelist of source MAC or IP addresses to accept the sysrq?

I'm thinking on a situation where we leave the systems with this enabled
and then an ordinary user starts pinging the network guessing the hexa to
cause reboots.

Tested here and it works.
Tested-by: Flavio Leitner <fbl@redhat.com>

fbl

> +	if (network_sysrq_armed()) {
> +		found = strnstr(skb->data, (char *)network_sysrq_magic,
> +				skb->len - skb->data_len);
> +		if (found)
> +			handle_sysrq(found[network_sysrq_magic_len]);
> +		network_sysrq_disable();
> +	}
> +end:
> +	kfree_skb(skb);
> +	return 0;
> +}
> +
> +static struct packet_type network_sysrq_type = {
> +	.type = cpu_to_be16(ETH_P_IP),
> +	.func = network_sysrq_func,
> +};
> +
> +static struct dentry *network_sysrq_enable_dentry;
> +static struct dentry *network_sysrq_magic_dentry;
> +
> +int __init init_network_sysrq(void)
> +{
> +	network_sysrq_enable_dentry = debugfs_create_u8("network_sysrq_enable",
> +							S_IWUGO | S_IRUGO,
> +							NULL,
> +							&network_sysrq_enable);
> +	if (!network_sysrq_enable_dentry)
> +		return -EIO;
> +
> +	network_sysrq_magic_dentry = debugfs_create_file("network_sysrq_magic",
> +							S_IWUGO | S_IRUGO,
> +							NULL,
> +							&network_sysrq_magic,
> +							&xnetwork_sysrq_fops);
> +	if (!network_sysrq_magic_dentry) {
> +		debugfs_remove(network_sysrq_enable_dentry);
> +		return -EIO;
> +	}
> +
> +	dev_add_pack(&network_sysrq_type);
> +	return 0;
> +}
> +
> +void __exit cleanup_network_sysrq(void)
> +{
> +	dev_remove_pack(&network_sysrq_type);
> +	debugfs_remove(network_sysrq_enable_dentry);
> +	debugfs_remove(network_sysrq_magic_dentry);
> +}
> +
> +module_init(init_network_sysrq);
> +module_exit(cleanup_network_sysrq);
> +
> +MODULE_LICENSE("GPL");
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 06/21/2011 04:09 PM, Randy Dunlap wrote:
>
>
>> +remote sysrq.
>> +
>> +To use this do:
>> +
>> +mount -t debugfs none /sys/kernel/debug/
>> +echo 1 > /proc/sys/kernel/sysrq
>> +echo <hex digit val> > /sys/kernel/debug/network_sysrq_magic
>> +echo 1 > /sys/kernel/debug/network_sysrq_enable
>>     
> so all of this (insecure) stuff has to be done before you suspect that
> you need it .. in case the local keyboard/console is dead.
>
>   

Randy,

Yes -- this needs to be enabled before the system is unresponsive.  I've
used it to sysrq-c systems that were reproducibly hanging in ways that
the system could not be accessed via console or keyboard.  It really is
a debug feature.

[I've also sysrq-t'd and sysrq-m'd systems as well fwiw]

>> +
>> +Then on another system you can do:
>> +
>> +ping -c 1 -p <hex digit val><hex val of sysrq> <target_system_name>
>>     
> What are the <up to 30 hex digits> for?
>
>   

A secret key.  By turning this on you're leaving a system susceptible to
a panic by anyone else who decides to ping your system.  The ping
payload accepts up to a 32 digit hex number.  Two of those are needed
for the sysrq type, so that leaves 30.

I figured a 30 digit hash, with multiple enabling steps, and single-shot
mode, was good enough to alleviate any security issues anyone would have
about this code.

I'll take all of your suggestions into [v2].

Thanks for the review,

P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

>> + * Note that the network sysrq automatically disables after the receipt of
>> + * *ANY* ping.  If you want to use this again, you must complete the
>> + * above four steps again.
>>     
>   

Hi Flavio,

> why not leave it running? perhaps disable it for some time to
> filter any ping retransmission, for instance netsysrq_reenable=10 seconds. 
> and then do if (time_after(jiffies, last_ping + netsysrq_reenable))...
>
>   

To be honest I have no objection to that, except that I thought the
community's objection would be about the security of this code -- I
specifically set out to make this as secure as I could (multiple
enabling steps, 30 digit hash, single-shot mode).

With your method someone *theoretically* could ping -p a system and
eventually crash it.  Of course anyone using this in production is crazy
and probably would end up getting what they deserved ;)

I'm certainly willing to consider your proposal -- but I have a feeling
that people would want this in single-shot mode over a rearming-mode.

> I have an USB KVM console, so often when testing workload, the USB takes
> ages to register the keyboard and I'd like to shot some sysrq+t or w,
> so if it disables after the first one, I can't get more traces because
> I simply can't enable it again.
>
>   

Yup.  One of the thoughts was to have the enable sysfs file take a
number from 1 to 5, which would become a countdown.  I thought it better
to get the base code in than muddy the waters with a potentially risky
(security-wise) proposal. (see later comment on ping flooding)

>   
>>     
>
> What about a whitelist of source MAC or IP addresses to accept the sysrq?
>   

Oh ... that's a great idea :).  I'll think about that for a later
enhancement if that's okay.  One of the other ideas that came up during
discussion was a maskfile of some sort that indicated what sysrq options
were available (maybe you *don't* want a sysrq-c to be executed).

> I'm thinking on a situation where we leave the systems with this enabled
> and then an ordinary user starts pinging the network guessing the hexa to
> cause reboots.
>
>   

Good point Flavio, but that's *exactly* why I wrote this in single-shot
mode.  I really think the code might be a bit too risky for most people
to deploy in production environments.  It's too risky for me to let
someone ping and ping and ping until they luckily hit the magic number
and figure out how to bring *all* of my systems down.  What are the
chances that a lab admin is smart enough to set the password to
different numbers across different machines in a single lab?

P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Flavio Leitner <fbl@redhat.com> wrote:
> What about a whitelist of source MAC or IP addresses to accept the sysrq?

This is one of the reasons why I still think that
xt_SYSREQ would be the better solution, you get all
kinds of filtering features for free.

You could even use crazy things like '-m time' to restrict
sysreq availability to working hours and whatnot.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

From: Florian Westphal <fw@strlen.de>
Date: Wed, 22 Jun 2011 00:56:45 +0200

> Flavio Leitner <fbl@redhat.com> wrote:
>> What about a whitelist of source MAC or IP addresses to accept the sysrq?
> 
> This is one of the reasons why I still think that
> xt_SYSREQ would be the better solution, you get all
> kinds of filtering features for free.
> 
> You could even use crazy things like '-m time' to restrict
> sysreq availability to working hours and whatnot.

Agreed.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 06/21/2011 07:26 PM, Prarit Bhargava wrote:
>> I'm thinking on a situation where we leave the systems with this enabled
>> and then an ordinary user starts pinging the network guessing the hexa to
>> cause reboots.
>>   
> 
> Good point Flavio, but that's *exactly* why I wrote this in single-shot
> mode.  I really think the code might be a bit too risky for most people
> to deploy in production environments.  It's too risky for me to let
> someone ping and ping and ping until they luckily hit the magic number
> and figure out how to bring *all* of my systems down.  What are the
> chances that a lab admin is smart enough to set the password to
> different numbers across different machines in a single lab?

I see your point.  I liked the patch because of the simplicity but
oh well, if we care that much about the security, then in the end
we will have something similar to what the xt_SYSRQ does already.

fbl

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Tue, 21 Jun 2011 09:00:40 -0400, Prarit Bhargava wrote:

> Add Network Sysrq Support
> 
> In some circumstances, a system can hang/lockup in such a way that the
> system is completely unresponsive to keyboard or console input but is
> still responsive to ping.  The config option, CONFIG_SYSRQ_PING, builds
> net/ipv4/sysrq-ping.ko which allows a root user to configure the system
> for a remote sysrq.
> 

Cool work, Prarit! Now we have remote sysrq!

I am just wondering why the module is in net/ipv4/sysrq-ping.c?
Not drivers/net/sysrq-ping.c? or perhaps net/core/sysrq-ping.c?
I understand it only supports IPv4 now, but we can add IPv6 support,
can't we? ;)

Thanks!

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 06/21/2011 06:58 PM, David Miller wrote:
> From: Florian Westphal <fw@strlen.de>
> Date: Wed, 22 Jun 2011 00:56:45 +0200
>
>   
>> Flavio Leitner <fbl@redhat.com> wrote:
>>     
>>> What about a whitelist of source MAC or IP addresses to accept the sysrq?
>>>       
>> This is one of the reasons why I still think that
>> xt_SYSREQ would be the better solution, you get all
>> kinds of filtering features for free.
>>
>> You could even use crazy things like '-m time' to restrict
>> sysreq availability to working hours and whatnot.
>>     
> Agreed.
>   

It is not the same thing.  At least AFAICT ... I have a system in which
we're experiencing random pauses.  That is, unlike a hang or a
softlockup the system just (for lack of a better word) slows down.

Typing through the keyboard is lagged as is output on the console. 
Running the date command in the background shows that the date is not
updated for 30-40 seconds.  I got lucky this morning, logged in an
noticed that the system I've been watching was in this state :)

Using the ping, I can get a sysrq-c/t/m/whatever into the system
immediately.

Using the netfilter xt-SYSRQ code seems to store the entered code and
execute it later after the system has returned to a normal state....
which is much too late to be useful.

I want/need an *immediate* way to sysrq the machine.

P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

From: Prarit Bhargava <prarit@redhat.com>
Date: Wed, 22 Jun 2011 06:26:23 -0400

> Using the netfilter xt-SYSRQ code seems to store the entered code and
> execute it later after the system has returned to a normal state....
> which is much too late to be useful.

So add a feature that allows xt-SYSRQ to not queue the request
and instead execute it immediately in softirq context.

I don't see what the big deal is.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 06/22/2011 06:35 AM, David Miller wrote:
> From: Prarit Bhargava <prarit@redhat.com>
> Date: Wed, 22 Jun 2011 06:26:23 -0400
>
>   
>> Using the netfilter xt-SYSRQ code seems to store the entered code and
>> execute it later after the system has returned to a normal state....
>> which is much too late to be useful.
>>     
> So add a feature that allows xt-SYSRQ to not queue the request
> and instead execute it immediately in softirq context.
>
> I don't see what the big deal is.
>   

I'm not familiar with the netfilter code and was about to ask some other
colleagues about it ... but I think you answered my main question which
was "does netfilter do things in interrupt context?"

P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Prarit Bhargava <prarit@redhat.com> wrote:

[ cc'd John Haxby, who worked on xt_SYSREQ ]

> On 06/21/2011 06:58 PM, David Miller wrote:
> > From: Florian Westphal <fw@strlen.de>
> > Date: Wed, 22 Jun 2011 00:56:45 +0200
> >> This is one of the reasons why I still think that
> >> xt_SYSREQ would be the better solution, you get all
> >> kinds of filtering features for free.
> >>
> >> You could even use crazy things like '-m time' to restrict
> >> sysreq availability to working hours and whatnot.
> >>
> > Agreed.
> Using the netfilter xt-SYSRQ code seems to store the entered code and
> execute it later after the system has returned to a normal state....
> which is much too late to be useful.

The target handler of the kernel part invokes handle_sysrq(),
I don't see any delaying/queueing?

FWIW, the old discussion is in the archives:
search for subject "nf-next: sysrq and condition 20100421" from Jan
Engelhardt, or try
http://thread.gmane.org/gmane.comp.security.firewalls.netfilter.devel/33615/focus=34808

As far as i understand the use case described by John Haxby matches yours.

Patrick McHardy suggested an alternative standalone method involving
encapsulation sockets; perhaps the reasons why this path was not chosen
have changed.

I think that a standalone module (i.e. not requiring netfilter) that
runs the sysreq handling after all netfilter hooks would be optimal,
but I don't see a simple method to implement that.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 06/22/2011 06:54 AM, Florian Westphal wrote:
> Prarit Bhargava <prarit@redhat.com> wrote:
>
> [ cc'd John Haxby, who worked on xt_SYSREQ ]
>
>   
>> On 06/21/2011 06:58 PM, David Miller wrote:
>>     
>>> From: Florian Westphal <fw@strlen.de>
>>> Date: Wed, 22 Jun 2011 00:56:45 +0200
>>>       
>>>> This is one of the reasons why I still think that
>>>> xt_SYSREQ would be the better solution, you get all
>>>> kinds of filtering features for free.
>>>>
>>>> You could even use crazy things like '-m time' to restrict
>>>> sysreq availability to working hours and whatnot.
>>>>
>>>>         
>>> Agreed.
>>>       
>> Using the netfilter xt-SYSRQ code seems to store the entered code and
>> execute it later after the system has returned to a normal state....
>> which is much too late to be useful.
>>     
> The target handler of the kernel part invokes handle_sysrq(),
> I don't see any delaying/queueing?
>   

Yeah ... It just occurred to me on the way into the office that I made a
mistake.

... What if the "slow down" I'm seeing is due to the network layer?  ...
but why would ping work and the xt_SYSRQ not work? 

Either way ... I have a feeling that I'm on to something,

P.

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 22/06/11 11:54, Florian Westphal wrote:
> Prarit Bhargava <prarit@redhat.com> wrote:
>
> [ cc'd John Haxby, who worked on xt_SYSREQ ]
>
>> On 06/21/2011 06:58 PM, David Miller wrote:
>>> From: Florian Westphal <fw@strlen.de>
>>> Date: Wed, 22 Jun 2011 00:56:45 +0200
>>>> This is one of the reasons why I still think that
>>>> xt_SYSREQ would be the better solution, you get all
>>>> kinds of filtering features for free.
>>>>
>>>> You could even use crazy things like '-m time' to restrict
>>>> sysreq availability to working hours and whatnot.
>>>>
>>> Agreed.
>> Using the netfilter xt-SYSRQ code seems to store the entered code and
>> execute it later after the system has returned to a normal state....
>> which is much too late to be useful.
> The target handler of the kernel part invokes handle_sysrq(),
> I don't see any delaying/queueing?
>
> FWIW, the old discussion is in the archives:
> search for subject "nf-next: sysrq and condition 20100421" from Jan
> Engelhardt, or try
> http://thread.gmane.org/gmane.comp.security.firewalls.netfilter.devel/33615/focus=34808
>
> As far as i understand the use case described by John Haxby matches yours.
>
> Patrick McHardy suggested an alternative standalone method involving
> encapsulation sockets; perhaps the reasons why this path was not chosen
> have changed.
>
> I think that a standalone module (i.e. not requiring netfilter) that
> runs the sysreq handling after all netfilter hooks would be optimal,
> but I don't see a simple method to implement that.

The xt_SYSRQ calls handle_sysrq() in BH context, much the same context
as ping is handled in.  (Actually, it's likely xt_SYSRQ will work even
if ping doesn't since nothing has to come back.)

It's possible for xt_SYSRQ to fail.   My usual case for failure was
simply not enabling it :-)  However, as you typically have to fight your
way through iptables to get to xt_SYSRQ then you can get into trouble
that way.

Although I wasn't sure that it could happen, it's also possible that the
cryptographic functions can get in your way.  xt_SYSRQ does its best to
avoid problems by pre-allocating everything it can so there is as little
as possible to do when it is needed, but it is possible for it to fail.

The module that Patrick McHardy suggested works up to a point:
handle_sysrq() can still be called in BH context but unfortunately I
couldn't get it working for IPv6: the necessary hook isn't implemented
for IPv6 (or rather, it wasn't, I don't know if something has changed
since then).

jch
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Prarit Bhargava <prarit@redhat.com> writes:
>
> Note that the network sysrq automatically disables after the receipt of
> the ping, ie) it is single-shot mode.  If you want to use this again, you
> must complete the above four steps again.
 
Good idea, but: why not add a single key (e.g. a hash digest with 
the key hashed in) and make it reasonably secure?  
That should not be that hard and would make it a lot more useful.

It would not defend against all attacks, but probably make it
"LAN grade" at least.

-Andi

> Although I wasn't sure that it could happen, it's also possible that the
> cryptographic functions can get in your way.  xt_SYSRQ does its best to
> avoid problems by pre-allocating everything it can so there is as little
> as possible to do when it is needed, but it is possible for it to fail.
>
>   

My running theory as to the failure is that the CPU that took the sysrq
is also the CPU that was having problems that resulted in the "slow
down" of the system.

On a known-good system, xt_SYSRQ behaves properly AFAICT.  It functions
exactly the way we want it to.

So ... I read the following discussion of xt_SYSRQ from last year:

http://www.kerneltrap.com/mailarchive/linux-netdev/2010/4/21/6275199/thread

And it seems there were no technical objections to the code, but there
were other concerns.

davem -- as I don't monitor this list, are you indicating that you are
more amenable to this code being accepted upstream?  Or is that part of
the debate still ongoing?

P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 22 Jun 2011, at 18:39, Prarit Bhargava wrote:

> 
>> Although I wasn't sure that it could happen, it's also possible that the
>> cryptographic functions can get in your way.  xt_SYSRQ does its best to
>> avoid problems by pre-allocating everything it can so there is as little
>> as possible to do when it is needed, but it is possible for it to fail.
>> 
>> 
> 
> My running theory as to the failure is that the CPU that took the sysrq
> is also the CPU that was having problems that resulted in the "slow
> down" of the system.
> 
> On a known-good system, xt_SYSRQ behaves properly AFAICT.  It functions
> exactly the way we want it to.
> 
> So ... I read the following discussion of xt_SYSRQ from last year:
> 
> http://www.kerneltrap.com/mailarchive/linux-netdev/2010/4/21/6275199/thread
> 
> And it seems there were no technical objections to the code, but there
> were other concerns.
> 
> davem -- as I don't monitor this list, are you indicating that you are
> more amenable to this code being accepted upstream?  Or is that part of
> the debate still ongoing?
> 
> 

I've just re-read the thread and I'm reminded that I never did submit the xt_SYSRQ hash update I mentioned at the end of the thread.

davem: I'm more than happy to push that patch through if it will make xt_SYSRQ more acceptable.

jch

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 22 Jun 2011, at 18:39, Prarit Bhargava wrote:

> 
>> Although I wasn't sure that it could happen, it's also possible that the
>> cryptographic functions can get in your way.  xt_SYSRQ does its best to
>> avoid problems by pre-allocating everything it can so there is as little
>> as possible to do when it is needed, but it is possible for it to fail.
>> 
>> 
> 
> My running theory as to the failure is that the CPU that took the sysrq
> is also the CPU that was having problems that resulted in the "slow
> down" of the system.
> 
> On a known-good system, xt_SYSRQ behaves properly AFAICT.  It functions
> exactly the way we want it to.
> 
> So ... I read the following discussion of xt_SYSRQ from last year:
> 
> http://www.kerneltrap.com/mailarchive/linux-netdev/2010/4/21/6275199/thread
> 
> And it seems there were no technical objections to the code, but there
> were other concerns.
> 
> davem -- as I don't monitor this list, are you indicating that you are
> more amenable to this code being accepted upstream?  Or is that part of
> the debate still ongoing?
> 
> 

I've just re-read the thread and I'm reminded that I never did submit the xt_SYSRQ hash update I mentioned at the end of the thread.



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

From: Prarit Bhargava <prarit@redhat.com>
Date: Wed, 22 Jun 2011 13:39:15 -0400

> davem -- as I don't monitor this list, are you indicating that you are
> more amenable to this code being accepted upstream?  Or is that part of
> the debate still ongoing?

I'd rather have something like xt_SYSRQ in the tree than the original
proposal in this thread.

Work it out with the netfilter developers if you want to see it
integrated.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

From: John Haxby <jch@thehaxbys.co.uk>
Date: Wed, 22 Jun 2011 19:46:20 +0100

> davem: I'm more than happy to push that patch through if it will
> make xt_SYSRQ more acceptable.

It's not me who reviews and approves netfilter patches, it's
the netfilter developers who do.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 22/06/11 11:54, Florian Westphal wrote:
> Patrick McHardy suggested an alternative standalone method involving
> encapsulation sockets; perhaps the reasons why this path was not chosen
> have changed.
>
> I think that a standalone module (i.e. not requiring netfilter) that
> runs the sysreq handling after all netfilter hooks would be optimal,
> but I don't see a simple method to implement that.

Having just read the entire thread I have some different comments.

I think I mentioned that the standalone module didn't fly because I was
using an encapsulation socket: this preserved the ability to protect the
sysrq with iptables and also kept things nice and simple.  Unfortunately
that also meant I lost IPv6 ....

One of the comments in the thread (sorry, I've lost the attribution, not
to mention the exact quote) was that you would be crazy to run this in
production.    Hmmm.   One of the principle use cases of this is
precisely to run the code in production: machines in production do go
AWOL for all kinds of reasons and being able to run sysrq-m, t, s and c
is particularly useful.   It would be nice to be able to go up to the
machine and type on its keyboard.  If only it was even on the same
continent.   If only it a keyboard.  Or even a PS/2 keyboard socket
(getting a USB keyboard to configure itself when the machine is wedged
is, well, unlikely).

The changes I made to the xt_SYSRQ hashing a while back to avoid things
like replay were precisely because it needs to be run in a production
environment.   I've just submitted a patch that makes replay to other
machines that have the same password less likely to succeed, again, with
a view to how this thing would be used in production.

Sorry if I've repeated some things that have already been said.

jch
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html