[OpenWrt-Devel] uqmi: add timeout parameter
diff mbox series

Message ID 20191107115408.13013-1-zefir.kurtisi@neratec.com
State Under Review
Delegated to: Piotr Dymacz
Headers show
Series
  • [OpenWrt-Devel] uqmi: add timeout parameter
Related show

Commit Message

Zefir Kurtisi Nov. 7, 2019, 11:54 a.m. UTC
Working with Quectel EM12 LTE-module, we observe
regular stalls of the QMI interface which cause
a request issued by uqmi to hang forever.

Most reproducibly this happens after the device
has been power-cycled and left untouched for a
while (~ 60s+). Most of the time the very first
QMI request fails, since it is not responded by
the module. This is the strace from such a run
(from --get-pin-status):

 open("/dev/cdc-wdm0", O_RDWR|O_EXCL|O_NOCTTY|O_NONBLOCK|O_LARGEFILE) = 6
 fcntl64(6, F_GETFL)                     = 0x10802 (flags O_RDWR|O_NONBLOCK|O_LARGEFILE)
 fcntl64(6, F_SETFL, O_RDWR|O_NONBLOCK|O_LARGEFILE) = 0
 epoll_ctl(3, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLRDHUP|EPOLLET, {u32=268567076, u64=1153486808202346496}}) = 0
 write(6, "\1\17\0\0\0\0\0\1\"\0\4\0\1\1\0\2", 16) = 16
 clock_gettime(CLOCK_MONOTONIC, {tv_sec=95, tv_nsec=583444789}) = 0
 clock_gettime(CLOCK_MONOTONIC, {tv_sec=95, tv_nsec=583770264}) = 0
 epoll_pwait(3,
 [ hang forever ]

After killing the blocked uqmi process, the next
request works as expected.

We don't know whether this is a device FW issue
(we use the latest EM12GPAR01A15M4G) or whether
the device enters some undocumented power-save
mode after idling for some time.

This patch extends uqmi with a timeout option
(-t, --timeout <ms>) which if set terminates a
request after the given amount of msecs. In
our usecase it provides a means of preventing
infinitively stuck QMI requests. Since we
observe the issue only for the very first
request after cold-boot, we use a dummy access
early in qmi.sh, e.g.
  uqmi -d /dev/cdc-wdm0 --get-pin-status -t 3000 >/dev/null 2>&1

This ensures the QMI interface is un-stuck in
case it entered the stall-state observed. The
change is intentionally not included in this
commit, since you don't need it if it works
for you.


Signed-off-by: Zefir Kurtisi <zefir.kurtisi@neratec.com>
---
 main.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

Comments

Piotr Dymacz Nov. 16, 2019, 2:59 p.m. UTC | #1
Hi Zefir,

On 07.11.2019 12:54, Zefir Kurtisi wrote:
> Working with Quectel EM12 LTE-module, we observe
> regular stalls of the QMI interface which cause
> a request issued by uqmi to hang forever.
> 
> Most reproducibly this happens after the device
> has been power-cycled and left untouched for a
> while (~ 60s+). Most of the time the very first
> QMI request fails, since it is not responded by
> the module. This is the strace from such a run
> (from --get-pin-status):
> 
>   open("/dev/cdc-wdm0", O_RDWR|O_EXCL|O_NOCTTY|O_NONBLOCK|O_LARGEFILE) = 6
>   fcntl64(6, F_GETFL)                     = 0x10802 (flags O_RDWR|O_NONBLOCK|O_LARGEFILE)
>   fcntl64(6, F_SETFL, O_RDWR|O_NONBLOCK|O_LARGEFILE) = 0
>   epoll_ctl(3, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLRDHUP|EPOLLET, {u32=268567076, u64=1153486808202346496}}) = 0
>   write(6, "\1\17\0\0\0\0\0\1\"\0\4\0\1\1\0\2", 16) = 16
>   clock_gettime(CLOCK_MONOTONIC, {tv_sec=95, tv_nsec=583444789}) = 0
>   clock_gettime(CLOCK_MONOTONIC, {tv_sec=95, tv_nsec=583770264}) = 0
>   epoll_pwait(3,
>   [ hang forever ]
> 
> After killing the blocked uqmi process, the next
> request works as expected.
> 
> We don't know whether this is a device FW issue
> (we use the latest EM12GPAR01A15M4G) or whether
> the device enters some undocumented power-save
> mode after idling for some time.

Could you share this firmware version, is that a generic Quectel or a 
customized one? I would like to reproduce and debug the problem but the 
EM12 I have here has 'EM12GPAR01A_11_M4G'.

Also, what platform do you use this modem with?
Zefir Kurtisi Nov. 17, 2019, 10:23 a.m. UTC | #2
On 11/16/19 3:59 PM, Piotr Dymacz wrote:
> Hi Zefir,
> 
> On 07.11.2019 12:54, Zefir Kurtisi wrote:
>> Working with Quectel EM12 LTE-module, we observe
>> regular stalls of the QMI interface which cause
>> a request issued by uqmi to hang forever.
>>
>> Most reproducibly this happens after the device
>> has been power-cycled and left untouched for a
>> while (~ 60s+). Most of the time the very first
>> QMI request fails, since it is not responded by
>> the module. This is the strace from such a run
>> (from --get-pin-status):
>>
>>   open("/dev/cdc-wdm0", O_RDWR|O_EXCL|O_NOCTTY|O_NONBLOCK|O_LARGEFILE) = 6
>>   fcntl64(6, F_GETFL)                     = 0x10802 (flags
>> O_RDWR|O_NONBLOCK|O_LARGEFILE)
>>   fcntl64(6, F_SETFL, O_RDWR|O_NONBLOCK|O_LARGEFILE) = 0
>>   epoll_ctl(3, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLRDHUP|EPOLLET, {u32=268567076,
>> u64=1153486808202346496}}) = 0
>>   write(6, "\1\17\0\0\0\0\0\1\"\0\4\0\1\1\0\2", 16) = 16
>>   clock_gettime(CLOCK_MONOTONIC, {tv_sec=95, tv_nsec=583444789}) = 0
>>   clock_gettime(CLOCK_MONOTONIC, {tv_sec=95, tv_nsec=583770264}) = 0
>>   epoll_pwait(3,
>>   [ hang forever ]
>>
>> After killing the blocked uqmi process, the next
>> request works as expected.
>>
>> We don't know whether this is a device FW issue
>> (we use the latest EM12GPAR01A15M4G) or whether
>> the device enters some undocumented power-save
>> mode after idling for some time.
> 
> Could you share this firmware version, is that a generic Quectel or a customized
> one? I would like to reproduce and debug the problem but the EM12 I have here has
> 'EM12GPAR01A_11_M4G'.
> 
> Also, what platform do you use this modem with?
> 

Hi Piotr,

we use our own products [1], which are built around a PowerPC (8540) based platform.

The FW we received from Codico [2], Quectel's distributor and support proxy for
Switzerland. We get preview versions on request, therefore I am not sure if it can
be posted publicly. I'll check for restrictions and provide the FW if able.


Cheers


[1] https://www.neratec.com/products
[2] https://www.codico.com/
Zefir Kurtisi Nov. 19, 2019, 10:34 a.m. UTC | #3
On 11/17/19 11:23 AM, Zefir Kurtisi wrote:
> On 11/16/19 3:59 PM, Piotr Dymacz wrote:
>> Hi Zefir,
>>
>> On 07.11.2019 12:54, Zefir Kurtisi wrote:
>>> Working with Quectel EM12 LTE-module, we observe
>>> regular stalls of the QMI interface which cause
>>> a request issued by uqmi to hang forever.
>>>
>>> Most reproducibly this happens after the device
>>> has been power-cycled and left untouched for a
>>> while (~ 60s+). Most of the time the very first
>>> QMI request fails, since it is not responded by
>>> the module. This is the strace from such a run
>>> (from --get-pin-status):
>>>
>>>   open("/dev/cdc-wdm0", O_RDWR|O_EXCL|O_NOCTTY|O_NONBLOCK|O_LARGEFILE) = 6
>>>   fcntl64(6, F_GETFL)                     = 0x10802 (flags
>>> O_RDWR|O_NONBLOCK|O_LARGEFILE)
>>>   fcntl64(6, F_SETFL, O_RDWR|O_NONBLOCK|O_LARGEFILE) = 0
>>>   epoll_ctl(3, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLRDHUP|EPOLLET, {u32=268567076,
>>> u64=1153486808202346496}}) = 0
>>>   write(6, "\1\17\0\0\0\0\0\1\"\0\4\0\1\1\0\2", 16) = 16
>>>   clock_gettime(CLOCK_MONOTONIC, {tv_sec=95, tv_nsec=583444789}) = 0
>>>   clock_gettime(CLOCK_MONOTONIC, {tv_sec=95, tv_nsec=583770264}) = 0
>>>   epoll_pwait(3,
>>>   [ hang forever ]
>>>
>>> After killing the blocked uqmi process, the next
>>> request works as expected.
>>>
>>> We don't know whether this is a device FW issue
>>> (we use the latest EM12GPAR01A15M4G) or whether
>>> the device enters some undocumented power-save
>>> mode after idling for some time.
>>
>> Could you share this firmware version, is that a generic Quectel or a customized
>> one? I would like to reproduce and debug the problem but the EM12 I have here has
>> 'EM12GPAR01A_11_M4G'.
>>
>> Also, what platform do you use this modem with?
>>
> 
> Hi Piotr,
> 
> we use our own products [1], which are built around a PowerPC (8540) based platform.
> 
> The FW we received from Codico [2], Quectel's distributor and support proxy for
> Switzerland. We get preview versions on request, therefore I am not sure if it can
> be posted publicly. I'll check for restrictions and provide the FW if able.
> 
> 
> Cheers
> 
> 
> [1] https://www.neratec.com/products
> [2] https://www.codico.com/
> 
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-devel
> 

Hi Piotr,

Codico/Quectel confirmed that the EM12 FW has no NDA limitations and can be
provided to developers. The archive is available for the next 5 days from our
company file-server at [1]. Let me know if you need something else, otherwise
looking forward whether you can reproduce the observed issue.


Cheers


[1] https://transfer.neratec.com/5e7a98bb1a51e/EM12GPAR01A15M4G.zip
Piotr Dymacz Nov. 19, 2019, 1:21 p.m. UTC | #4
Hi Zefir,

On 19.11.2019 11:34, Zefir Kurtisi wrote:

[...]

>>>> We don't know whether this is a device FW issue
>>>> (we use the latest EM12GPAR01A15M4G) or whether
>>>> the device enters some undocumented power-save
>>>> mode after idling for some time.
>>>
>>> Could you share this firmware version, is that a generic Quectel or a customized
>>> one? I would like to reproduce and debug the problem but the EM12 I have here has
>>> 'EM12GPAR01A_11_M4G'.
>>>
>>> Also, what platform do you use this modem with?
>>>
>> 
>> Hi Piotr,
>> 
>> we use our own products [1], which are built around a PowerPC (8540) based platform.
>> 
>> The FW we received from Codico [2], Quectel's distributor and support proxy for
>> Switzerland. We get preview versions on request, therefore I am not sure if it can
>> be posted publicly. I'll check for restrictions and provide the FW if able.

[...]

> Hi Piotr,
> 
> Codico/Quectel confirmed that the EM12 FW has no NDA limitations and can be
> provided to developers. The archive is available for the next 5 days from our
> company file-server at [1]. Let me know if you need something else, otherwise
> looking forward whether you can reproduce the observed issue.
> 
> 
> Cheers
> 
> 
> [1] https://transfer.neratec.com/5e7a98bb1a51e/EM12GPAR01A15M4G.zip

Thanks! I also contacted my Quectel contact person about that.
Will get back to you with my results in next days.

Patch
diff mbox series

diff --git a/main.c b/main.c
index 9b43e5e..aa4634c 100644
--- a/main.c
+++ b/main.c
@@ -44,6 +44,7 @@  static const struct option uqmi_getopt[] = {
 	{ "keep-client-id", required_argument, NULL, 'k' },
 	{ "release-client-id", required_argument, NULL, 'r' },
 	{ "mbim",  no_argument, NULL, 'm' },
+	{ "timeout", required_argument, NULL, 't' },
 	{ NULL, 0, NULL, 0 }
 };
 #undef __uqmi_command
@@ -57,6 +58,7 @@  static int usage(const char *progname)
 		"  --keep-client-id <name>:          Keep Client ID for service <name>\n"
 		"  --release-client-id <name>:       Release Client ID after exiting\n"
 		"  --mbim, -m                        NAME is an MBIM device with EXT_QMUX support\n"
+		"  --timeout, -t                     response timeout in msecs\n"
 		"\n"
 		"Services:                           dms, nas, pds, wds, wms\n"
 		"\n"
@@ -103,6 +105,14 @@  static void handle_exit_signal(int signal)
 	uloop_end();
 }
 
+static void _request_timeout_handler(struct uloop_timeout *timeout)
+{
+	fprintf(stderr, "Request timed out\n");
+	handle_exit_signal(0);
+}
+
+struct uloop_timeout request_timeout = { .cb = _request_timeout_handler, };
+
 int main(int argc, char **argv)
 {
 	static struct qmi_dev dev;
@@ -112,7 +122,7 @@  int main(int argc, char **argv)
 	signal(SIGINT, handle_exit_signal);
 	signal(SIGTERM, handle_exit_signal);
 
-	while ((ch = getopt_long(argc, argv, "d:k:sm", uqmi_getopt, NULL)) != -1) {
+	while ((ch = getopt_long(argc, argv, "d:k:smt:", uqmi_getopt, NULL)) != -1) {
 		int cmd_opt = CMD_OPT(ch);
 
 		if (ch < 0 && cmd_opt >= 0 && cmd_opt < __UQMI_COMMAND_LAST) {
@@ -136,6 +146,9 @@  int main(int argc, char **argv)
 		case 'm':
 			dev.is_mbim = true;
 			break;
+		case 't':
+			uloop_timeout_set(&request_timeout, atol(optarg));
+			break;
 		default:
 			return usage(argv[0]);
 		}