mwifiex_usb_submit_rx_urb: dev_alloc_skb failed when conected to 5GHz

Hi BR,

I'm using 3.9 mainline mwifiex driver for wireless usb card. Doing some
[ 221.521799] usb 1-1: mwifiex_usb_submit_rx_urb: dev_alloc_skb failed

This is skb allocation failure returned by kernel. 4k buffer is always allocated for Rx packets. This issue doesn't seem to be specific to 5Ghz.

I checked which which size fails to allocate and it's 4096 bytes. I was
looking to changes in never kernel releases but I cannot find anything
obvious. When connected to 2.4GHz I cannot reproduce issue though. I'm
using FW version mwifiex 1.0 (14.68.29.p26).

Could you please provide the platform details?
How often the problem occurs during throughput testing? Are there any specific steps?

Regards,
Amitkumar Karwar
��{.n�+��+%��lzwm��b�맲��r��zX��"��^�ȧ��ܨ}��Ơz�&j:+v��n�r��6;靫3��\
nnX��f�z��

Belisko Marek

2014-09-11 19:14:58 UTC

Dear Amitkumar Karwar,

I'm using 3.9 mainline mwifiex driver for wireless usb card. Doing some
[ 221.521799] usb 1-1: mwifiex_usb_submit_rx_urb: dev_alloc_skb failed

This is skb allocation failure returned by kernel. 4k buffer is always allocated for Rx packets. This issue doesn't seem to be specific to 5Ghz.

OK. But I cannot reproduce this issue when connected to 2.4GHz (same
steps as described below).

Post by Amitkumar Karwar
.

Could you please provide the platform details?
How often the problem occurs during throughput testing? Are there any specific steps?

I can reproduce problem 100% by running iperf as server on board where
wifi is connected to (iperf -s -u -w1M -i1)
and connecting with iperf client from other PC (iperf -c IPADDR -u -b 100m).
Then immediately I can see allocation failures. If you need more info
please let me know. Thanks.

Post by Amitkumar Karwar
Regards,
Amitkumar Karwar

BR,

marek
--
as simple and primitive as possible
-------------------------------------------------
Marek Belisko - OPEN-NANDRA
Freelance Developer

Ruska Nova Ves 219 | Presov, 08005 Slovak Republic
Tel: +421 915 052 184
skype: marekwhite
twitter: #opennandra
web: http://open-nandra.com
--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
the body of a message to majordomo-***@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Belisko Marek

2014-09-17 09:57:12 UTC

Dear Amitkumar Karwar,

some additional info.

I'm using 3.9 mainline mwifiex driver for wireless usb card. Doing some
[ 221.521799] usb 1-1: mwifiex_usb_submit_rx_urb: dev_alloc_skb failed

This is skb allocation failure returned by kernel. 4k buffer is always allocated for Rx packets. This issue doesn't seem to be specific to 5Ghz.

Yes you're right. I can reproduce issue also with 2.4GHz (doing iperf
testing as mentioned in other email) by pinging device with card.

Could you please provide the platform details?
How often the problem occurs during throughput testing? Are there any specific steps?

One more observation is that when problem occurred complete system is
unresponsive (console is almost completely dead).
I can workaround issue by decreasing iperf bandwidth to ~40m. I think
in this situation we're running out of memory by exhaustive skb
allocations.

Post by Amitkumar Karwar
Regards,
Amitkumar Karwar

Amitkumar Karwar

2014-09-17 10:52:52 UTC

Hi BR,

Post by Belisko Marek
Dear Amitkumar Karwar,
some additional info.

Post by Belisko Marek
I'm using 3.9 mainline mwifiex driver for wireless usb card. Doing
some throughput testing (with iperf) in 5GHz I got following
[ 221.521799] usb 1-1: mwifiex_usb_submit_rx_urb: dev_alloc_skb failed

This is skb allocation failure returned by kernel. 4k buffer is

always allocated for Rx packets. This issue doesn't seem to be specific
to 5Ghz.
Yes you're right. I can reproduce issue also with 2.4GHz (doing iperf
testing as mentioned in other email) by pinging device with card.

Post by Belisko Marek
I checked which which size fails to allocate and it's 4096 bytes. I
was looking to changes in never kernel releases but I cannot find
anything obvious. When connected to 2.4GHz I cannot reproduce issue
though. I'm using FW version mwifiex 1.0 (14.68.29.p26).

Could you please provide the platform details?
How often the problem occurs during throughput testing? Are there any

specific steps?
One more observation is that when problem occurred complete system is
unresponsive (console is almost completely dead).

Thanks for the more information.
Skb alloc failure should be gracefully handled. We will look into this issue.

Post by Belisko Marek
I can workaround issue by decreasing iperf bandwidth to ~40m. I think
in this situation we're running out of memory by exhaustive skb
allocations.

Actually 6 4K size buffers are being allocated for Rx and Tx data during traffic.
Probably your platform runs out of memory after these allocations.

Could you please try changing this number(MWIFIEX_TX_DATA_URB/MWIFIEX_RX_DATA_URB macros) to 3?

Regards,
Amitkumar Karwar
N��r��y��b�X��ǧv�^�)޺{.n�+��{��*ޕ�,�{ay�ʇڙ�,j��f��h��/oSc��ڳ9�u��&jw��

Belisko Marek

2014-09-17 12:07:21 UTC

Dear Amitkumar Karwar,

Post by Belisko Marek
Dear Amitkumar Karwar,
some additional info.

This is skb allocation failure returned by kernel. 4k buffer is

Could you please provide the platform details?
How often the problem occurs during throughput testing? Are there any

specific steps?
One more observation is that when problem occurred complete system is
unresponsive (console is almost completely dead).

Thanks for the more information.
Skb alloc failure should be gracefully handled. We will look into this issue.

OK. Thanks. Can you please

Post by Belisko Marek
I can workaround issue by decreasing iperf bandwidth to ~40m. I think
in this situation we're running out of memory by exhaustive skb
allocations.

Actually 6 4K size buffers are being allocated for Rx and Tx data during traffic.
Probably your platform runs out of memory after these allocations.

This could be the issue. I'm running some apps when performing
throughput test and this could lead
to out of memory. When I stop all apps and perform the test it behave
better (I cannot reproduce issue) except of
fact that whole system is unresponsive for ~ 10 secs as was already mentioned.

Post by Amitkumar Karwar
Could you please try changing this number(MWIFIEX_TX_DATA_URB/MWIFIEX_RX_DATA_URB macros) to 3?
Regards,
Amitkumar Karwar

Best Regards,

marekb
--
as simple and primitive as possible
-------------------------------------------------
Marek Belisko - OPEN-NANDRA
Freelance Developer

Ruska Nova Ves 219 | Presov, 08005 Slovak Republic
Tel: +421 915 052 184
skype: marekwhite
twitter: #opennandra
web: http://open-nandra.com
--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
the body of a message to majordomo-***@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Belisko Marek

2014-09-17 12:18:40 UTC

Hi,

I was too fast when hitting Send :)

Post by Belisko Marek
Dear Amitkumar Karwar,

Post by Belisko Marek
Dear Amitkumar Karwar,
some additional info.

This is skb allocation failure returned by kernel. 4k buffer is

Could you please provide the platform details?
How often the problem occurs during throughput testing? Are there any

specific steps?
One more observation is that when problem occurred complete system is
unresponsive (console is almost completely dead).

Thanks for the more information.
Skb alloc failure should be gracefully handled. We will look into this issue.

OK. Thanks. Can you please

Can you please ping me when you will have some patch so I can test it
also on my device.

Post by Belisko Marek
I can workaround issue by decreasing iperf bandwidth to ~40m. I think
in this situation we're running out of memory by exhaustive skb
allocations.

Actually 6 4K size buffers are being allocated for Rx and Tx data during traffic.
Probably your platform runs out of memory after these allocations.

This is not fully true. I run loop iperf test (-t 600) and it will
fail after 4 minutes with (there was around 190MB free memory):

[ 875.249551] usb 1-1: mwifiex_usb_submit_rx_urb: dev_alloc_skb failed
[ 875.256645] usb 1-1: mwifiex_usb_submit_rx_urb: dev_alloc_skb failed
[ 875.263694] usb 1-1: mwifiex_usb_submit_rx_urb: dev_alloc_skb failed
[ 875.270940] usb 1-1: mwifiex_usb_submit_rx_urb: dev_alloc_skb failed
[ 875.277993] usb 1-1: mwifiex_usb_submit_rx_urb: dev_alloc_skb failed
[ 875.285068] usb 1-1: mwifiex_usb_submit_rx_urb: dev_alloc_skb failed
[ 3] 0.0- 1.0 sec 81.8 KBytes 670 Kbits/sec 0.034 ms 0/ 57 (0%)
[ 3] 1.0- 2.0 sec 0.00 Bytes 0.00 bits/sec 0.034 ms 0/ 0 (nan%)
[ 3] 2.0- 3.0 sec 0.00 Bytes 0.00 bits/sec 0.034 ms 0/ 0 (nan%)
[ 3] 3.0- 4.0 sec 0.00 Bytes 0.00 bits/sec 0.034 ms 0/ 0 (nan%)
[ 3] 4.0- 5.0 sec 0.00 Bytes 0.00 bits/sec 0.034 ms 0/ 0 (nan%)
[ 3] 5.0- 6.0 sec 0.00 Bytes 0.00 bits/sec 0.034 ms 0/ 0 (nan%)
[ 3] 6.0- 7.0 sec 0.00 Bytes 0.00 bits/sec 0.034 ms 0/ 0 (nan%)
[ 3] 7.0- 8.0 sec 0.00 Bytes 0.00 bits/sec 0.034 ms 0/ 0 (nan%)
[ 3] 8.0- 9.0 sec 0.00 Bytes 0.00 bits/sec 0.034 ms 0/ 0 (nan%)
[ 3] 9.0-10.0 sec 0.00 Bytes 0.00 bits/sec 0.034 ms 0/ 0 (nan%)
[ 3] 10.0-11.0 sec 0.00 Bytes 0.00 bits/sec 0.034 ms 0/ 0 (nan%)
[ 3] 11.0-12.0 sec 0.00 Bytes 0.00 bits/sec 0.034 ms 0/ 0 (nan%)
[ 3] 12.0-13.0 sec 0.00 Bytes 0.00 bits/sec 0.034 ms 0/ 0 (nan%)
[ 3] 13.0-14.0 sec 0.00 Bytes 0.00 bits/sec 0.034 ms 0/ 0 (nan%)
[ 3] 14.0-15.0 sec 81.8 KBytes 670 Kbits/sec 24.506 ms 60/ 117 (51%)

[ 1019.644302] usb 1-1: mwifiex_cmd_timeout_func: Timeout cmd id
(1410955919.418091) = 0x6, act = 0x3
[ 1019.653826] usb 1-1: num_data_h2c_failure = 0
[ 1019.658409] usb 1-1: num_cmd_h2c_failure = 0
[ 1019.662907] usb 1-1: num_cmd_timeout = 1
[ 1019.667072] usb 1-1: num_tx_timeout = 0
[ 1019.671117] usb 1-1: last_cmd_index = 4
[ 1019.675186] usb 1-1: last_cmd_id: 28 00 28 00 28 00 5b 00 06 00
[ 1019.681426] usb 1-1: last_cmd_act: 13 01 13 01 13 01 01 00 03 00
[ 1019.687778] usb 1-1: last_cmd_resp_index = 3
[ 1019.692281] usb 1-1: last_cmd_resp_id: 28 80 28 80 28 80 5b 80 28 80
[ 1019.698997] usb 1-1: last_event_index = 2
[ 1019.703223] usb 1-1: last_event: 17 00 33 00 08 00 33 00 08 00
[ 1019.709389] usb 1-1: data_sent=0 cmd_sent=0
[ 1019.713820] usb 1-1: ps_mode=1 ps_state=0
[ 1019.718211] usb 1-1: cmd timeout

Post by Amitkumar Karwar
Could you please try changing this number(MWIFIEX_TX_DATA_URB/MWIFIEX_RX_DATA_URB macros) to 3?

I tried but it doesn't help (also sometimes I see odd driver crashes -
NULL pointer access)

Post by Amitkumar Karwar
Regards,
Amitkumar Karwar

Best Regards,

marek
--
as simple and primitive as possible
-------------------------------------------------
Marek Belisko - OPEN-NANDRA
Freelance Developer

Ruska Nova Ves 219 | Presov, 08005 Slovak Republic
Tel: +421 915 052 184
skype: marekwhite
twitter: #opennandra
web: http://open-nandra.com
--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
the body of a message to majordomo-***@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

James Cameron

2014-09-17 21:57:19 UTC

Post by Amitkumar Karwar
Hi BR,
=20

Post by Belisko Marek
Dear Amitkumar Karwar,
=20
some additional info.
=20

com>

I'm using 3.9 mainline mwifiex driver for wireless usb card. Doi=

some throughput testing (with iperf) in 5GHz I got following
[ 221.521799] usb 1-1: mwifiex_usb_submit_rx_urb: dev_alloc_skb failed

This is skb allocation failure returned by kernel. 4k buffer is

always allocated for Rx packets. This issue doesn't seem to be spec=

ific

Post by Belisko Marek
to 5Ghz.
Yes you're right. I can reproduce issue also with 2.4GHz (doing ipe=

Post by Belisko Marek
testing as mentioned in other email) by pinging device with card.

I checked which which size fails to allocate and it's 4096 bytes=

=2E I

was looking to changes in never kernel releases but I cannot fin=

anything obvious. When connected to 2.4GHz I cannot reproduce is=

sue

though. I'm using FW version mwifiex 1.0 (14.68.29.p26).

Could you please provide the platform details?
How often the problem occurs during throughput testing? Are there=

any

Post by Belisko Marek
specific steps?
One more observation is that when problem occurred complete system =

Post by Belisko Marek
unresponsive (console is almost completely dead).

=20
Thanks for the more information.
Skb alloc failure should be gracefully handled. We will look into this issue.

If you get time, I'd also appreciate a look into the issue on sdio.c
during data receive.

When dev_alloc_skb fails the interrupt handler does not rewind
the driver state in preparation for a retry. This is not graceful.

http://dev.laptop.org/ticket/12694 has details, and an adequate
solution we are using in 3.5 to rewind the driver state:

http://dev.laptop.org/git/olpc-kernel/commit/?h=3Darm-3.5&id=3D59fcaf10=
cce5bbdc370ec1c262b12aeb66ed1dca

We're using 8787.

Post by Amitkumar Karwar
=20

Post by Belisko Marek
I can workaround issue by decreasing iperf bandwidth to ~40m. I thi=

Post by Belisko Marek
in this situation we're running out of memory by exhaustive skb
allocations.

=20
Actually 6 4K size buffers are being allocated for Rx and Tx data dur=

ing traffic.

Post by Amitkumar Karwar
Probably your platform runs out of memory after these allocations.
=20
Could you please try changing this number(MWIFIEX_TX_DATA_URB/MWIFIEX=

_RX_DATA_URB macros) to 3?

Post by Amitkumar Karwar
=20
Regards,
Amitkumar Karwar
N?????r??y????b?X??=C7=A7v?^?)=DE=BA{.n?+????{??*=DE=95?,?{ay?=1D=CA=87=

=DA=99?,j=07??f???h???z?=1E?w???=0C???j:+v???w?j?m????=07????zZ+?????=DD=
=A2j"??!

--=20
James Cameron
http://quozl.linux.org.au/
--
To unsubscribe from this list: send the line "unsubscribe linux-wireles=
s" in
the body of a message to majordomo-***@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Belisko Marek

2014-10-09 08:31:53 UTC

Dear Amitkumar Karwar,

Post by Belisko Marek
Dear Amitkumar Karwar,
some additional info.

This is skb allocation failure returned by kernel. 4k buffer is

Could you please provide the platform details?
How often the problem occurs during throughput testing? Are there any

specific steps?
One more observation is that when problem occurred complete system is
unresponsive (console is almost completely dead).

Thanks for the more information.
Skb alloc failure should be gracefully handled. We will look into this issue.

I did small investigation (will my limited networking knowledge :))
and to avoid usb issue
I did small hack to free received packet in skb (with specific size
1574 which sends iperf) before sending for
processing to driver workqueue. With this small hack I can run iperf
-b100m on client size without any
allocation issue. Bit more testing shows that 11n rx reordering is in
place when receiving packets send from iperf.
Is there any know issue in this area which could lead to issues
described in my report?

BTW I tested backports driver from 3.17-rc3 and issue is still present.

Post by Belisko Marek
I can workaround issue by decreasing iperf bandwidth to ~40m. I think
in this situation we're running out of memory by exhaustive skb
allocations.

Amitkumar Karwar

2014-10-09 09:30:04 UTC

Hi Marek,

Sorry for late reply. We have tried to simulate dev_alloc_skb() failure on our reference platform after 100th, 200th, 300th etc. packets. Our observation is when the failure occurs, corresponding URB won't get submitted, but traffic continues. Traffic stops when the failure count reaches 6 (mwifiex Rx URB count). We don't see any crash or system unresponsiveness.

That's good :) Actually kernel will take care of freeing skb when driver submits received packet using netif_rx(). Could you please share your hack?

Regards,
Amit
��{.n�+��+%��lzwm��b�맲��r��zX��"��^�ȧ��ܨ}��Ơz�&j:+v��n�r��6;靫3��\
nnX��f�z��

Belisko Marek

2014-10-09 11:57:41 UTC

Dear Amitkumar Karwar,

Post by Amitkumar Karwar
Hi Marek,
Sorry for late reply. We have tried to simulate dev_alloc_skb() failure on our reference platform after 100th, 200th, 300th etc. packets. Our observation is when the failure occurs, corresponding URB won't get submitted, but traffic continues. Traffic stops when the failure count reaches 6 (mwifiex Rx URB count). We don't see any crash or system unresponsiveness.

No I don't see such behavior. In my case usb still sending packets
over URB (also proven by below hack). My platform is am335x (same chip
as on beaglebone).

That's good :) Actually kernel will take care of freeing skb when driver submits received packet using netif_rx(). Could you please share your hack?

Yes it should be freed when netif_rx() is processed but I have feeling
that sometimes (my case) during high load (-b 100m) packets are not
send to kernel (not free skb)
until skb allocation fails. I need to better understand 11n reordering code :).

Hack (it's dirty I know but packets of that size are send only during
iperf session):

diff --git a/drivers/net/wireless/mwifiex/usb.c
b/drivers/net/wireless/mwifiex/usb.c
index f90fe21..cc548a1 100644
--- a/drivers/net/wireless/mwifiex/usb.c
+++ b/drivers/net/wireless/mwifiex/usb.c
@@ -178,6 +178,12 @@ static void mwifiex_usb_rx_complete(struct urb *urb)
dev_dbg(adapter->dev, "info: recv_length=%d, status=%d\n",
recv_length, status);
if (status == -EINPROGRESS) {
+
+ if (skb->len >= 1574) {
+ dev_kfree_skb_any(skb);
+ goto setup_for_next;
+ }
+
queue_work(adapter->workqueue, &adapter->main_work);

/* urb for data_ep is re-submitted now;

Post by Amitkumar Karwar
Regards,
Amit

Amitkumar Karwar

2014-10-13 13:41:59 UTC

Hi Marek,

Post by Amitkumar Karwar
That's good :) Actually kernel will take care of freeing skb when

driver submits received packet using netif_rx(). Could you please share
your hack?
Yes it should be freed when netif_rx() is processed but I have feeling
that sometimes (my case) during high load (-b 100m) packets are not send
to kernel (not free skb) until skb allocation fails. I need to better
understand 11n reordering code :).

Rx reorder logic can buffer upto 32 packets, if they aren't received in correct order. You are right. Your platform may run out of memory in this case.
Could you please check if the problem fixes with any of the attached changes?
1) Change Rx window size from 32 to 8.
2) Disable AMPDU feature.

Post by Amitkumar Karwar
Hack (it's dirty I know but packets of that size are send only during
diff --git a/drivers/net/wireless/mwifiex/usb.c
b/drivers/net/wireless/mwifiex/usb.c
index f90fe21..cc548a1 100644
--- a/drivers/net/wireless/mwifiex/usb.c
+++ b/drivers/net/wireless/mwifiex/usb.c
@@ -178,6 +178,12 @@ static void mwifiex_usb_rx_complete(struct urb *urb)
dev_dbg(adapter->dev, "info: recv_length=%d,
status=%d\n",
recv_length, status);
if (status == -EINPROGRESS) {
+
+ if (skb->len >= 1574) {
+ dev_kfree_skb_any(skb);
+ goto setup_for_next;
+ }
+

The change doesn't look correct. The skb is being processed/used. We cannot free it here. This may create problem.

Regards,
Amit

Belisko Marek

2014-10-14 06:37:45 UTC

Dear Amitkumar Karwar,

Post by Amitkumar Karwar
Hi Marek,

Post by Amitkumar Karwar
That's good :) Actually kernel will take care of freeing skb when

Thanks for patches.
I tried both (slightly modified as we're in 3.9 kernel) but issue is
still reproducible. My patch against 3.9 sources:
diff --git a/drivers/net/wireless/mwifiex/decl.h
b/drivers/net/wireless/mwifiex/decl.h
index e8a569a..916924c 100644
--- a/drivers/net/wireless/mwifiex/decl.h
+++ b/drivers/net/wireless/mwifiex/decl.h
@@ -42,7 +42,7 @@
#define MWIFIEX_MAX_RX_BASTREAM_SUPPORTED 16

#define MWIFIEX_AMPDU_DEF_TXWINSIZE 32
-#define MWIFIEX_AMPDU_DEF_RXWINSIZE 16
+#define MWIFIEX_AMPDU_DEF_RXWINSIZE 8
#define MWIFIEX_DEFAULT_BLOCK_ACK_TIMEOUT 0xffff

#define MWIFIEX_RATE_BITMAP_MCS0 32
diff --git a/drivers/net/wireless/mwifiex/sta_event.c
b/drivers/net/wireless/mwifiex/sta_event.c
index 41aafc7..05a36f0 100644
--- a/drivers/net/wireless/mwifiex/sta_event.c
+++ b/drivers/net/wireless/mwifiex/sta_event.c
@@ -378,10 +378,7 @@ int mwifiex_process_sta_event(struct mwifiex_private *priv)
HostCmd_ACT_GEN_GET, 0, NULL);
break;
case EVENT_ADDBA:
- dev_dbg(adapter->dev, "event: ADDBA Request\n");
- mwifiex_send_cmd_async(priv, HostCmd_CMD_11N_ADDBA_RSP,
- HostCmd_ACT_GEN_SET, 0,
- adapter->event_body);
+ dev_dbg(adapter->dev, "event: ADDBA Request ignoring\n");
break;
case EVENT_DELBA:
dev_dbg(adapter->dev, "event: DELBA Request\n");
diff --git a/drivers/net/wireless/mwifiex/wmm.c
b/drivers/net/wireless/mwifiex/wmm.c
index 2670676..3d0aa02 100644
--- a/drivers/net/wireless/mwifiex/wmm.c
+++ b/drivers/net/wireless/mwifiex/wmm.c
@@ -415,8 +415,8 @@ mwifiex_wmm_init(struct mwifiex_adapter *adapter)

for (i = 0; i < MAX_NUM_TID; ++i) {
priv->aggr_prio_tbl[i].amsdu = tos_to_tid_inv[i];
- priv->aggr_prio_tbl[i].ampdu_ap = tos_to_tid_inv[i];
- priv->aggr_prio_tbl[i].ampdu_user = tos_to_tid_inv[i];
+ priv->aggr_prio_tbl[i].ampdu_ap = BA_STREAM_NOT_ALLOWED;
+ priv->aggr_prio_tbl[i].ampdu_user =
BA_STREAM_NOT_ALLOWED;
}

priv->aggr_prio_tbl[6].amsdu

One thing which is not yet still clear to me why kernel console is
completely unresponsive when
receiving packets in high rates. When use iperf (on client) with -b40m
it is OK but when increase to -b100m then console
is completely unresponsive until iperf finish. Any other ideas what to
change to check? Thanks.

The change doesn't look correct. The skb is being processed/used. We cannot free it here. This may create problem.
Regards,
Amit

Amitkumar Karwar

2014-10-14 07:08:00 UTC

Hi Marek,

Post by Belisko Marek
I tried both (slightly modified as we're in 3.9 kernel) but issue is

Thanks a lot for the tests.

Post by Belisko Marek
One thing which is not yet still clear to me why kernel console is
completely unresponsive when receiving packets in high rates. When use
iperf (on client) with -b40m it is OK but when increase to -b100m then
console is completely unresponsive until iperf finish.

Does the system recover when "-b100M" iperf is finished? Can we run iperf with "-b40M" later?
Do you see "dev_alloc_skb failed" messages in dmesg when console is unresponsive?

Post by Belisko Marek
Any other ideas
what to change to check? Thanks.

Could you please share dmesg log with dynamic debug enabled (using attached script) captured when the problem occurs?

Best regards,
Amit

Belisko Marek

2014-10-14 08:25:01 UTC

Hi Amitkumar,

Post by Amitkumar Karwar
Hi Marek,

Post by Belisko Marek
I tried both (slightly modified as we're in 3.9 kernel) but issue is

Thanks a lot for the tests.

Does the system recover when "-b100M" iperf is finished? Can we run iperf with "-b40M" later?
Do you see "dev_alloc_skb failed" messages in dmesg when console is unresponsive?

When we get "dev_alloc_skb failed" then interface is dead (cannot ping
...) so no recovery is possible only system reboot.
I don't see dev_alloc_skb failed when console is unresponsive.

Post by Belisko Marek
Any other ideas
what to change to check? Thanks.

Could you please share dmesg log with dynamic debug enabled (using attached script) captured when the problem occurs?

I tried to capture logs but when enable DYNAMIC_DEBUG I cannot
reproduce issue (running test > 30 minutes without allocation
failure).

Post by Amitkumar Karwar
Best regards,
Amit

James Cameron

2014-10-14 10:20:51 UTC

Post by Belisko Marek
Hi Amitkumar,

Post by Amitkumar Karwar
Hi Marek,

Post by Belisko Marek
I tried both (slightly modified as we're in 3.9 kernel) but

Thanks a lot for the tests.

Post by Belisko Marek
One thing which is not yet still clear to me why kernel console
is completely unresponsive when receiving packets in high
rates. When use iperf (on client) with -b40m it is OK but when
increase to -b100m then console is completely unresponsive until
iperf finish.

Does the system recover when "-b100M" iperf is finished? Can we
run iperf with "-b40M" later? Do you see "dev_alloc_skb failed"
messages in dmesg when console is unresponsive?

When we get "dev_alloc_skb failed" then interface is dead (cannot
ping ...) so no recovery is possible only system reboot.

This symptom was familiar to me, but on sdio.c, which is very
different code.

I've had a brief look at usb.c and offer the following comments:

- a list of six data endpoint urb is allocated in mwifiex_usb_rx_init,
because MWIFIEX_RX_DATA_URB is 6,

- when data endpoint urb is submitted, a new skb is allocated, in
mwifiex_usb_submit_rx_urb, and this is the only source of
"dev_alloc_skb failed" message,

- in normal situation, when data endpoint urb is complete, skb is
either freed or handed up to mwifiex_usb_recv, and the urb is
resubmitted, which causes a new skb to be allocated.

- if "dev_alloc_skb failed" message appears, one data endpoint urb has
been lost and is not re-used,

- if six "dev_alloc_skb failed" messages appear, the interface should
be dead for data receive only.

Amitkumar mentioned this on 9th October; "corresponding URB won't get
submitted". I think this should be fixed; dev_alloc_skb should be
harmless failure, please retry.

I don't see why interface is dead with only one "dev_alloc_skb
failed" message.

Post by Belisko Marek
I don't see dev_alloc_skb failed when console is unresponsive.

Post by Belisko Marek
Any other ideas
what to change to check? Thanks.

Could you please share dmesg log with dynamic debug enabled (using
attached script) captured when the problem occurs?

I tried to capture logs but when enable DYNAMIC_DEBUG I cannot
reproduce issue (running test > 30 minutes without allocation
failure).

Yes, I've seen similar; turn on debugging, and timing critical bug
goes away.

Serial console? If so, try turning it off, and logging to dmesg
buffer only.

--
James Cameron
http://quozl.linux.org.au/
--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
the body of a message to majordomo-***@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Belisko Marek

2014-10-14 10:48:07 UTC

Dear James Cameron,

Post by Belisko Marek
Hi Amitkumar,

Post by Amitkumar Karwar
Hi Marek,

Post by Belisko Marek
I tried both (slightly modified as we're in 3.9 kernel) but

Thanks a lot for the tests.

Post by Belisko Marek
One thing which is not yet still clear to me why kernel console
is completely unresponsive when receiving packets in high
rates. When use iperf (on client) with -b40m it is OK but when
increase to -b100m then console is completely unresponsive until
iperf finish.

Does the system recover when "-b100M" iperf is finished? Can we
run iperf with "-b40M" later? Do you see "dev_alloc_skb failed"
messages in dmesg when console is unresponsive?

When we get "dev_alloc_skb failed" then interface is dead (cannot
ping ...) so no recovery is possible only system reboot.

This symptom was familiar to me, but on sdio.c, which is very
different code.
- a list of six data endpoint urb is allocated in mwifiex_usb_rx_init,
because MWIFIEX_RX_DATA_URB is 6,
- when data endpoint urb is submitted, a new skb is allocated, in
mwifiex_usb_submit_rx_urb, and this is the only source of
"dev_alloc_skb failed" message,
- in normal situation, when data endpoint urb is complete, skb is
either freed or handed up to mwifiex_usb_recv, and the urb is
resubmitted, which causes a new skb to be allocated.
- if "dev_alloc_skb failed" message appears, one data endpoint urb has
been lost and is not re-used,
- if six "dev_alloc_skb failed" messages appear, the interface should
be dead for data receive only.
Amitkumar mentioned this on 9th October; "corresponding URB won't get
submitted". I think this should be fixed; dev_alloc_skb should be
harmless failure, please retry.
I don't see why interface is dead with only one "dev_alloc_skb
failed" message.

Maybe my description wasn't not correct. I see 6 "dev_alloc_skb
failed" messages and then interface is dead
as you wrote.

Post by Belisko Marek
I don't see dev_alloc_skb failed when console is unresponsive.

Post by Belisko Marek
Any other ideas
what to change to check? Thanks.

Could you please share dmesg log with dynamic debug enabled (using
attached script) captured when the problem occurs?

I tried to capture logs but when enable DYNAMIC_DEBUG I cannot
reproduce issue (running test > 30 minutes without allocation
failure).

Yes, I've seen similar; turn on debugging, and timing critical bug
goes away.
Serial console? If so, try turning it off, and logging to dmesg
buffer only.

When turning off serial console how then I get kernel messages from
dmesg buffer?

Post by James Cameron
--
James Cameron
http://quozl.linux.org.au/

James Cameron

2014-10-14 20:44:35 UTC

Post by Belisko Marek
Dear James Cameron,

Post by Belisko Marek
Hi Amitkumar,

Post by Amitkumar Karwar
Hi Marek,

Post by Belisko Marek
I tried both (slightly modified as we're in 3.9 kernel) but

Thanks a lot for the tests.

Post by Belisko Marek
One thing which is not yet still clear to me why kernel console
is completely unresponsive when receiving packets in high
rates. When use iperf (on client) with -b40m it is OK but when
increase to -b100m then console is completely unresponsive until
iperf finish.

Does the system recover when "-b100M" iperf is finished? Can we
run iperf with "-b40M" later? Do you see "dev_alloc_skb failed"
messages in dmesg when console is unresponsive?

When we get "dev_alloc_skb failed" then interface is dead (cannot
ping ...) so no recovery is possible only system reboot.

Maybe my description wasn't not correct. I see 6 "dev_alloc_skb
failed" messages and then interface is dead
as you wrote.

Thanks. That is what I expected.

Post by Belisko Marek
I don't see dev_alloc_skb failed when console is unresponsive.

Post by Belisko Marek
Any other ideas
what to change to check? Thanks.

Could you please share dmesg log with dynamic debug enabled (using
attached script) captured when the problem occurs?

I tried to capture logs but when enable DYNAMIC_DEBUG I cannot
reproduce issue (running test > 30 minutes without allocation
failure).

Yes, I've seen similar; turn on debugging, and timing critical bug
goes away.
Serial console? If so, try turning it off, and logging to dmesg
buffer only.

When turning off serial console how then I get kernel messages from
dmesg buffer?

I use three techniques depending on how usable the system is afterwards:

0. capture kernel messages to disk, using rsyslog or other user space
assistance; often there is enough information saved before the system
is unresponsive,

1. type dmesg at a console shell prompt,

2. watchdog trigger system reset, and in firmware locate the dmesg
buffer structures in memory, and dump them to serial.

http://lists.laptop.org/pipermail/devel/2012-January/034253.html

Belisko Marek

2014-10-22 13:36:25 UTC

Below reply I post by accident in html which was rejected by ML.
Re-posting again. Sorry about that.

Post by Belisko Marek
Hi Amitkumar,

Post by Amitkumar Karwar
Hi Marek,

Post by Belisko Marek
I tried both (slightly modified as we're in 3.9 kernel) but issue is

Thanks a lot for the tests.

Does the system recover when "-b100M" iperf is finished? Can we run iperf with "-b40M" later?
Do you see "dev_alloc_skb failed" messages in dmesg when console is unresponsive?

When we get "dev_alloc_skb failed" then interface is dead (cannot ping
...) so no recovery is possible only system reboot.
I don't see dev_alloc_skb failed when console is unresponsive.

Post by Belisko Marek
Any other ideas
what to change to check? Thanks.

Could you please share dmesg log with dynamic debug enabled (using attached script) captured when the problem occurs?

I tried to capture logs but when enable DYNAMIC_DEBUG I cannot
reproduce issue (running test > 30 minutes without allocation
failure).

Any update on this? Should I provide some other logs?

Post by Amitkumar Karwar
Best regards,
Amit

Thanks,

marek
--
as simple and primitive as possible
-------------------------------------------------
Marek Belisko - OPEN-NANDRA
Freelance Developer

Ruska Nova Ves 219 | Presov, 08005 Slovak Republic
Tel: +421 915 052 184
skype: marekwhite
twitter: #opennandra
web: http://open-nandra.com
--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
the body of a message to majordomo-***@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Amitkumar Karwar

2014-10-23 12:40:54 UTC

Hi Marek,