New Installtion on 5Big - Trouble adding disks

Re: New Installtion on 5Big - Trouble adding disks

Postby mattenwright » Sat Oct 24, 2020 8:56 pm

Okay, so i finally managed to get things working correctly... I had an extremely small short that was not always showing up, which was causing at least part of the error.
So, I now have a serial port working correctly on the NAS with a nice little headphone jack. :-)

Here is my output confirming the "irq 52, no one cared" section:

Code: Select all
[   18.871519] mv643xx_eth_port mv643xx_eth_port.0 eth0: link up, 1000 Mb/s, full duplex, flow control disabled
[   18.894095] console [netcon0] enabled
[   18.897793] netconsole: network logging started
[   18.902440] fvdw gpiolib offset 8 =gpio nr
[   18.906579] fvdw gpio_keys button->gpio = 8 irq = 40
[   18.935827] random: nonblocking pool is initialized
[   19.387246] irq 52: nobody cared (try booting with the "irqpoll" option)


I grabbed the entire boot process if you need something different.

Thanks!
mattenwright
Donator VIP
Donator VIP
 
Posts: 26
Joined: Thu Aug 20, 2020 4:01 pm

Re: New Installtion on 5Big - Trouble adding disks

Postby fvdw » Sat Oct 24, 2020 9:10 pm

ok thank you

In the meantime I did some tests with Uelpenich. In principle we have now an alternative own written gpio_keys driver that could be used on 5big1 in combination with kernel 3.14.2

However I still want to find out why the original gpio_keys driver fails. We are using currently a kernel in which the driver is compiled as module. That makes it easier to test different driver setups.
Attached is kernel 26 and a test module. This is not the alternative driver but ta modified original driver. It disables irq sharing for buttons defined in the driver.
In the kernel several pacthes are made to solve issue with initializing pci bus.

You can load the kernel using fvdw-sl console using option load and start external kernel. You can also write it to sda6 and then it will be loaded at every boot, but if kernel hangs you will need to use standalone kernel to rewrite old kernel to sda6. Therefor I suggest to use the fvdw-sl console option. Using this option implies that you if you reboot the nas and want again use this new kernel you have again to load it via fvdw-sl console. If everything works ok we will write it to sda6.

After the kernel has been loaded and started. Connect via ssh (use putty client) and load the module gpio_keys.ko
You have to place this module in the folder /lib/modules/3.14.2. Do not use the serial console to load the module as the system is running in a chrooted environment, use ssh client.
To load the module give these commands
Code: Select all
depmod
modprobe gpio-keys


now watch what happens on the serial console and post the output.
You do not have the required permissions to view the files attached to this post.
fvdw
Site Admin - expert
 
Posts: 13239
Joined: Tue Apr 12, 2011 2:30 pm
Location: Netherlands

Re: New Installtion on 5Big - Trouble adding disks

Postby mattenwright » Sat Oct 24, 2020 10:16 pm

The kernel started up perfectly (I used the uboot console and loaded that way to keep things temporary).
Below is the output after the gpiokeys module loaded using modprobe:

Code: Select all

[  546.731304] fvdw gpiolib offset 4 =gpio nr
[  546.735453] fvdw gpio-keys setup keys do not enable IRQ line sharing for irq 36
[  547.646388] irq 52: nobody cared (try booting with the "irqpoll" option)
[  547.653094] CPU: 0 PID: 2059 Comm: modprobe Not tainted 3.14.2 #26
[  547.659259] Backtrace:
[  547.661773] [<c0010cb0>] (dump_backtrace) from [<c0010f74>] (show_stack+0x18/0x1c)
[  547.669341]  r6:00000034 r5:c03d9abc r4:c03d9af0 r3:00400100
[  547.675204] [<c0010f5c>] (show_stack) from [<c02d4f64>] (dump_stack+0x20/0x28)
[  547.682438] [<c02d4f44>] (dump_stack) from [<c0047040>] (__report_bad_irq.isra.7+0x2c/0xb4)
[  547.690789] [<c0047014>] (__report_bad_irq.isra.7) from [<c00472cc>] (note_interrupt+0x198/0x244)
[  547.699658]  r5:c03d9abc r4:c03d9abc
[  547.703347] [<c0047134>] (note_interrupt) from [<c0045748>] (handle_irq_event_percpu+0x184/0x1a4)
[  547.712222]  r10:c03eedab r9:00000000 r8:00000000 r7:00000000 r6:00000034 r5:c03d9abc
[  547.720242]  r4:00000000 r3:00000000
[  547.723916] [<c00455c4>] (handle_irq_event_percpu) from [<c0045798>] (handle_irq_event+0x30/0x40)
[  547.732794]  r10:c03d9370 r9:00000001 r8:00100000 r7:00000034 r6:00100000 r5:00000014
[  547.740839]  r4:c03d9abc
[  547.743435] [<c0045768>] (handle_irq_event) from [<c0047dd8>] (handle_level_irq+0xd8/0xe8)
[  547.751707]  r4:c03d9abc r3:00022008
[  547.755384] [<c0047d00>] (handle_level_irq) from [<c0045028>] (generic_handle_irq+0x28/0x38)
[  547.763815]  r4:00000034 r3:c0047d00
[  547.767514] [<c0045000>] (generic_handle_irq) from [<c0018be4>] (gpio_irq_handler+0xcc/0xd8)
[  547.775958]  r4:c03ef3bc r3:00000008
[  547.779632] [<c0018b18>] (gpio_irq_handler) from [<c0045028>] (generic_handle_irq+0x28/0x38)
[  547.788072]  r9:60000013 r8:00000024 r7:c4393b2c r6:c4393be8 r5:00000008 r4:00000008
[  547.796047] [<c0045000>] (generic_handle_irq) from [<c000e894>] (handle_IRQ+0x68/0x88)
[  547.803970]  r4:c03dec20 r3:00000040
[  547.807630] [<c000e82c>] (handle_IRQ) from [<c0008170>] (asm_do_IRQ+0x10/0x14)
[  547.814877]  r6:fe020200 r5:60000013 r4:c001ea6c r3:c001de2c
[  547.820749] [<c0008160>] (asm_do_IRQ) from [<c0011a74>] (__irq_svc+0x34/0x64)
[  547.827873] Exception stack(0xc4393af8 to 0xc4393b40)
[  547.832922] 3ae0:                                                       00000001 00400100
[  547.841097] 3b00: 0000000a c03ef940 c03d6c48 c4392000 00000002 c03ef920 00000024 60000013
[  547.849271] 3b20: c03d9370 c4393b9c c4393b40 c4393b40 c001de2c c001ea6c 60000013 ffffffff
[  547.857449] [<c001e9e4>] (__do_softirq) from [<c001ee64>] (irq_exit+0x88/0xe8)
[  547.864683]  r10:c03d9370 r9:60000013 r8:00000024 r7:c4393c1c r6:00000000 r5:00000008
[  547.872756]  r4:c4392020
[  547.875361] [<c001eddc>] (irq_exit) from [<c000e898>] (handle_IRQ+0x6c/0x88)
[  547.882421]  r4:c03dec20 r3:00000040
[  547.886106] [<c000e82c>] (handle_IRQ) from [<c0008170>] (asm_do_IRQ+0x10/0x14)
[  547.893334]  r6:fe020200 r5:60000013 r4:c0046840 r3:c0047a44
[  547.899190] [<c0008160>] (asm_do_IRQ) from [<c0011a74>] (__irq_svc+0x34/0x64)
[  547.906317] Exception stack(0xc4393be8 to 0xc4393c30)
[  547.911379] 3be0:                   00000000 00000024 00000000 00000003 c03d933c c40f7380
[  547.919557] 3c00: 00000000 00000000 00000024 60000013 c03d9370 c4393c64 c4393c10 c4393c30
[  547.927723] 3c20: c0047a44 c0046840 60000013 ffffffff
[  547.932788] [<c00464d0>] (__setup_irq) from [<c0046a90>] (request_threaded_irq+0xd0/0x124)
[  547.941042]  r10:00000024 r9:c42abb14 r8:00000003 r7:00000000 r6:c03d933c r5:c40f7380
[  547.949115]  r4:bf3c3c28
[  547.951720] [<c00469c0>] (request_threaded_irq) from [<c0046b58>] (request_any_context_irq+0x74/0x7c)
[  547.960942]  r10:c03d3f1c r9:00000003 r8:c035e721 r7:c42abb14 r6:00000024 r5:bf3c3c28
[  547.969005]  r4:00000003 r3:00000003
[  547.972709] [<c0046ae4>] (request_any_context_irq) from [<bf3c3a78>] (gpio_keys_probe+0x2dc/0x48c [gpio_keys])
[  547.982707]  r8:c03d3f00 r7:c42abb00 r6:c03d3d50 r5:c7097000 r4:c42abb14
[  547.989669] [<bf3c379c>] (gpio_keys_probe [gpio_keys]) from [<c01c2904>] (platform_drv_probe+0x20/0x54)
[  547.999064]  r10:c40f7fa4 r9:00000000 r8:bf3c4004 r7:bf3c3f24 r6:c041b1dc r5:bf3c3f24
[  548.007159]  r4:c03d3d50
[  548.009760] [<c01c28e4>] (platform_drv_probe) from [<c01c134c>] (driver_probe_device+0xa8/0x1e4)
[  548.018544]  r5:c03d3d84 r4:c03d3d50
[  548.022234] [<c01c12a4>] (driver_probe_device) from [<c01c1544>] (__driver_attach+0x70/0x94)
[  548.030685]  r7:c03e61f0 r6:bf3c3f24 r5:c03d3d84 r4:c03d3d50
[  548.036557] [<c01c14d4>] (__driver_attach) from [<c01bfb14>] (bus_for_each_dev+0x74/0x98)
[  548.044741]  r6:c01c14d4 r5:bf3c3f24 r4:00000000 r3:00000001
[  548.050579] [<c01bfaa0>] (bus_for_each_dev) from [<c01c0eb4>] (driver_attach+0x20/0x28)
[  548.058590]  r6:c42ab860 r5:00000000 r4:bf3c3f24
[  548.063350] [<c01c0e94>] (driver_attach) from [<c01c0b30>] (bus_add_driver+0xe8/0x1d4)
[  548.071271] [<c01c0a48>] (bus_add_driver) from [<c01c1984>] (driver_register+0xa4/0xe8)
[  548.079283]  r7:c40f7f80 r6:bf3c3fbc r5:c4392010 r4:bf3c3f24
[  548.085161] [<c01c18e0>] (driver_register) from [<c01c28d0>] (__platform_driver_register+0x50/0x64)
[  548.094205]  r5:c4392010 r4:bf3c6000
[  548.097915] [<c01c2880>] (__platform_driver_register) from [<bf3c6018>] (gpio_keys_init+0x18/0x24 [gpio_keys])
[  548.107923] [<bf3c6000>] (gpio_keys_init [gpio_keys]) from [<c0008598>] (do_one_initcall+0x9c/0x140)
[  548.117050] [<c00084fc>] (do_one_initcall) from [<c005902c>] (load_module+0x1438/0x19b4)
[  548.125129]  r9:00000001 r8:bf3c4004 r7:c40f7f80 r6:bf3c3fbc r5:00000001 r4:c4392008
[  548.133131] [<c0057bf4>] (load_module) from [<c0059674>] (SyS_init_module+0xcc/0xd4)
[  548.140862]  r10:00000000 r9:c4392000 r8:c000dae8 r7:00000080 r6:00002821 r5:000e9d90
[  548.148956]  r4:01a46820
[  548.151557] [<c00595a8>] (SyS_init_module) from [<c000d960>] (ret_fast_syscall+0x0/0x2c)
[  548.159651]  r6:01a14a3c r5:01a46820 r4:01a28dc8
[  548.164407] handlers:
[  548.166699] [<c01fd0a4>] mv_interrupt
[  548.170383] Disabling IRQ #52

mattenwright
Donator VIP
Donator VIP
 
Posts: 26
Joined: Thu Aug 20, 2020 4:01 pm

Re: New Installtion on 5Big - Trouble adding disks

Postby fvdw » Sun Oct 25, 2020 10:49 pm

thx, so sharing is not the cause of the problem, need to think on next step

I will be back tomorrow with a new test kernel
fvdw
Site Admin - expert
 
Posts: 13239
Joined: Tue Apr 12, 2011 2:30 pm
Location: Netherlands

Re: New Installtion on 5Big - Trouble adding disks

Postby fvdw » Mon Oct 26, 2020 9:26 am

attached kernel 27, you can use the same module as you used for kernel 26, this kernel has some print statements in the function _setup_irq which seems to initiate the issue
You do not have the required permissions to view the files attached to this post.
fvdw
Site Admin - expert
 
Posts: 13239
Joined: Tue Apr 12, 2011 2:30 pm
Location: Netherlands

Re: New Installtion on 5Big - Trouble adding disks

Postby mattenwright » Mon Oct 26, 2020 11:58 pm

Below is the output from the serial console, it does appear to be different than the last attempt, though it still fails:

Code: Select all

*** fvdw-sl NAS firmware
    This is fvdw-sl version: fvdw-sl 18.2
    built on: October 18 2019

    LAN IP address: 192.168.2.222 (DHCP)

    Port configuration:

    LAN   -> eth0
php based setup routines finished
web permission on resolv.conf
set device tuning for dms performance
move smbd en nmbd db files away from ram disk to prevent hanging samba server
LED settings
else loop1 finished
rcS finished
------------------------
rcS finished
------------------------
 list boot.log file
start boot log
is a physical machine
detect and set platform
5big1
5big1
UIMAGE-3142-5BIG1-11
Current kernel: 3.14.2 #27 Mon Oct 26 10:23:20 CET 2020
5big1
5big1
 mount dev/pts
update dev nodes
booting using sda1 file system...
else loop1 finished
rcS finished
[  320.206171] fvdw gpiolib offset 4 =gpio nr
[  320.210350] fvdw gpio-keys setup keys do not enable IRQ line sharing for irq 36
[  320.217672] fvdw manage.c 2 irq=36
[  320.221119] fvdw manage.c 3 irq=36
[  320.224549] fvdw manage.c 9 irq=36
[  320.227955] fvdw manage.c 11 irq=36
[  321.110406] irq 52: nobody cared (try booting with the "irqpoll" option)
[  321.117112] CPU: 0 PID: 1995 Comm: modprobe Not tainted 3.14.2 #27
[  321.123276] Backtrace:
[  321.125773] [<c0010cb0>] (dump_backtrace) from [<c0010f74>] (show_stack+0x18/0x1c)
[  321.133343]  r6:00000034 r5:c03d9abc r4:c03d9af0 r3:00400100
[  321.139222] [<c0010f5c>] (show_stack) from [<c02d5104>] (dump_stack+0x20/0x28)...

mattenwright
Donator VIP
Donator VIP
 
Posts: 26
Joined: Thu Aug 20, 2020 4:01 pm

Re: New Installtion on 5Big - Trouble adding disks

Postby fvdw » Tue Oct 27, 2020 9:42 pm

Thank you , yes I add print statement in the kernel code to find location where the error occurs

Attached kernel 29 with some more printk's
and kernel 30 in which the irq handler for irq 52 is set to not shared

(its a painfull process as there are many functions involved in irq handling)

ps I have an alternative driver for gpio-keys that works on 5big1 but still I want to understand why the original linux driver fails on kernel 3.14.2 and not on 2.6.39.4

edit----
removed kernel 29 and 30, no solution instead kernel 31
You do not have the required permissions to view the files attached to this post.
fvdw
Site Admin - expert
 
Posts: 13239
Joined: Tue Apr 12, 2011 2:30 pm
Location: Netherlands

Re: New Installtion on 5Big - Trouble adding disks

Postby fvdw » Wed Oct 28, 2020 10:15 am

I edited my previous post
fvdw
Site Admin - expert
 
Posts: 13239
Joined: Tue Apr 12, 2011 2:30 pm
Location: Netherlands

Re: New Installtion on 5Big - Trouble adding disks

Postby mattenwright » Thu Oct 29, 2020 2:20 am

Here are the results from kernel 31, using the same gpio-keys module:

Code: Select all

*** fvdw-sl NAS firmware
    This is fvdw-sl version: fvdw-sl 18.2
    built on: October 18 2019

    LAN IP address: 192.168.2.222 (DHCP)

    Port configuration:

    LAN   -> eth0
php based setup routines finished
web permission on resolv.conf
set device tuning for dms performance
move smbd en nmbd db files away from ram disk to prevent hanging samba server
LED settings
else loop1 finished
rcS finished
------------------------
rcS finished
------------------------
 list boot.log file
start boot log
is a physical machine
detect and set platform
5big1
5big1
UIMAGE-3142-5BIG1-11
Current kernel: 3.14.2 #31 Wed Oct 28 23:07:55 CET 2020
5big1
5big1
 mount dev/pts
update dev nodes
booting using sda1 file system...
else loop1 finished
rcS finished
[  227.030617] fvdw gpiolib offset 4 =gpio nr
[  227.034799] fvdw gpio-keys setup keys do not enable IRQ line sharing for irq 36
[  227.042142] fvdw manage.c 2 irq=36
[  227.045635] fvdw manage.c 3 irq=36
[  227.049065] fvdw manage.c 9 irq=36
[  227.052469] fvdw manage.c 11 irq=36
[  227.055955] fvdw manage.c 11a irq=36



No worries on the process at all... and now that I have the serial cable matter sorted out, its just a matter of finding a few minutes to test :-) That's a much easier ask.
mattenwright
Donator VIP
Donator VIP
 
Posts: 26
Joined: Thu Aug 20, 2020 4:01 pm

Re: New Installtion on 5Big - Trouble adding disks

Postby fvdw » Thu Oct 29, 2020 8:52 am

Probably the kernel 31 hangs after reporting printk 11a

My final conclusion is as follow. When the linux gpio-keys driver is loaded using a irq handler based on trigger the kernel seems to generate interrupts also on irq 52 of the pci bus for sata controllers events as soon as setup of the new irq for the push button (irq 36) is finished indicate by the function raw_spin_unlock_irqrestore(&desc->lock, flags); just after printk 11a.
The sata-mv irq handler sees these irq but does not recognize the cause and returns not handled, as result the irq disable kicks in after 100000 interrupts on irq 52. Modifying the sata-mv driver as done in kernel 31 to let the handler of irq 52 always report handled the system prevent disabling the irq and sata driver but the kernel will hang, or better be 100% busy with handling interrupts on irq 52 as the constant stream of interrupts on that keeps the kernel busy.
The issue must be something in the assembler code that initiate interrupts. But that is a bridge to far for me to investigate. (too complicated as my knowledge of assembler is zero)
I will finalize the gpio-keys fvdw driver that doesn't use trigger and implement that in the kernel.
fvdw
Site Admin - expert
 
Posts: 13239
Joined: Tue Apr 12, 2011 2:30 pm
Location: Netherlands

PreviousNext

Return to Lacie 5Big Network vs1

Who is online

Users browsing this forum: No registered users and 1 guest

cron