Sharing a case of controller restart caused by a CIFS bug in NetApp Fas series

Customer has one IBM Of N3240, Corresponding to NetApp Namely FAS2240, Controller restart often occurs , If you restart successfully , The customer basically didn't feel , But in some cases , Restart failed , A controller fell , Customer business has obvious perception . The customer's approach is to restart the controller , Is in the loader Next ,Boot_ontap once , The machine started smoothly again . This is repeated over and over again , One day at last , End users are angry , Ask the maintenance provider to analyze the reasons . Here's this case Analysis process sharing .

Check out the event The log will find , This controller restarts almost every day , Even several times a day ,2022 For half a year , It's restarted 40 many times , Of course, some of this is for problem restart , Some systems restart automatically . The following is part of event log.

Record 3180: Sun Jan 16 05:33:45 2022 [SP.critical]: Filer Reboots

Record 3195: Mon Jan 17 15:21:30 2022 [SP.critical]: Filer Reboots

Record 3226: Sat Feb  5 05:54:55 2022 [SP.critical]: Filer Reboots

Record 3241: Mon Feb  7 15:07:24 2022 [SP.critical]: Filer Reboots

Record 3280: Sun Mar  6 03:43:36 2022 [SP.critical]: Filer Reboots

Record 3294: Tue Mar  8 01:52:48 2022 [SP.critical]: Filer Reboots

Record 3307: Tue Mar  8 05:42:47 2022 [SP.critical]: Filer Reboots

Record 3328: Mon Mar 14 15:43:53 2022 [SP.critical]: Filer Reboots

Record 3349: Wed Mar 23 07:33:02 2022 [SP.critical]: Filer Reboots

Record 3362: Wed Mar 23 10:08:18 2022 [SP.critical]: Filer Reboots

Record 3377: Fri Mar 25 05:57:46 2022 [SP.critical]: Filer Reboots

Record 3393: Mon Mar 28 05:42:58 2022 [SP.critical]: Filer Reboots

Record 3411: Sat Apr  2 02:12:31 2022 [SP.critical]: Filer Reboots

Record 3429: Tue Apr  5 15:05:40 2022 [SP.critical]: Filer Reboots

Record 3449: Wed Apr 13 00:53:43 2022 [SP.critical]: Filer Reboots

Record 3476: Wed Apr 27 13:09:58 2022 [SP.critical]: Filer Reboots

Record 3493: Sun May  1 13:18:01 2022 [SP.critical]: Filer Reboots

Record 3524: Thu May 19 01:49:50 2022 [SP.critical]: Filer Reboots

Record 3539: Sat May 21 06:40:10 2022 [SP.critical]: Filer Reboots

Record 3553: Sun May 22 16:17:47 2022 [SP.critical]: Filer Reboots

Record 3568: Tue May 24 13:24:54 2022 [SP.critical]: Filer Reboots

Record 3598: Fri Jun 10 13:26:04 2022 [SP.critical]: Filer Reboots

Record 3615: Wed Jun 15 00:14:00 2022 [SP.critical]: Filer Reboots

Record 3629: Thu Jun 16 11:33:34 2022 [SP.critical]: Filer Reboots

Record 3644: Fri Jun 17 05:47:57 2022 [SP.critical]: Filer Reboots

Record 3657: Fri Jun 17 12:12:42 2022 [SP.critical]: Filer Reboots

Record 3676: Fri Jun 24 00:05:47 2022 [SP.critical]: Filer Reboots

Record 3690: Sat Jun 25 16:26:57 2022 [SP.critical]: Filer Reboots

Record 3705: Mon Jun 27 05:35:27 2022 [SP.critical]: Filer Reboots

Record 3720: Wed Jun 29 10:53:57 2022 [SP.critical]: Filer Reboots

Record 3736: Sat Jul  2 12:43:12 2022 [SP.critical]: Filer Reboots

Record 3750: Mon Jul  4 03:23:30 2022 [SP.critical]: Filer Reboots

Record 3766: Thu Jul  7 10:30:59 2022 [SP.critical]: Filer Reboots

Record 3779: Thu Jul  7 12:00:53 2022 [SP.critical]: Filer Reboots

Record 3794: Fri Jul  8 07:10:19 2022 [SP.critical]: Filer Reboots

Record 3807: Sat Jul  9 01:27:50 2022 [SP.critical]: Filer Reboots

Record 3822: Sun Jul 10 11:46:48 2022 [SP.critical]: Filer Reboots

Record 3836: Tue Jul 12 04:32:41 2022 [SP.critical]: Filer Reboots

Record 3850: Wed Jul 13 22:39:10 2022 [SP.critical]: Filer Reboots

Record 3864: Thu Jul 14 01:28:26 2022 [SP.critical]: Filer Reboots

Record 3877: Thu Jul 14 07:43:41 2022 [SP.critical]: Filer Reboots

Record 3892: Sat Jul 16 12:42:43 2022 [SP.critical]: Filer Reboots

Record 3906: Mon Jul 18 03:35:09 2022 [SP.critical]: Filer Reboots

Record 3919: Mon Jul 18 04:23:21 2022 [SP.critical]: Filer Reboots

Record 3933: Tue Jul 19 04:24:35 2022 [SP.critical]: Filer Reboots

Record 3946: Tue Jul 19 10:15:06 2022 [SP.critical]: Filer Reboots

Then let's look at the log when restarting , Basically, they are all the following panic Information , It's all similar

 

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 06
fault virtual address    = 0x28
fault code        = supervisor read data, page not present
instruction pointer    = 0x8:0xffffffff842d9ef3
stack pointer            = 0x10:0xfffffe0008764bd0
frame pointer            = 0x10:0xfffffe0008764bf8
code segment        = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags    = interrupt enabled, resume, IOPL = 0
current process        = 1553 (ontap: cpu2)
trap number        = 12
PANIC  : page fault (supervisor read, page not present) on VA 0x28 cs:rip = 0x8:0xffffffff842d9ef3 rflags = 0x10246
version: 8.1.2: Tue Oct 30 19:56:51 PDT 2012
conf   : x86_64
cpuid = 2
Uptime: 44m28s

PANIC: page fault (supervisor read, page not present) on VA 0x28 cs:rip = 0x8:0xffffffff842d9ef3 rflags = 0x10246 in SK process Auth09 on release 8.1.2 on Mon Jul 18 12:20:08 CST 2022

version: 8.1.2: Tue Oct 30 19:56:51 PDT 2012
compile flags: x86_64
HA: current time (in sk_msecs) 2648596 (in sk_cycles) 4702800163010
DUMPCORE: START
Dumping to disks: 0a.00.11
Writing panic info to sparecore disk.

This leads to panic The root cause is NetApp ONTAP 8.1 Version of CIFS（SMB 2.0） bug 552397 Lead to .

Here is NetApp For this bug Official explanation ：

If durable handles are enabled and all the following conditions are met, controller disruption might occur: 1. Multiple SMB 2 sessions from a workstation share the same TCP connection; and 2. There are open files on at least one of those sessions; and 3. Network error detected by the workstation triggers session reconnects.

Okay , find panic The problem. , The next step is the solution . In fact, the fundamental solution is to upgrade Ontap operating system , At least to 8.1.4P9 in the future . If you don't want to upgrade , Deal with it like this , But I don't want to panic Occurs every day , Namely disable durable handles Or simply put smb 2.0 perhaps 2.1 all disable fall , Don't use .

Consult bloggers for detailed solutions @ wechat : StorageExpert.