Posted by: manilageek | September 14, 2011

Netscreen Firewall High CPU Utilization issue:

(M)-> get perf cpu detail

Average System Utilization:  2%
Last 60 seconds from 09/07/2011 08:56:11:
59:  2    58:  2    57:  2    56:  2    55:  2    54:  2
53:  2    52:  2    51:  2    50:  2    49:  2    48:  2
Last 60 minutes:
59:  2    58:  3    57:  2    56:  2    55:  2    54:  2
53:  3    52:  1    51:  2    50:  2    49:  2    48:  3
Last 24 hours:
23:  2    22:  2    21:  2    20:  2    19:  2    18:  2
17:  2    16:  2    15:  2    14:  2    13:  2    12:  2

(M)->get perf cpu all detail

Average System Utilization: 55% (61  5)
Last 60 seconds:
59: 86(96  2)*** 58: 85(95  0)**  57: 86(96  2)*** 56: 85(95  0)**
55: 85(95  2)**  54: 86(96  0)*** 53: 86(96  2)*** 52: 86(96  0)***
Last 60 minutes:
59: 85(95  1)**  58: 85(95 24)**  57: 84(94  1)**  56: 84(94  1)**
55: 84(94  1)**  54: 84(94  1)**  53: 83(93  1)**  52: 83(93  1)**
Last 24 hours:
23: 44(48 10)    22: 66(74  1)*   21: N/A          20: N/A     
19: N/A          18: N/A          17: N/A          16: N/A     

Note:

  • A single asterisk  *  indicates the CPU is nearing a warning threshold.   It is marked when utilization is  ≥ 50%  &  ≤ 70%.
  • Double asterisks  **  indicates to the administrator that CPU is nearing a high level; the administrator should investigate the cause of why CPU is nearing this level.  It is marked when utilization ≥ 70% & ≤ 85%
  • Triple asterisks  ***  indicates the CPU utilization is high; the administrator should investigate the cause of why CPU is high.  It is marked when utilization is ≥ 85%.

Investigate what could be causing the High CPU:

(M)->get task

ID

Task Name

State 

Stack

Heap/
Used   

Schedule

Run Time

Lock Latency

1

100ms timer

IDLE (Suspend) 

8fffff7c/02fc0

30/ 0

801

0.202,

0.000

2

1s timer 

IDLE (Suspend)

8fffff7c/02fc0

30/ 0

201

0.714,

0.000

3

10s timer

IDLE (Suspend)

8fffff7c/02fc0

30/ 0

20

0.004,

0.000

4

1s stimer 

BLOCK (Semaphore)

8fffff68/02fc0

30/ 0

259

0.382,

0.000

Issue the ‘get task’ command twice and take the difference between the run time for each ‘get task’ command to determine the delta Run Time. The task with the largest delta run time indicates the task that is occupying the most CPU cycles.  In the example below,  the task “av worker” has the greatest delta Run Time.

(M)->set fprofile packet enable
(M)->set fprofile packet start

By default, the profiling buffer is set to nowrap (unset fprofile packet wrap), so the packet profiling will auto stop when profiling buffer is full.

If the fprofile is set to wrap, press ESC or set fprofile packet stop (to stop packet profiling)

Display the output:
(M)->get fprofile packet
(M)->get fprofile packet ip
(M)->get fprofile packet none-ip
(M)->get fprofile packet ip proto

 

(M)-> get fprofile packet

packet buffer size(in kilo-packets): 64
total ip packet: 19089
total ip packet time(us): 1937221
total none-ip packet: 3386
total none-ip packet time(us): 119447
     Id  Type        Protocol    Source            Destination             Sport       Dport        Time  Percentage
      1  ip          0x11        10.234.150.1      10.234.150.63          49584         514      394811  19.19%
      2  ip          0x11        10.190.0.112      10.234.150.63            514         514       36852   1.79%

 (M)-> get fprofile packet ip

total entries: 3873
total time(usec): 1937221
     Id  Protocol    Source            Destination            Sport       Dport        Time  Percentage
      1  0x11        10.234.150.1      10.234.150.63          49584         514      394811  20.38%
      2  0x11        10.190.0.112      10.234.150.63            514         514       36852   1.90%

(M)-> get fprofile packet none-ip

total entries: 300
total time(usec): 119447
     Id  Protocol    Source                      Destination                       Time  Percentage
      1  0x0032      d0:d0:fd:d8:82:8a           00:0c:cc:cc:cd                   34803  29.13%
      2  0x8133      00:10:db:bc:ce:23           10:db:f0:f0:f0                   25664  21.48%
 (M)-> get fprofile packet ip proto
total entries: 6
total time(usec): 1937221
     Id  Protocol            Time  Percentage
      1  0x11              988848  51.04%
      2  0x06              812335  41.93%
 

Capture debugs:
Capture flow and tag debugs until the dbuffer fills up to 4Mb (normally it takes only a few seconds under heavy traffic to fill up the buffer).
Note:  Both the ‘debug tag info’ and ‘debug flow basic’ debug (run together) are most beneficial for analysis.

set db size 4096         ##set debug buffer to 4 meg
debug tag info           ##enter this if ISG or NS5000 device
debug flow basic      ##use with CAUTION; may cause higher CPU, so run only a few seconds during the high CPU
clear db                      ##clear debug buffer
                                    <wait a few seconds for buffer to fill up>
undebug all                ##to stop all debugs
get db stream > tftp   or get db stream     ##to view the debug output
unset db size                           ##to return the debug buffer to default size

Also Perform the following:

Session Table –  Check session table information to see the total number of sustained sessions and whether there are any session allocation failures. 

(M)-> get session info

 

Attacks – Check if the network is under any kind of attack or if there are a high number of packets getting processed by the screen options

(M)-> get counter screen zone
(M)-> get alarm event
(M)-> get log event

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Categories

%d bloggers like this: