We have been managing FortiGate firewalls for more than a decade and we gathered our own toolset to properly start troubleshooting and fixing the issues that arise with these firewalls. In this blog post, we are going to present the best firewall troubleshooting commands that we use when we start investigating issues that appear with FortiGate firewalls. Hope you too use this as a reference when you start troubleshooting your own FortiGate.
Table of Contents
FortiGate Troubleshooting Commands
Below is the list that we most often use. We tried to group them under small, general chapters to make it easy for you to find them quickly.
General System References
All these commands should be used when you are first checking the total configuration of the firewall.
Show FortiGate Details
This command shows you the current Firewall version, the IPS/Virus/App-DB versions, the serial number, and also the maximum number of VDOMs (in our case, 10). The HA mode and many more.
FG-HIFENCE # get system status
Version: FortiGate-101F v6.4.11,build2030,221031 (GA.M)
Firmware Signature: certified
Virus-DB: 90.09945(2023-01-24 12:19)
Extended DB: 90.09945(2023-01-24 12:19)
IPS-DB: 22.00481(2023-01-21 04:32)
IPS-ETDB: 0.00000(2001-01-01 00:00)
APP-DB: 22.00479(2023-01-18 20:47)
INDUSTRIAL-DB: 6.00741(2015-12-01 02:30)
IPS Malicious URL Database: 4.00601(2023-01-24 12:20)
Serial-Number: FG101FTK21XXXXX
BIOS version: 05000024
System Part-Number: P24605-20
Log hard disk: Available
Hostname: FG-HIFENCE
Private Encryption: Disable
Operation Mode: NAT
Current virtual domain: root
Max number of virtual domains: 10
Virtual domains status: 1 in NAT mode, 0 in TP mode
Virtual domain configuration: disable
FIPS-CC mode: disable
Current HA mode: a-p, primary
Cluster uptime: 28 days, 21 hours, 25 minutes, 27 seconds
Cluster state change time: 2022-12-27 03:30:42
Branch point: 2030
Release Version Information: GA
System time: Wed Jan 25 00:56:08 2023
Check the CPU, Memory, and Load Usage
FG-HIFENCE # get system performance status
CPU states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU0 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU1 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU2 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU3 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU4 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU5 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU6 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU7 states: 0% user 0% system 0% nice 99% idle 0% iowait 0% irq 1% softirq
Memory: 7770644k total, 2420196k used (31.1%), 2995600k free (38.6%), 2354848k freeable (30.3%)
Average network usage: 30971 / 30131 kbps in 1 minute, 30565 / 29635 kbps in 10 minutes, 29574 / 28703 kbps in 30 minutes
Average sessions: 9328 sessions in 1 minute, 8019 sessions in 10 minutes, 8769 sessions in 30 minutes
Average session setup rate: 50 sessions per second in last 1 minute, 44 sessions per second in last 10 minutes, 50 sessions per second in last 30 minutes
Average NPU sessions: 3089 sessions in last 1 minute, 2359 sessions in last 10 minutes, 2340 sessions in last 30 minutes
Average nTurbo sessions: 864 sessions in last 1 minute, 965 sessions in last 10 minutes, 952 sessions in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 28 days, 21 hours, 30 minutes
Check WHO is using CPU, Memory
Tip: You can sort by CPU (press “c“) or by Memory (press “m“)
FG-HIFENCE#diag sys top-all
Run Time: 28 days, 21 hours and 33 minutes
0U, 0N, 0S, 100I, 0WA, 0HI, 0SI, 0ST; 7588T, 2927F
bcm.user 149 S < 3.1 0.2
ipsengine 312 S < 0.1 1.2
urlfilter 321 S < 0.1 0.1
voipd 225 S 0.1 0.1
ipshelper 261 S < 0.0 2.0
ipsengine 315 S < 0.0 1.1
ipsengine 313 S < 0.0 1.1
ipsengine 318 S < 0.0 1.1
ipsengine 317 S < 0.0 1.1
ipsengine 314 S < 0.0 1.1
ipsengine 316 S < 0.0 1.1
cmdbsvr 175 S 0.0 0.6
reportd 224 S 0.0 0.5
miglogd 201 S 0.0 0.5
miglogd 289 S 0.0 0.4
miglogd 288 S 0.0 0.4
miglogd 291 S 0.0 0.4
miglogd 290 S 0.0 0.4
updated 309 S 0.0 0.3
cw_acd 256 S 0.0 0.3
Show Hardware Acceleration
Not sure what hardware acceleration you have on your firewall? You can check it easily with the below 2 commands. More info can be found here
FG-HIFENCE # get system npu
dos-options:
npu-dos-meter-mode : global
npu-dos-tpe-mode : enable
policy-offload-level: disable
switch-np-hash : src-dst-ip
capwap-offload : enable
prp-port-in :
prp-port-out :
FG-HIFENCE # get system np6xlite
== [ np6xlite_0 ]
name: np6xlite_0
Check HA Status
You can check the HA status of your FortiGate cluster by using the following commands:
FG-HIFENCE # get system ha status
HA Health Status:
WARNING: FG101FTKXXXXX has mondev down;
WARNING: FG101FTKXXXXX has mondev down;
Model: FortiGate-101F
Mode: HA A-P
Group: 0
Debug: 0
Cluster Uptime: 28 days 21:45:20
Cluster state change time: 2022-12-27 03:30:42
Primary selected using:
<2022/12/27 03:30:42> FG101FTK21XXXXX is selected as the primary because it has the largest value of override priority.
ses_pickup: disable
override: disable
Configuration Status:
FG101FTKXXXXX(updated 4 seconds ago): in-sync
FG101FTKXXXXX(updated 4 seconds ago): in-sync
System Usage stats:
FG101FTKXXXXX(updated 4 seconds ago):
sessions=9543, average-cpu-user/nice/system/idle=0%/0%/0%/99%, memory=31%
FG101FTKXXXXX(updated 4 seconds ago):
sessions=0, average-cpu-user/nice/system/idle=0%/0%/0%/99%, memory=23%
HBDEV stats:
FG101FTKXXXXX(updated 4 seconds ago):
ha1: physical/1000auto, up, rx-bytes/packets/dropped/errors=7111038425/16920231/0/0, tx=16783907464/39802001/0/0
ha2: physical/1000auto, up, rx-bytes/packets/dropped/errors=6488635311/12570702/0/0, tx=6572639254/12570789/0/0
FG101FTKXXXXX(updated 4 seconds ago):
ha1: physical/1000auto, up, rx-bytes/packets/dropped/errors=16784168354/39801945/0/0, tx=7110212297/16920235/0/0
ha2: physical/1000auto, up, rx-bytes/packets/dropped/errors=6572640832/12570792/0/0, tx=6488636995/12570706/0/0
MONDEV stats:
FG101FTKXXXXX(updated 4 seconds ago):
mgmt: physical/00, down, rx-bytes/packets/dropped/errors=0/0/0/0, tx=0/0/0/0
FG101FTKXXXXX(updated 4 seconds ago):
mgmt: physical/00, down, rx-bytes/packets/dropped/errors=0/0/0/0, tx=0/0/0/0
Primary : FG-HIFENCE , FG101FTK21XXXXX, HA cluster index = 1
Secondary : FortiGate-101F , FG101FTK21YYYYY, HA cluster index = 0
number of vcluster: 1
vcluster 1: work 169.254.0.2
Primary: FG101FTKXXXXX, HA operating index = 0
Secondary: FG101FTKXXXXX, HA operating index = 1
FG-HIFENCE # diagnose sys ha status
HA information
Statistics
traffic.local = s:0 p:533046958 b:102460244156
traffic.total = s:0 p:532813242 b:102437901726
activity.ha_id_changes = 2
activity.fdb = c:0 q:0Model=100, Mode=2 Group=0 Debug=0
nvcluster=1, ses_pickup=0, delay=0
[Debug_Zone HA information]
HA group member information: is_manage_primary=1.
FG101FTKXXXXX: Primary, serialno_prio=1, usr_priority=250, hostname=FG-HIFENCE
FG101FTKXXXXX: Secondary, serialno_prio=0, usr_priority=128, hostname=FortiGate-101F
[Kernel HA information]
vcluster 1, state=work, primary_ip=169.254.0.2, primary_id=0:
FG101FTK21XXXXX: Primary, ha_prio/o_ha_prio=0/0
FG101FTK21YYYYY: Secondary, ha_prio/o_ha_prio=1/1
Check the session table of the firewall:
The values from the session table of the firewall (the max against the used)
FG-HIFENCE # diagnose sys session full-stat
session table:
table_size=1048576 max_depth=3 used=14780
misc info: session_count=6410 setup_rate=28 exp_count=47 clash=361 memory_tension_drop=0 ephemeral=0/501248 removeable=0 npu_session_count=609 nturbo_session_count=471 delete=2412, flush=0, dev_down=54/175 ses_walkers=0 TCP sessions: 11 in NONE state 361 in ESTABLISHED state 3 in SYN_SENT state 13 in TIME_WAIT state 11 in CLOSE state 6 in CLOSE_WAIT state firewall error stat: error1=00000000 error2=00000000 error3=00000000 error4=00000000 tt=00000000 cont=00000137 ids_recv=006993b5 url_recv=00000000 av_recv=007b6ad3 fqdn_count=00000007 fqdn6_count=00000000
Show interface settings
You can use this if you suspect that the interface is broken or if you have drops on that port.
FG-HIFENCE # diagnose hardware deviceinfo nic port1
Description :FortiASIC NP6XLITE
Adapter Driver Name :FortiASIC
NP6XLITE Driver Board :101F lif id :6 lif oid :70 netdev oid :70
Current_HWaddr 00:09:0f:09:00:06
Permanent_HWaddr e8:ed:d6:07:af:96
========== Link Status ==========
Admin :up
netdev status :up
autonego_setting:1
link_setting :1
speed_setting :1000
duplex_setting :0
Speed :1000
Duplex :Full
link_status :Up
============ Counters ===========
Rx Pkts :1034893546
Rx Bytes :505348844494
Tx Pkts :1982810099
Tx Bytes :2134972340708
Host Rx Pkts :64309750
Host Rx Bytes :8814250118
Host Tx Pkts :36406556
Host Tx Bytes :8244211448
Host Tx dropped :0
FragTxCreate :0
FragTxOk :0
FragTxDrop :0
Check IP addresses:
With the below command you can see all the IP addresses assigned to the firewall in one go!
FG-HIFENCE # diag ip address list
IP=1.1.183.205->1.1.183.205/255.255.255.248 index=5 devname=dmz
IP=10.250.1.132->10.250.1.132/255.255.255.0 index=6 devname=mgmt
IP=1.1.1.1->1.1.1.1/255.255.255.255 index=7 devname=wan1
IP=1.1.145.46->1.1.145.46/255.255.255.252 index=8 devname=wan2
IP=10.27.10.1->10.27.10.1/255.255.252.0 index=11 devname=port1
IP=10.200.1.1->10.200.1.1/255.255.255.0 index=12 devname=port2
IP=1.1.12.133->1.1.12.133/255.255.255.0 index=14 devname=port4
IP=1.1.185.70->1.1.185.70/255.255.255.252 index=16 devname=port6
IP=10.239.239.1->10.239.239.1/255.255.255.0 index=18 devname=port8
IP=127.0.0.1->127.0.0.1/255.0.0.0 index=36 devname=root
IP=192.168.0.1->192.168.0.1/255.255.254.0 index=38 devname=LAN Aggregate
IP=10.252.0.1->10.252.0.1/255.255.255.0 index=39 devname=Management VLAN
IP=127.0.0.1->127.0.0.1/255.0.0.0 index=42 devname=vsys_ha
IP=169.254.0.2->169.254.0.2/255.255.255.192 index=43 devname=port_ha
IP=127.0.0.1->127.0.0.1/255.0.0.0 index=44 devname=vsys_fgfm
IP=169.254.0.65->169.254.0.65/255.255.255.192 index=45 devname=havdlink0
IP=169.254.0.66->169.254.0.66/255.255.255.192 index=46 devname=havdlink1
Show ARP entries
FG-HIFENCE # get system arp
Address Age(min) Hardware Addr Interface
192.168.0.191 0 74:da:38:e8:8e:29 LAN Aggregate
192.168.0.148 1 00:1e:5e:04:2d:41 LAN Aggregate
10.252.0.122 1 00:5f:86:8a:e2:42 Management VLAN
10.23.11.2 3 2c:ea:dc:8b:cf:f7 LAN Aggregate
Add single ARP entry:
#diag ip arp add <INTERFACE> <IP.addr> <MAC>
Get all the routing table
FG-HIFENCE # get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default
Routing table for VRF=0
S* 0.0.0.0/0 [1/0] via 1.1.12.1, port4
[1/0] via 1.1.183.201, dmz
[1/0] via 1.1.185.69, port6
C 1.1.12.0/24 is directly connected, port4
C 192.168.0.0/23 is directly connected, LAN Aggregate
C 10.27.8.0/22 is directly connected, port1
S 10.100.1.0/24 [8/0] is directly connected, VPN
S 10.101.1.0/24 [8/0] is directly connected, VPN
S 10.150.1.0/24 [8/0] is directly connected, VPN
S 10.212.134.0/24 [8/0] is directly connected, VPN
C 10.239.239.0/24 is directly connected, port8
C 10.252.0.0/24 is directly connected, Management VLAN
S 10.255.250.0/24 [8/0] is directly connected, VPN
C 1.1.145.44/30 is directly connected, wan2
C 1.1.183.200/29 is directly connected, dmz
C 1.1.185.68/30 is directly connected, port6
FortiGate VPN troubleshoot Guide
Get all VPN tunnel list
FG-HIFENCE # diagnose vpn tunnel list
list all ipsec tunnel in vd 0
------------------------------------------------------
name=VPN_TLK ver=1 serial=2 1.1.183.205:0->1.1.231.68:0 dst_mtu=1500
bound_if=5 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/520 options[0208]=npu frag-rfc run_state=0 role=sync-primary accept_traffic=0 overlay_id=0proxyid_num=1 child_num=0 refcnt=9 ilast=1599110 olast=1599113 ad=/0
stat: rxp=0 txp=0 rxb=0 txb=0
dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=0
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=VPN_TLK proto=0 sa=0 ref=1 serial=3
src: 0:192.168.0.0/255.255.254.0:0 0:10.252.0.0/255.255.255.0:0
dst: 0:10.100.1.0/255.255.255.0:0 0:10.150.1.0/255.255.255.0:0 0:10.101.1.0/255.255.255.0:0 0:10.255.250.0/255.255.255.0:0 0:10.212.134.0/255.255.255.0:0
run_tally=1
------------------------------------------------------
name=VPN ver=1 serial=1 1.1.12.133:0->1.1.11.55:0 dst_mtu=1500
bound_if=14 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/520 options[0208]=npu frag-rfc run_state=0 role=sync-primary accept_traffic=1 overlay_id=0
proxyid_num=1 child_num=0 refcnt=349 ilast=0 olast=0 ad=/0
stat: rxp=379684811 txp=462753648 rxb=139735558746 txb=191841360458
dpd: mode=on-idle on=1 idle=20000ms retry=3 count=0 seqno=3
natt: moN proto=0 sa=1 ref=4389 serial=3src: 0:192.168.0.0/255.255.254.0:0 0:10.252.0.0/255.255.255.0:0
dst: 0:10.100.1.0/255.255.255.0:0 0:10.150.1.0/255.255.255.0:0 0:10.101.1.0/255.255.255.0:0 0:10.255.250.0/255.255.255.0:0 0:10.212.134.0/255.255.255.0:0
SA: ref=6 options=10226 type=00 soft=0 mtu=1438 expire=2722/0B replaywin=2048
seqno=8816eb esn=0 replaywin_lastseq=0072cf61 qat=0 rekey=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=42931/43200
dec: spi=5b11ebb2 esp=aes key=16 93ea2fd11bb67bd28f05ecc49f142e0a
ah=sha1 key=20 bfb8c847d58911f9de50bff4b3da100f4e721526
enc: spi=a1626f84 esp=aes key=16 9d7365efc02556b5dc61c0dec0da9aab
ah=sha1 key=20 3a7cd8052603be480734dcbbc26b38e0e3b65268
dec:pkts/bytes=8497247/3973245342, enc:pkts/bytes=11246971/3264320239
npu_flag=03 npu_rgwy=1.1.11.55 npu_lgwy=1.1.12.133 npu_selid=5 dec_npuid=1 enc_npuid=1run_tally=1de=none draft=0 interval=0 remote_port=0
proxyid=VP
Enable VPN debugging for a specific VPN (useful in case you have more than 1 VPN tunnel)
diagnose debug enable
diagnose debug console timestamp enable
diagnose vpn ike log filter name <VPN-name>
diagnose debug application ike -1
Bring UP the VPN tunnel
diag vpn tunnel up IPSEC_PHASE2 IKE_Phase1
Authentication Debugging
RADIUS
Test if Radius is Live and try to connect with the known shared key
#diagnose test authserver radius-direct <RADIUS_IP> <RADIUS_PORT> <RADIUS_PASSWORD>
If successful, try to login with an username and password
#diagnose test authserver radius <RADIUS_NAME> <protocol-chap|pap|mschap|mschap2> <username> <password>
FSSO
Show total events of FSSO users that were sent towards the Fortigate (this can be filtered on the FSSO Collector Agent sitting on the Domain Controller) # diag debug authd fsso summary
Check all the users that were received by Fortigate
#diag debug enable
#diag debub authd fsso list
Check if the FSSO Server is active and connected:
#diag debug authd fsso server-status
Restart FortiGate daemons
Based on our experience, the most common daemon that you will have to restart due to memory over-utilizations is “ipsmonitor”.
You will mostly use this when the Fortigate is in conserve mode. Most daemons have the code “99” as set to restart.
FG-HIFENCE # diag test application ipsmonitor 99
To see all possible actions, hit an “enter” after the “ipsmonitor” as can be seen below.
FG-HIFENCE # diag test application ipsmonitor
IPS Engine Test Usage:
1: Display IPS engine information
2: Toggle IPS engine enable/disable status
3: Display restart log
4: Clear restart log
5: Toggle bypass status
6: Submit attack characteristics now
10: IPS queue length
11: Clear IPS queue length
12: IPS L7 socket statistics
13: IPS session list
14: IPS NTurbo statistics
15: IPSA statistics
18: Display session info cache
19: Clear session info cache
21: Reload FSA malicious URL database
22: Reload whitelist URL database
24: Display Flow AV statistics
25: Reset Flow AV statistics
32: Reload certificate blacklist database
40: Display packet log statistics
41: Reset packet log statistics
96: Toggle IPS engines watchdog timer
97: Start all IPS engines
98: Stop all IPS engines
99: Restart all IPS engines and monitor
Packet sniffing and flow monitor
Sniffing traffic
If you want to do a packet capture and export it to Wireshark, you will need to use the below command: #diag sniffer packet any <'filter'> 6 0 a We mostly use the “4 0” as it shows us the source and destination interfaces.As can be seen below (source here) # diag sniffer packet vlan206 "host 11.11.11.9" 4 0
interfaces=[vlan206]
filters=[host 11.11.11.9]
0.968800 vlan206 -- 11.11.11.9 -> 11.11.11.1: icmp: echo request
0.968858 vlan206 -- 11.11.11.1 -> 11.11.11.9: icmp: echo reply
1.982626 vlan206 -- 11.11.11.9 -> 11.11.11.1: icmp: echo request
1.982683 vlan206 -- 11.11.11.1 -> 11.11.11.9: icmp: echo reply
Session Flows
#diag debug enable
Example of running the filter with 8.8.8.8 as shown in the bolded example above.
#diag debug flow filter <filter> (eg. diag debug flow filter addr 8.8.8.8)
#diag debug flow trace start 100id=20085 trace_id=1 func=print_pkt_detail line=5824 msg="vd-root:0 received a packet(proto=17, 192.168.0.139:52954->8.8.8.8:53) from LAN Aggregate. "
id=20085 trace_id=1 func=init_ip_session_common line=5995 msg="allocate a new session-0ab5d1d0"
id=20085 trace_id=1 func=vf_ip_route_input_common line=2589 msg="Match policy routing id=2132213761: to 8.8.8.8 via ifindex-14"
id=20085 trace_id=1 func=vf_ip_route_input_common line=2615 msg="find a route: flag=04000000 gw-1.1.12.1 via port4"
id=20085 trace_id=1 func=fw_forward_handler line=811 msg="Allowed by Policy-6: SNAT"
id=20085 trace_id=1 func=ids_receive line=298 msg="send to ips"
id=20085 trace_id=1 func=__ip_session_run_tuple line=3519 msg="SNAT 192.168.0.139->1.1.12.133:52954"
id=20085 trace_id=1 func=__ip_session_run_tuple line=3570 msg="run helper-dns-udp(dir=original)"
id=20085 trace_id=1 func=ipd_post_route_handler line=490 msg="out port4 vwl_zone_id 1, state2 0x1, quality 1.
"
id=20085 trace_id=2 func=print_pkt_detail line=5824 msg="vd-root:0 received a packet(proto=17, 8.8.8.8:53->1.1.12.133:52954) from port4. " <--- Traffic back from 8.8.8.8
id=20085 trace_id=2 func=resolve_ip_tuple_fast line=5905 msg="Find an existing session, id-0ab5d1d0, reply direction"
id=20085 trace_id=2 func=__ip_session_run_tuple line=3533 msg="DNAT 1.1.12.133:52954->192.168.0.139:52954"
id=20085 trace_id=2 func=vf_ip_route_input_common line=2615 msg="find a route: flag=00000000 gw-192.168.0.139 via LAN Aggregate"
Common FortiGate Flows outputs
iprope_in_check() check failed, drop
1) When accessing the FortiGate for remote management (ping, telnet, ssh, etc), the service that is being accessed is not enabled on the interface. So enable it on the interface level.
2) When accessing the FortiGate for remote management (ping, telnet, ssh…), the service that is being accessed is enabled on the interface, but there is “Trusted host” configured on the administrative level. Don’t forget to add your own IP address range
3) When accessing a FortiGate interface for remote management (ping, telnet, ssh…), via another interface of this same FortiGate, and there is no firewall policy. To solve this, create a firewall rule to allow traffic from the source interface to the destination interface.
4) This error also appears if you modify the default authentication ports for HTTPs or SSH (or others). You need to try to access it on the correct ports!
Denied by forward policy check
1) There is no firewall policy matching the traffic that needs to be routed or forwarded by the FortiGate (Traffic will hit the Implicit Deny rule).
2) The traffic is matching a firewall policy with DENY statement.
3) The traffic is matching a ALLOW firewall policy, but DISCLAIMER is enabled.
Reverse path check fail, drop
1- There is not path(or route) to the source IP of the packet.
2. The Fortigate Firewall has a route to the source IP of the packet on another interface causing asymmetric routing, which by default is denied
How Does HIFENCE Managed Firewall Help?
With decades of cybersecurity experience, HIFENCE security experts will provide our advanced Managed Firewall that delivers 24/7 monitoring, detection, analysis, and the rapid response needed to protect your entire infrastructure from today’s most sophisticated threats. If you are interested in a company to help you manage your firewall and also provide managed network services you can contact us!