Network maintenance prevents issues and less troubleshooting
Change control
Schedule maintenance windows
Documentation
Communication
Consistency
Cycling equipment (upgrade network stuff every 5 years)
Monitoring, graphing and alerting
Troubleshooting laptop
Thin, light and long battery life.
USB to serial
USB to ethernet
SecureCRT, mRemoteNG, RoyalTS
TFTPd32
Filezilla
IP scanner (angry 2.x, or nmap)
wireshark
Noepad++ sublime text
Iperf
Syslog (kiwi version one is 8.3.52)
HFS (http file server)
TCPView
NMap
MTURoute
Pathping
Netflow Analyzer (PRTG)
Portable apps
Getting started IOS TS tools
Show the networks inside 10.52.0.0/16
show ip route 10.52.0.0 255.255.0.0 longer-prefixes
Show information on the CPU
show process cpu | i CPU
See what's using CPU right now (exclude stuff not using CPU)
show process cpu | e 0.00%
Keeping mind its case sensitive
show process cpu | i IP Input
You can make an alias for the commands you like
alias exec get-processes show process cpu | e 0.00%
Show interfaces that are up
Show interfaces | i is up
Show interfaces that have an IP assigned
Show ip int br | e unass
You can use brackets so it doesn't pick up REGISTERED/UNREGISTERED.
sh ephone | i ( REGISTERED )
ping can be used to test MTU
ping 10.50.1.1 size 1470 df-bit
Using extended commands to find MTU
ping (press enter)
target IP : 10.50.1.1
repeat count: 1
Extended commands: Y
Set DF bit: Y
Sweep min size:1450
Sweep max size 1500
You will see where the pings start to fail and can help figure out MTU faster.
SPAN and RSPAN
find out how many sessions are supported
monitor session ?
monitor session 1 source interface fa0/20
monitor session 1 destination interface fa0/10
Anything sent/received on fa0/20 will get sent out fa0/10. We can plug in a laptop with wireshark runing there.
RSPAN
You need to setup an RSPAN VLAN in your network and add it onto trunks etc.
You may also need a free port (or one you can shutdown). The port gives up its resources for RSPAN.
conf t
vlan 999
remote-span
exit
monitor session 1 source interface fa0/20
monitor session 1 destination remote vlan 999 reflector port fa0/30
recieving
monitor session 2 source remote vlan 999
monitor session 2 destination interface fa0/5
Your laptop is plugged into fa0/5
Switch performance troubleshooting
Cisco switches CPU usage should be low around 5% - 10%
Components of a catalyst switch
Ports
Forwarding logic (ASICS)
Backplane (fabric that connects everything)
Control plane
With switches we get intelligence with almost wirespeed.
It's rare to overload the backplane
Most of the time it will be in the ASICS
Usually this is because a lower grade switch is being used as a 1000 user core switch.
Modern switches can do wirespeed routing
Control plane is where all the intelligence is. If we need to do a job that we don't have an ASIC chip to do the specific job for us. It needs to get done here.
How to diagnose "slow" issues
show interfaces counters
show interface counters errors
Find if there are half duplex ports
sh interfaces | i Half
Find all ports with Full/auto/half
sh interfaces | i line protocol|duplex
Auto MDIX
Remove the need for cross over cable.
Needs to be set as auto on both ends.
TCAM can max out.
Make sure the right SDM template.
Show sdm prefer
See what a template prefers
show sdm prefer access
Changing the SDM template requires a switch reboot.
See whats in use and max
Show platform tcam utilization
CPU utilization on the switch matters,
Cisco 5% - 10% tends to be normal.
What could cause high CPU on a router ?
TCAM overload (STP loop)
debug command left running (u all)
Fequent routing or FHRP updates (hello timer too low)
Fequent SNMP scans (solarwinds scans too low)
Router CPU will go up as the amount of normal traffic increases
See whats using the CPU
show process cpu | i ex 0.00%
Top sources of high CPU:
ARP input
A flurry of arp packets, can be cause by a default route pointed it an ethernet interface
Net background
When an interface queue is full, net backgroud pulls from a global buffer (causes a CPU interrupt)
IP Background
Handles interfaces up/down states
TCP timer process
Manages developing and current TCP connections to the router (Telnet, SSH, SYN, etc)
Looking for these issues (^ means line starts with)
sh process cpu | i ^CPU|^PID| ARP in
sh process cpu | i ^CPU|^PID| Net Back
sh process cpu | i ^CPU|^PID| IP Back
sh process cpu | i ^CPU|^PID| TCP Tim
Show the CPU usage over time
show process cpu history
High mem usage
1 - Check your memory is supported on the new versions you are upgrading to
2 - Memory leak, you need monitoring to spot.
MALLOC_FAIL - the router can't allocate memory to something probably ran out of memory
3 - Buffer leak (IOS Bug)
4 - BGP massive routing table
Troubleshooting trunks and VTP
A trunk = a tagged port
Puts VLAN info into each frame
Layer 2 feature
Show all the trunk interfaces
show interfaces trunk
Encapsulation mismatch
Check if the only allowed vlan is the native vlan 1. This means the other vlans are not making it across. Check the trunk ports and ensure you have
switchport trunk encapsulation dot1q
on both sides
DTP mismatch
show interface fa0/10 switchport
look for
Administrative mode: dynamic auto
Operational mode: static access
dynamic auto -> dynamic auto won't make a trunk
dynamic auto -> dynamic desirable will make a trunk
Change the mode:
edit the interfaces and select one of the following modes:
switchport mode dynamic auto
switchport mode dynamic desirable
switchport mode trunk
switchport mode access
Native VLAN mismatch
Default is vlan 1
switchport trunk native vlan 20
vlan 20 will be untagged
Allowed vlan misconfig
show interface trunk, check the allowed vlans
Side A
switchport trunk allowed vlan 1,5,80
Side B
switchport trunk allowed vlan 1,5,80,200
Check the config on each trunk interface
Show interfaces trunk
Check Vlans allowed on trunk
VTP
show vtp status
vtp domain name is blank by default
You can config it to make it a bit safer to use
vtp domain name JACKSVTP
vtp password CoolPassword
Troubleshooting VLANs
1 vlan = 1 broadcast domain = 1 subnet
What can go wrong with vlans:
IP's are not in the subnet.
Inactive vlan (VTP) vlan database gets deleted
Port assigned to the wrong vlan
PC plugged into wrong port / vlan (find mac address)
Spanning tree
Step 1 - STP elects the root bridge
Root bridge is the switch with the lowest bridge ID
Bridge ID is combination of priority (set in config) and the mac address (unique to each switch)
The default bridge ID = 32768. We can force a switch to be root by changing this value.
By default the oldest switch will be come the root bridge
Step 2 - Other switches find the best path to the root bridge
Lowest cost (bandwidth)
Lowest bridge ID
Costs
10gig 2
1gig 4
100mbps 19
10mbps 100
We select the port with the lowest cost. The other ports are blocked. This is the root port
Designated port is fowarding
Blocked port is not forwarding
Blocking is done
Step 3 - Block what is left over
Only one side blocks the link in STP
The highest mac blocks it
Last tie breaker is the port number
How to identify a loop
The switch should show lots of mac address flapping and broadcast storm
All ports will be blinking in unison or on solid.
We can have temp loops with port fast
Don't have port fast on uplinks between switches.
Common causes uses plugging back into switches etc.
SVIs and routed ports
Making an SVI
interface vlan 10
ip address 192.168.0.254 255.255.255.0
vlan 10 must exist
we also need some ports assigned to vlan 10 with something plugged in (or a trunk with vlan 10)
Making a routed port
interface gig1/0/50
no switchport
ip address 192.168.0.100 255.255.255.0
Things that can go wrong
VLAN / trunk misconfig
IP addresses need to match
vlan not created / SVI shut down
Routed port missing the "no switchport" command
L2 and L3 etherchannel
We normally connect switches with 2 trunk cables. 1 cable will be used, 1 will be blocked but there for redundancy. If we need more bandwidth we can use an etherchannel.
channel-group 1
auto - pagp
desirable -pagp
active - lacp
passive - lacp
Both sides passive = no etherchannel will be formed
If we are using L3 we can do the same thing and add more cables.
interface gig1/0/1 - 4
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
switchport trunk allowed vlan add 1,100,200
channel-group 1 mode active
w - waiting to be aggregated
Need to do the same config on the other side.
show etherchannel summary
Flags
D - down
P - bundled in port channel (we want to see P)
I - stand-alone
s - suspended (config doesn't match other ports)
H - Hot-standby (LACP only)
R - Layer 3
S - Layer 2
U - in use
f - failed to allocate aggregator
M - not in use, min links not met
u - unsuitable
w - waiting
d - defualt port
We want to see this meanin they are up:
Po1(SP) - L2
Po2(RP) - L3
If we see SD/RD we have a problem.
LACP has 8 ports max. If you bundle 10 cables 2 will be in hotstandby mode
Config on switch ports must match each other
Config on switch ports must match config on channel-group
Changing the config later can cause an issue
Always config the channel-group last
show etherchannel summary
show etherchannel load-balance
sh run int gig1/0/1
FHRP - HSRP, VRRP, GLBP
These are not stateful. When a failover happens connections will need to reconnect.
HSRP - Cisco only
Hello/dead timers 3/10
Need 3 IP addresses RTR1, RTR2, VIP
VRRP - Open standard
Hello/dead timers 1/3
Can use the VIP on RTR1 so only 2 IP's needed
VRRP times are faster out of the box but hsrp can be tuned to be the same.
GLBP - Cisco only
Active active
We should use a routing protocol to avoid asymmetric routing.
If we can't we should use tracking and preemption
int fa0/1
standby preempt
Port security
Limit mac addresses
Modes
shutdown - shut the port down
protect - extra mac addresses traffic will be dropped
restrict - extra mac addresses traffic will be dropped and logged
sticky option - takes the mac that it see currently on the interface
int fa1/0/1
switchport mode access
switchport port-security (turn on, default locks to 1 mac)
switchport port-security mac-address sticky (now we can config commands)
show port-security int fa1/0/1
Common gotcha is that port-security config is in there but it hasn't be switched on.
If you use sticky you need to save the running to startup-
Spoofing features
Maybe they forgot to globally turn it on
ip dhcp snnoping (command to turn on)
What about turning it on under the interface
int fa0/1
ip dhcp snooping trust
show run int fa0/1
show ip dhcp snooping binding
sh ip verify source
Private VLANs
VLAN with vlans inside. We don't have to waste IP addresses
isolated
community
promiscuous
Private vlans light (protected ports)
switchport protected
IPv4 addressing and DHCP
192.168.1.127
255.255.255.224
IP might be a broadcast address
IP: 172.16.68.65
MASK: 255.255.255.240
GW: 172.16.68.62
The GW is in another network
Make sure helper address is correct and set on correct SVI
show ip dhcp conflict
show ip dhcp bindings
show ip dhcp server packet
NAT
We use NAT to ping LAN IP's behind the internet IP.
Inside IP: 192.168.1.50:6711
Outside IP: 100.1.1.1:6711
This is called PAT or NAT overload
Check our NAT ACL is correct and applied
Check the nat inside/outside are applied to correct interfaces
NAT can be setup with a pool as well.
Show commands:
sh ip nat translations
sh run | i ip nat
clear ip nat translations *
show ip nat statistics
debug ip nat
IPv6 addressing
Provides more addresses than we should ever need
IPv4 has run out and we have to use NAT alot
[ global prefix] [ subnet ID ] [ interface ID]
Neighbour discovery (arp replacment)
Neighbor solicitation
Joins a multicast group
Must have unicast enabled
ipv6 unicast-routing
Must have interface IP assigned and /64 subnet mask used for auto config
int gig1/1
ipv6 address 2001:1:1:1::/64
Router advertisements are suppressed
ipv6 nd suppress-ra
Show commands
sh run int gig1/1
show run | i ipv6
sh ipv6 interface
show run | s dhcp
show ipv6 dhcp binding
IPv6 access-list
Mostly the same as IPv4 ACL
There is an implicit permit above the implicit deny for neighbour discovery.
implicit permit icmp neighbour discovery
implicit deny
Only extended named ACLs are supported
ipv6 unicast-routing
int gig1/1
ipv6 address 2001:1:1:1::1/64
sh ipv6 int gig1/1
ipv6 access-list ACL1
the /128 is like /32 in ipv4
deny ipv6 host 2001:1:1::50 2001:2:2::50/128
int gig1/1
ipv6 traffic-filter ACL1
sh run | s access-list
sh ipv6 access-list ACL_NAME
show run | s interface gig1/1
sh ipv6 interface ACL_NAME
Prefix lists
Used for filtering routes in redistribution and working with BGP
Improved CPU utilization
better subnet maskj matching abilities
Similar to ACLs
Two stage matching system network and mask
This is filtering routes not for blocking a PC from a server
ip prefix-list PFL1 permit 172.30.0.0/16 le 20
Subnet mask has to be less than or equal to 20 bits
172.30.10.0/24 would not be matched
172.30.32.0/19 would be matched
ip prefix-list PFL2 permit 172.30.0.0/16 ge 20
Subnet mask has to be greater than or equal to 20 bits
172.30.1.0/24 would be matched
172.30.0.0/16 would not be matched (/16 not /20)
172.30.32.0/19 not matched
172.30.10./24 matched
Remember its a two stage check system.
Is it part of this network ? Is its subnet mask like this?
We can also make the second check more specific.
ip prefix-list PFL3 permit 172.30.0.0/16 ge 20 le 24
show commands
show ip prefix-list
show run | i ip prefix
show route-map
Static routes
By default routers only know about directly connected networks
We can config static routes to tell the router how to get to other places.
Our destination should have a route back.
Static route
ip route 10.3.3.3 255.255.255.255 10.1.1.100
Default route
ip route 0.0.0.0 0.0.0.0 10.1.1.100
Static routes are quick and dirty but do the job. We should use a routing protocol instead.
GRE tunnels
Just tunnels traffic
GRE is its own protocol 47. Its not TCP or UDP. ACLs might block.
RTR1 (51.10.1.1)
interface tunnel 1
ip address 192.168.1.1 255.255.255.0
tunnel source 51.10.1.1
tunnel destination 51.10.1.2
RTR2 (51.10.1.2)
interface tunnel 1
ip address 192.168.1.2 255.255.255.0
tunnel source 51.10.1.2
tunnel destination 51.10.1.1
RIP v2
router rip
version 2
no auto
network 192.168.1.0
network 192.168.2.0
default-information originate
passive-interface (turn off an interface)
Interface shutdown
Wrong rip version
Max hop count 15 in rip
auto summary on by default
RIP ng
This is RIP for IPv6
Make sure its turn on
ipv6 unicast-routing
ipv6 router rip MYRIP
int gig1/1
ipv6 rip MYRIP enable
Give out the default route
Setup the static route on the edge router
::/0 is 0.0.0.0 in ipv6, 2001:db8:6783:122::4 is our default GW.
ipv6 route ::/0 2001:db8:6783:122::4
Go under the interface we want to side it out, inside.
int gig1/2 (
ipv6 rip MYRIP default-information originate
ACL may block. RIPng is running on UDP 521
old rip uses UDP 520
Show commands
show ipv6 rip database
show ipv6 route rip
show ipv6 rip MYRIP
debug ipv6 rip
EIGRP
router eigrp 10
network 192.168.1.0 0.0.255.255
no auto
By default EIGRP does auto summary
Summary router - config under interface
int gig1/1
ip summary-address eigrp 10 200.1.1.0 255.255.255.0
Turn off all interfaces by default
Then enable on the interfaces we want to run EIGRP on
This is to stop making neighbors with unwanted routers
router eigrp 10
passive-interface default
no passive-interface int gig1/1
We should also use as specific network commands as we can
Authentication
key chain KC-MYKEY
key 1 (must match other side)
key-string password1 (must match other side)
int gig1/1
ip authentication mode eigrp 10 KC-MYKEY
Default route in eigrp
Easy method
setup static route
ip route 0.0.0.0 0.0.0.0 200.1.1.1
redistribute static route into EIGRP
router eigrp 10
redistribute static metric 1 1 1 1 1
Other method
ip default-network 200.1.1.1
Other method but will advertise every network
router eigrp 10
network 0.0.0.0
interfaces down
wrong AS number (debug ip eigrp packet)
Wrong network statements
passive interfaces or wrong subnet
Authentication, wrong key, password wrong
ACL EIGRP protcol 88 is not TCP/UDP
Mismatched timers flapping neighbors
show commands
show run | s eigrp
show ip eigrp nei
show ip route eigrp
show ip protocols
show ip eigrp interface
debug ip eigrp packet
EIGRPv6 (IPv6)
router ID is only needed if we have no ipv4 interfaces setup. we also have to do a no shut.
ipv6 router rigrp 10
router-id 1.1.1.1
no shut
exit
Enable under interfaces
int gig1/1
ipv6 eigrp 10
Show commands
sh run | s ipv6 router
sh run interface
sh ipv6 brief
show ipv6 protocols
show ipv6 interfaces detail
debug ipv6 eigrp
OSPF
router ospf 1
router-id 1.1.1.1
network 192.168.1.1 0.0.0.0 area 0
network 192.168.2.0 0.0.0.255 area 1
show ip ospf
Things that can go wrong
summary routes
hello timers mismatch
OSPFv3 (IPv6)
R1
ipv6 router ospf 1
router-id 1.1.1.1 (ospf uses ipv4 address as its router ID)
exit
int gig1/1
ipv6 ospf 1 area 0
gig1/1 connects to area 0
R2
ipv6 router ospf 1
router-id 2.2.2.2
exit
int gig1/1
ipv6 ospf 1 area 0
show ipv6 ospf neighbor
State should be FULL BDR
sh ipv6 route
What can go wrong
interface is down
ACL is blocking
mismatched area number / area type / timers
Duplicate router-ids
Passive interface or wrong subnet
authentication mismatch
wrong DR elected (In hub and spoke enviornment)
Show commands
show run | s ospf
show ipv6 ospf neighbors
show ipv6 route ospf
show ipv6 ospf
show ipv6 protocols
show ipv6 ospf interface
debug ipv6 ospf hello
debug ipv6 ospf adj
PBR and route-map fly by
Route maps are used in BGP alot.
Steps
Create ACL to match traffic (try to use permit)
Create route map tie the ACL to match and set commands together
Apply the router map to an interface
Check its working
Create the ACL
We need an ACL to match the traffic
ip access-list standard MY_USERS
permit host 192.168.1.50
Create the router map
route-map RM-NAME 10
Match
match ip address MY_USERS
Set
set ip next-hop 200.1.1.2
Apply to interface
int gig1/1
ip policy route-map RM-NAME
Check
show route-map
If matched the set part is performed. It needs to be applied to an interface. If policy is not matched it is dropped back into normal routing and hit specific or default route. We can deny items from matching in the route map but you should try to stick to permits.
What can go wrong
Use PBR as a crutch, try to setup routing properly first, only use when there is no way out.
Top down processing, rules are setup wrong
immediate execution on match
implicit deny all at the end
policy applied incorrectly or on wrong interface or not applied to an interface
Wrong set command sending it out the wrong IP
Show commands
show run | s router-map
show router-map
show ip policy
debug ip policy
ROUTE REDISTRIBUTION
Send metric / default metric
Route tagging / filtering
flagging internal /external
You have to make up metrics so you lose information on how far routes are really
show ip protocol (see whats running)
show ip route (see the routes)
Send RIP routes into EIGRP
router eigrp 10
redistribute rip metric 1 1 1 1 1500
Send OSPF routes into RIP
router RIP
redistribute ospf 1 metric 1
Now if we redistribute ospf into eigrp you can create a routing loop. This is where the route map comes in.
What can go wrong
Not knowing your network, you need a network diagram.
Failing to filter rotue properly
Lack of seed metric (no routes appearing) / metric is set too high
misconf on route map or ACL
Missing route map applied to redistribution (no routes)
Some routing protocol issue
When a route map in redistribution a deny really is a deny
routing protocol issue. The redistribution is working but the underlying routing protocols might have an issue.
BGP ROUTES
show ip bgp summary
show ip bgp (topology table)
router bgp 100
network 200.1.1.0 mask 255.255.255.0
Network command identifies routes that exist in the routing table. It has to match what is in the routing table. So we want them to be specific.
show ip bgp neighbor 150.1.1.5 advertised-routes
Distribution list
access-list 1 permit 200.1.1.0 0.0.0.255
neighbor 150.1.1.1 distribute-list 1 out
clear ip bgp *
show ip bgp
route map
neighbor 150.1.1.1 route-map RM-NAME out
What can go wrong
Network command, must match what is in the routing table
Next-hop device unreachable
router filter
bgp split horizon (router via iBGP)
Better path chosen via best path decision making process
Show commands
show run | s bgp
show ip bgp
show bgp ipv4 unicast
show ip router bgp
Syslog / snmp / ntp:
NTP
ping pool.ntp.org or look up on their site
router ntp server paste IP you saw above
SNMP
snmp-server v2c community passwd ro
SYSLOG
logging buffered - log to local device but gone after reboot
logging on
logging 100.1.1.1
What can go wrong
ntp: udp 123
snmp: udp 161-162
syslog: udp 514
service timestamps datetime
snmp-server ifindex persist
community string
IP SLA
sends probes to test validate / measure connections
ip sla monitor 1
type echo protocol icmpecho 150.1.1.1
timeout 100
frequency 1
ip sla schedule 1 lifetime forever start-time now
show ip sla monitor statistics
track 10 rtr 1 reachabilityu
exit
ip route 0.0.0.0 0.0.0.0 150.1.1.1 track 10
If track 10 goes down this route gets removed from the table
SPAN and RSPAN
SPAN port copy all traffic from one port to another plug in your laptop with wireshark to troubleshoot
RSPAN you can carry the RSPAN through the network back to your desk.
SPAN and RSPAN
Sometimes we need to watch the traffic with wireshark. Its only good when we can see the traffic. Switchport anayser
monitor session 1 source int fa0/12 both
monitor session 1 destination int fa0/1
*** Note status of interface / source port in an active SPAN will be up (connected)
Anything send/recieved on port 12 will be sent to out port 1. We can have the wireshark here watching the traffic.
SPAN is great when we are sitting beside the switch. What about a remote switch ? Make a RSPAN VLAN and trunk it to your workstation
On remote switch
vlan 999 (will have to be added into trunks)
remote-span
exit
minitor session 1 source int fa0/10 both
monitor session 1 destination remote vlan 999 reflector-port fa0/11
Reflector port gives up its asic resources. Make sure its not in use
On the local switch
vlan 999
remote-span
exit
monitor session 2 source remote vlan 999
monitor session 2 destination interface fa0/1
RSPAN recap
1 - Setup RPSPAN vlan on all switches
2 - Setup monitor session 1 source interface (what we want to monitor)
3 - Setup monitor session 1 destination RSPAN VLAN
4 - Setup monitor session 2 source RSPAN VLAN
5 - Setup monitor session 2 destination interface (where we have wireshark)
Telnet and SSH
Telnet is dead and should not be used but some old routers only have it
Password required none set
Base config
line vty 0 4
login
Logins are required but no password set
conf t
line vty 0 4
no login
You can login without a password. No negates the requirement for a login.
conf t
line vty 0 4
login
password PW
SSH
ip domain-name domain.com
crypto key generate RSA module 2048
line vty 0 4
transport input ssh
transport output ssh
login local
exit
username ssh-user privilege 15 password cisco
ssh from a router
ssh -l ssh-user@150.1.1.1
Or use putty or some other client etc
What can go wrong
IP of device not reachable
Incorrect transport protocols
forgotten or misconfig login or password commands under vty
ACL blocking TCP 22 (ssh) TCP 23 (Telnet) or maxed out vty lines
Correct ssh version an key size used
show commands
show run | s vty
show run | s aaa
show run | s username
sh user
show ssh
Troubleshooting AAA
We can't config usernames and passwords on all devices constantly
We can config AAA
aaa new-model
the default for all
aaa authentication login default radius local
This one has a custom name
aaa authentication login MY-CUSTOM group radius local
radius-server host 10.10.10.5
line vty 0 4
login authentication MY-CUSTOM
You want to be able to log into the console
aaa authentication CONSOLE-LOGIN local
line con 0
authentication CONSOLE-LOGIN
You don't want radius on the console during an outage as it can take time to timeout before logging in.
LABS
