can't ping SVI interface on remote switch across S2S VPN

 Had an issue could ping vlan1 (LAN) SVI but not vlan146 (VOICE)

The setup was

LAN client -> L2 VL1 -> L3 SVL VL1 -> Inside ASA -> S2S VPN -> HQ 

Phone client -> L2 VL146 -> L3SVI VL146 -> L3 SVI 1 -> Inside ASA-> S2S VPN -> HQ -> Phone server

I found some messed up NAT's

Removed the global

object network obj_any

 nat (any,outside) dynamic interface

The NoNat had a missing object in the destination

nat (voice,outside) source static obj-10.60.146.0 obj-10.60.146.0 destination static HQ-NET HQ-NETS no-proxy-arp route-lookup

Also needed this NAT on the INSIDE with the 146 networks to ping the SVI. This is becuase the route to the HQ network is through the inside interface of the ASA.

nat (inside,outside) source static obj-10.60.146.0 obj-10.60.146.0 destination static HQ-NET HQ-NETS no-proxy-arp route-lookup

new upgrade commands on 9200 and 9300 switch

Looks like the install/upgrade process has changed a bit on cisco 9200 switch. 

install add file flash:cat9k_lite_iosxe.16.10.01.SPA.bin activate commit

*This command will copy the file to other switches in the stack

If you forget the last 2 keywords you will need to run:

install inactive remove

Full guide

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9200/software/release/16-10/release_notes/ol-16-10-9200.html#id_67613

Needed this command when there was no space on the local disk

install add file tftp://172.30.180.160//cat9k_iosxe.17.06.05.SPA.bin activate commit

****

check for this before reboot

#show romvar | in STARTUP

SWITCH_IGNORE_STARTUP_CFG=0

If the variable is set, please:

no system ignore startupconfig switch all

setup netflow on cisco 9300 stack

Setup netflow

x.x.x.x = your netflow collector eg solarwinds etc.

flow exporter NETFLOW-EXP-TO-ORION
 destination x.x.x.x
 source vlan1
 transport udp 2055


Setup what you want to record

flow record NETFLOW-RECORD-IN
 match ipv4 tos
 match ipv4 protocol
 match ipv4 source address
 match ipv4 destination address
 match transport source-port
 match transport destination-port
 match interface input
 match flow direction
 collect interface output
 collect counter bytes long
 collect counter packets long

flow record NETFLOW-RECORD-OUT
 match ipv4 tos
 match ipv4 protocol
 match ipv4 source address
 match ipv4 destination address
 match transport source-port
 match transport destination-port
 match interface output
 match flow direction
 collect interface input
 collect counter bytes long
 collect counter packets long

Setup the monitors


flow monitor NETFLOW-MON-OUT
 exporter NETFLOW-EXP-TO-ORION
 cache timeout inactive 10
 cache timeout active 60
 record NETFLOW-RECORD-OUT

flow monitor NETFLOW-MON-IN
 exporter NETFLOW-EXP-TO-ORION
 cache timeout inactive 10
 cache timeout active 60
 record NETFLOW-RECORD-IN

Enable the monitors on the interfaces
Enable under the  interfaces you want to collect netflow data from usually these will be uplinks, links to other sites etc





interface GigabitEthernet2/0/36
 ip flow monitor NETFLOW-MON-IN input
 ip flow monitor NETFLOW-MON-OUT output



Don't forget to write your config.
You might need to check firewall rules between the two hosts.
Allow a few minutes for the data to populate in the collector.

installing line cards cisco in 6500

Install 720 sup in 6500

Fully open ejector levers on the new sup

Sups should be installed in
slot5 or slot6

Remove slot cover
look inside and make sure there is enough clearance, look at cables from other slots and anything inside the 6500.
line up the card and slot it in, push in slowly
Push down and in on the levers, left one then the right one you should feel it click in.

The LEDs should be green we don't want orange or red that needs to be investigated.


Install line card

Same process as above. Cards should be hot swappable but its always a good idea to schedule a maintenance window for this work.

packet capture on cisco router/switch

*** Setup ACL
ip access-list extended CAP_ACL
permit ip host x host y

*** Setup buffer
monitor capture buffer CAP_BUFF circular

*** Filter the buffer with the ACL
monitor capture buffer CAP_BUFF filter access-list CAP_ACL

*** Setup the cap point and on what interface
monitor capture point ip cef CAP_POINT fa0/0 both

*** Assign the buffer to point
monitor capture point associate CAP_POINT CAP_BUFF

*** Show the setup
show monitor capture buffer CAP_BUFF

*** Start the cap
monitor capture point start CAP_POINT

*** Send the test traffic
send test traffic ping or telnet on the port etc

*** Stop the cap
monitor capture point stop CAP_POINT

*** show brief
show monitor capture buffer CAP_BUFF brief

*** export the capture to tftp server
monitor capture buffer CAP_BUFF export tftp://10.50.50.22/mycap.pcap

*** Open the pcap in wireshark

For 3850 - but it didn't work for me

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/epc/configuration/xe-16/epc-xe-16-book/nm-packet-capture-xe.html#GUID-DCB20ADF-1F8E-434B-AE97-54802879F34F

stacking cisco switches

Before ordering

• Visual inspection of the racks
• Can we space the stack 1u apart from each member
• Usually need one 3m stack cable (top<->bottom), do we need to more ?
  
• Do we have power is it normal plug or female power cable ?
• Are power cables cisco notch or not ?
  

After the gear arrives we need to visit site to do the following:

• Check we got everything in the delivery.

• Unbox all the gear, take a picture of the serials.

• Check the power cables make sure they didn't ship the EU ones.

• Check if we got the correct power cables, do they matchup with what’s available in the racks, normal sockets or UPS (C13)

• Install any network modules and SFPs
• Put the ears on get the stack going (See building the stack below)
• Provision and set priority (See building the stack below)
• If we have dual power supplies, can the racks accommodate the extra cables?
• Check the front of the rack, could the switch be replaced are there any network cables in the way?
• Is there room to install the new stack with a space between each switch?
• Check the back of the rack, keep in mind new switches are longer and stack cables come out further, can the switch be replaced, any cables in the way?
• Get a backup of the config on the current stack
• Take note of VLANs and trunk ports

Next site visit
• Convert the config
• Install the new switches in the rack
• Swap over cables
• Deal with any issues after end user testing

Building the stack

All switches need to be on the same licence and software before they will form a stack.
Boot one switch at a time and run the licence command below and check the software version.
Download and update the software version to latest stable release recommended by cisco if required.
It will need a reboot after running it. Also switch provision / priority.
WR and shut them all down.
Connect stack cables, boot master first, 10 seconds and boot the rest

Commands
license right-to-use activate ipservices all acceptEULA
switch 1 provision ws-c3850-48p (? Get the switch make by sh ver and enter that here)
switch 2 provision ws-c3850-48p
conf t
switch 1 priority 15
switch 2 priority 14

Stack cabling
Top left to bottom right
right to next left
right to next lefts
continue until finished

Power stack cabling
Yellow cable -> yellow port
Green -> green

Copying the bin file from tftp server on my laptop to switch
copy tftp://10.56.3.200/cat3k_caa-universalk9.16.03.03.SPA.bin flash:


Install modes
Bundle Mode = BIN FILE

Install Mode = PACKAGES

The switch can't boot an image over 400mb. The later images are nearly 500mb. Bundle mode the bin file is extracted into memory on boot. Install mode the bin file is extracted to several packages with a packages.conf pointing to the packages. Install mode is the recommended.

Converting BUNDLE -> INSTALL

request platform software package expand switch 1 file flash:cat3k_caa-universalk9.16.03.03.SPA.bin to flash:

Copy the bin and extract on all switches in the stack
Set your boot to packages.conf
boot system switch all flash:packages.conf

Enable the stack port
In rare cases the switch might ship with stack port disabled
switch 1 stack port 1 enable
sh switch

Auto upgrade other switches
software auto-upgrade
wr

Stack show commands
show switch
show switch 2
show switch detail
show switch nei
show switch stack-ports
show switch stack-ports summary - to see cable lengths
show redundancy
show redundancy state

More on upgrading the stack
https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/16-8/release_notes/ol-16-8-3850.html#id_67699

renumbering switch stacks

Boot up one switch set it as master (priority 15)
set all its other values
Boot up  second switch set it as backup (pri 14)
and other values

Create the stack between the two devices.
Now add other switches to the stack and give them numbers etc.

set start priorty
provision switch 1 ws-3850-u etc

switch 2 renumber switch 1

Once you swap switch numbers you can't swap them again
Make the swaps you can
write mem
reload (takes about 10mins on 3850)

Check again
sh switch

Move the remaining switches

sample switch config

Setup a local user
username local-user privilege 15 password 0 P@55w0rd

username MYUSER privilege 15 algorithm-type scrypt secret P@55w0rd

Set the enable secret
enable secret s3cr3t

Save 
write mem

Set hostname
hostname SITE-3650

Don't try to look up hostnames
no ip domain lookup 

Set up the line settings
line con 0
 logging synchronous
line vty 0 4
 transport input ssh
line vty 5 15
 transport input ssh


Enable cdp
cdp run

set ntp server
ntp server 172.30.123.1

Set default gateway (L2)
ip default-gateway 172.30.1.1


Set banner
banner login ^
*** MY BANNER ***
^

Enable ip routing (L3 functions)
ip routing

Set L3 default route
ip route 0.0.0.0 0.0.0.0 vlan 10 172.172.10.249

Set domain name
ip domain-name mydomain.ie



Generate rsa key (for ssh)
conf t
crypto key generate rsa general-keys label MYLABEL modulus 2048

Setup spanning tree
spanning-tree mode rapid-pvst

Set up local logging buffer
Generally its set very low. Use dir to check how much space you have.
If you have a syslog server its not an issue
logging buffered 5000000 debugging

Set up AAA
aaa new-model
!
!
aaa group server radius NPS
 server 172.16.50.1 auth-port 1645 acct-port 1646
 server 172.16.50.2 auth-port 1645 acct-port 1646
!
aaa authentication login default group NPS local
aaa authentication login CON local
aaa authentication dot1x default group NPS local
!
!
!
aaa session-id common

Setup SVI on the switch
interface Vlan10
 ip address 172.30.10.253 255.255.255.0
 ip helper-address 172.16.1.50

Set management interface
interface Loopback100
 description SWITCH MGMT
 ip address 172.30.100.10 255.255.255.255

Setup so radius can come from the management IP
ip radius source-interface Loopback100

Set up radius
radius-server host 172.16.50.1 auth-port 1645 acct-port 1646
radius-server host 172.16.50.1 auth-port 1645 acct-port 1646
radius-server retransmit 0
radius-server timeout 1
radius-server key MY-SECRET-RADIUS-KEY


****
Side note - Upgraded a 3750E to 15.2 and it broke Radius
Change to calling the Group


aaa group server radius NPS
server name NPS-1
server name NPS-2

radius server NPS-1
address ipv4 172.16.35.63 auth-port 1645 acct-port 1646
pac key **********
!
radius server NPS-2
address ipv4 172.16.35.43 auth-port 1645 acct-port 1646
pac key ********
!

Setup your access ports
interface FastEthernet0/1
 switchport mode access (set the port as an access port)
 switchport access vlan 10 (data vlan for PC)
 switchport voice vlan 200 (voice vlan for IP phone)

 switchport port-security (turn on port security)
 switchport port-security maximum 2 (max 2 MAC's phone and PC)
 switchport port-security violation restrict (log and ignore the extra traffic)
 spanning-tree portfast (don't wait 60 seconds to bring the port up)
 spanning-tree bpduguard enable (err-disable the port if we detect switch/BPDU)
 no shutdown (bring the port up)

Setup trunk ports
interface GigabitEthernet0/3
 switchport trunk encapsulation dot1q
 switchport mode trunk





You may have to set tftp source interface
ip tftp source-interface

Setup VTP
You won't find settings in show run. Use "sh vtp status" and "sh vtp password" on another switch and configure the same settings on the new switch. Most likely you'll want to use the client mode.
SITE-3650#sh vtp status
VTP Version                     : running VTP2
Configuration Revision          : 15
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 30
VTP Operating Mode              : Client
VTP Domain Name                 : MYDOMAIN
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Enabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x7C 0x91 0x1E 0x52 0x99 0x80 0x60 0x5E




This one has ACL applied to SSH
line con 0
 session-timeout 15
 exec-timeout 0 0
 logging synchronous
 login authentication CON
 history size 256
line vty 0 4
 session-timeout 15
 access-class 2 in
 exec-timeout 15 0
 password 7 0034212757550A045E72
 logging synchronous
 length 0
 history size 256
 transport input ssh
 transport output ssh
line vty 5 15
 session-timeout 15
 access-class 2 in
 exec-timeout 15 0
 password 7 0034212757550A045E72
 logging synchronous
 history size 256
 transport input ssh
 transport output ssh

Setup port channel interface if needed
interface Port-channel40
 description "*** PortChan members gig1/1/1 and gig2/1/1 ***"
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 10
 switchport mode trunk
 switchport nonegotiate
 storm-control broadcast level 10.00
 ip dhcp snooping trust

Setup port channel members (channel-group)
interface GigabitEthernet1/1/1
 description *** Link to core ***
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 100
 switchport mode trunk
 switchport nonegotiate
 srr-queue bandwidth share 10 10 60 20
 priority-queue out
 mls qos trust dscp
 storm-control broadcast level 10.00
 channel-group 40 mode on
 ip dhcp snooping trust
end

interface GigabitEthernet2/1/1
 description *** link to core ***
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 100
 switchport mode trunk
 switchport nonegotiate
 srr-queue bandwidth share 10 10 60 20
 priority-queue out
 mls qos trust dscp
 storm-control broadcast level 10.00
 channel-group 40 mode on
 ip dhcp snooping trust

Setup eigrp
router eigrp 100
network 172.30.1.0 0.0.0.255 (wildcard mask)

Let cisco switch use 3rd party SFPs

Cisco SFP 
Left side = transmit
Right side = Recieve
http://www.cisco.com/c/en/us/td/docs/interfaces_modules/transceiver_modules/installation/note/78_15160.html


Commands to allow using non Cisco SFPs
service unsupported-transceiver
no errdisable detect cause gbic-invalid


taken from:
http://www.firewall.cx/cisco-technical-knowledgebase/cisco-switches/866-cisco-switches-3rd-party-sfp.html

Check your sfp
sh int gigabitEthernet 0/3 status 
sh int gigabitEthernet 0/3 capabilities (check type)

vtp config on cisco switch

You won't find vtp settings in sh run

You need sh vtp status on a working switch and copy settings


Set your vtp domain
password
vtp pruning

configuring ssh on hp procurve swtiches

Log in via console
conf t

password manager user-name [Insert_username_here]
[type password]
[confirm password]

crypto key generate ssh

Wait 1 min for this to complete (this will be pretty instant unless the switch is CPU busy)

show crypto host-public-key (if you see the keys there its all good to proceed)

ip ssh (turn on ssh)

wr mem

logout

loging via ssh

no telnet-server

command for finding interfaces which have not been used on cisco switches

Only available on 4500's with supervisor
# SHOW INTERFACE LINK

# show int | i proto|Last in

# show int | i proto.*notconnect|proto.*administratively down|Last in.* [6-9]w|Last in.*[0-9][0-9]w|[0-9]y|disabled|Last input never, output never, output hang never

This last command and filters out text you don't need.

Investigating high CPU usage on cisco switches

show processes cpu sorted | excl 0.00%  0.00%  0.00%

This command will show you the process that is using the most CPU. If its over 5% then there is a problem. Google the process name to see what it does and take it from there. View the graphs in your monitoring system to narrow down when it started. Check the logs from the switch.

9200 16.x TS steps
https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-xe-16/213549-troubleshoot-high-cpu-usage-in-catalyst.html

• IOSd
• LSMPI
• FED 
• Doppler ASIC
• Physical interface

IOSd: This is the Cisco IOS® daemon that runs on the Linux kernel. It is run as a software process within the kernel

LSMPI: Linux Shared Memory Punt Interface

Forwarding Engine Driver (FED): This is the heart of the Cisco Catalyst switch and is responsible for all hardware programming/forwarding

• Packet Delivery System (PDS): This is the architecture and process of how packets are delivered to and from the various subsystems. As an example, it controls how packets are delivered from the FED to the IOSd and vice versa
• Control Plane (CP): The control plane is a generic term used to group together the functions and traffic that involve the CPU of the Catalyst Switch. This includes traffic such as Spanning Tree Protocol (STP), Hot Standby Router Protocol (HSRP), and routing protocols that are destined to the switch, or sent from the switch. This also includes application layer protocols like Secure Shell (SSH), and Simple Network Management Protocol (SNMP) that must be handled by the CPU
• Data Plane (DP): Typically the data plane encompasses the hardware ASICs and traffic that is forwarded without assistance from the Control Plane
• Punt: Ingress protocol control packet which intercepted by DP sent to the CP to process it
• Inject: CP generated protocol packet sent to DP to egress out on IO interface(s)

show processes cpu sorted 5min | e 0.00%  0.00%  0.00% problem.

Look for highest execution time

show platform hardware fed switch active qos queue stats internal cpu policer

show platform software fed switch active punt cause summary

show platform software fed switch active punt cause clear

show platform software fed switch active punt cause summary

show platform software fed switch active punt cpuq rates | e 0        0        0        0        0        0

show platform software fed switch active punt rates interfaces

show platform software fed switch active punt rates interfaces 0x000001d2

show platform software fed switch active punt rates interfaces 0x000001d2 | e 0        0        0        0

show monitor capture cpuCap buffer brief

show monitor capture cpuCap buffer detailed

Packet captures on switch

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/16-3/configuration_guide/b_163_consolidated_3850_cg/b_163_consolidated_3850_cg_chapter_01001011.html

show processes cpu history
*'s show spikes, #'s are used for average

Script for  Intermittent High CPU

From https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-xe-16/213549-troubleshoot-high-cpu-usage-in-catalyst.html#anc28

In the event that the high CPU on the switch is intermittent, it is possible to set up a script on the switch to automatically run these commands at the time of high CPU events. The entry-val is used to determine how high the CPU is before the script triggers. The script monitors the 5 second CPU average SNMP OID. Two files are written to the flash, tac-cpu-<timestamp>.txt contains the command outputs, and tac-cpu-<timestamp>.pcap contains the CPU ingress capture. These files can then be reviewed at a later date.

config t
no event manager applet high-cpu authorization bypass
event manager applet high-cpu authorization bypass
event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 get-type next entry-op gt entry-val 80 poll-interval 1 ratelimit 300 maxrun 180
action 0.01 syslog msg "High CPU detected, gathering system information."
action 0.02 cli command "enable"
action 0.03 cli command "term exec prompt timestamp"
action 0.04 cli command "term length 0"
action 0.05 cli command "show clock"
action 0.06 regex "([0-9]|[0-9][0-9]):([0-9]|[0-9][0-9]):([0-9]|[0-9][0-9])" $_cli_result match match1
action 0.07 string replace "$match" 2 2 "."
action 0.08 string replace "$_string_result" 5 5 "."
action 0.09 set time $_string_result
action 1.01 cli command "show proc cpu sort | append flash:tac-cpu-$time.txt"
action 1.02 cli command "show proc cpu hist | append flash:tac-cpu-$time.txt"
action 1.03 cli command "show proc cpu platform sorted | append flash:tac-cpu-$time.txt"
action 1.04 cli command "show interface | append flash:tac-cpu-$time.txt"
action 1.05 cli command "show interface stats | append flash:tac-cpu-$time.txt"
action 1.06 cli command "show log | append flash:tac-cpu-$time.txt"
action 1.07 cli command "show ip traffic | append flash:tac-cpu-$time.txt"
action 1.08 cli command "show users | append flash:tac-cpu-$time.txt"
action 1.09 cli command "show platform software fed switch active punt cause summary | append flash:tac-cpu-$time.txt"
action 1.10 cli command "show platform software fed switch active cpu-interface | append flash:tac-cpu-$time.txt"
action 1.11 cli command "show platform software fed switch active punt cpuq all | append flash:tac-cpu-$time.txt"
action 2.08 cli command "no monitor capture tac_cpu"
action 2.09 cli command "monitor capture tac_cpu control-plane in match any file location flash:tac-cpu-$time.pcap"
action 2.10 cli command "monitor capture tac_cpu start" pattern "yes"
action 2.11 cli command "yes"
action 2.12 wait 10
action 2.13 cli command "monitor capture tac_cpu stop"
action 3.01 cli command "term default length"
action 3.02 cli command "terminal no exec prompt timestamp"
action 3.03 cli command "no monitor capture tac_cpu" 

getting information about stacked cisco switches

show switch ?