hardening anyconnect ssl on cisco secure firewall 7.7

 Hardening

https://www.cisco.com/c/en/us/support/docs/security/secure-client/221880-implement-hardening-measures-for-secure.html

Service access object (geo block anyconnect):

https://www.cisco.com/c/en/us/support/docs/security/secure-firewall-threat-defense/222810-configure-geolocation-based-policies-for.html

Auto shun (flex config)

https://www.cisco.com/c/en/us/support/docs/security/secure-firewall-threat-defense/222383-configure-threat-detection-for-remote-ac.html#toc-hId--1375473333

Threat detection service remote-access-authenticaiton hold-down 15 threshold 5

hold down - how long after the last failed attempt when new failures are counterd

thresthold - number of failed auths, if we fail login with in 5 mins 3 times

Show commands:

show threat-detection service remote-access-authentication

show threat-detection service remote-access-client-initiations

show threat-detection service invalid-vpn-access

Logs locations in FMC:

Some of them change depending on version of interface 

Cog (top right) > monitoring > syslog

Monitoring → VPN → Remote Access (anyconnect)

Analysis → Connections → Events (won't show auth success/failure here)

Health → Events (shows cpu/memory issues)

There is a unified log as well

Log level

FMC GUI → Devices → Platform Settings

syslog > logging

Syslog messages:

%ASA-6-113004: AAA user authenticated successfully

%ASA-6-113005: AAA user authentication Rejected

%ASA-4-722051: Group <group> User <user> IP <ip> Session connected

%ASA-6-722041: Session disconnected (reason...)

***

Enable Access Control Logging for VPN User Traffic

On the rule that allows anyconnect in (log at start/log at end/ send to FMC) (still won't show login attempts)

search messsage *x.x.x.x*

Ok so lessons learned there:

You won't see anything in analysis > connections > events (doesn't show ssl rejected stuff, only successful https etc connections)

SAML (staff profile) rejection was not showing in the logs. I will ask cisco about this maybe it's a different event ID or Duo doesn't send anytning back to the firewall when SAML fails. He could see it in Duo logs so was happy enough there.

AAA/radius (contractors profile) was where we saw the issues coming in.

We needed to enable informational level logging under devices > platform settings. (we hit a bug there it didn't set first time not sure what happened, I watched him set it so watch out for that)

Once we had informational we could see some stuff under the cog > monitor > syslog.

Filter "message" for *x.x.x.x* where x.x.x.x is the IP you are looking for

We saw ssl denied for the German IP based on geoblock

We saw username ***** denied from my UK IP

The username was hidden in the logs

Added flex config "no loggin hide username" cisco said you have to leave off the "g" loggin is not a typo

Shun settings were 20 failed logins in 10 mins, he said he had to set it high because users were actually typing password wrong 10 times so that is something he will have to live with. Attackers can still try from the approved IPs for 20 tries then get shun'd but at this stage they can change IP address and try again

*** SAML

SAML failed logins won't show in the FMC log in the same way. Best to review these in the SAML provider dashboard/logs (Duo/Azure etc). These logs can be forwarded from there to a central SIEM. Duo have a tool called duo log sync.

filter to include class “vpnc”, which outputs logs like:

%FTD-6-611102: User authentication failed: IP = IP address, Uname: user

TS FTD like TAC

 https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2020/pdf/R6BGArNQ/TECSEC-3004.pdf

system support trace

packet tracer not working well in FMC/FTD

Packet-tracer never worked well with VPN traffic, that was ok but now in FMC/FTD its also not working at all if you have snort or geoblocking rules. You will see an ip any any allow. Instead you must use the system support trace on live traffic. The whole point of packet tracer is that we don't always have live traffic or access to generate live traffic.

From cisco:

Indeed, from the packet tracer side it looks like the packet is going through in that IP permit any any, but that rule in reality does not exist.

Any rule which relies on snort will be classified by the box as a L4 permit ip any any, and unfortunately having a geodb rule looks like a snort rule for the box.

This is documented here:

https://www.cisco.com/c/en/us/support/docs/security/secure-firewall-threat-defense/218196-understand-how-lina-rules-configured-wit.html

Rules with Snort Features Are Deployed As Permit Any Any

When you create a rule with features that are run by Snort side, like Geolocation, URL (Universal Resource Locator) filter, Application detection, etc, they are deployed on Lina side as a permit any any rule.

At a first glance, this can confuse you and make you think that the FTD allows all the traffic on that rule and stops the rule match verification for the rules that follow.

We also have an enhancement request for this: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwd00446

Unfortunately, this breaks the usage of the packet tracer feature, and in this case, you should rather use “> system support trace” if there is live traffic.

Change the mgmt IP on FTD with minimal impact

Update mgmt IP on FTD


1. Disable management of the device in FMC. Do that via Device Management > edit the Device > Device tab > move slider next to management section.

2. Change the address on the device directly using "configure network ..." command from the cli.

3. Edit the management address in FMC from the same place you disabled management. Then move the slider back to enable management.

Firepower hotfix and patch schedule info

 When cisco find an issue they sometimes release a hotfix as a small quick fix.

The real fix will be put into the next maintenance release.

6.6 last FMC that supports cisco user agent ID. If you want to go above you need to uninstall useragent and install Cisco ISE-PIC agent

6.7.0 removes support for old ciphers. If you are using FTD you need to make sure all VPNs have been migrated to IKEv2 and updated ciphers.

ASA55xx-X devices only support up to certain sensor patches at time of writing 6.6.5.2

When you upgrade to 6.7.0 for example you should also upgrade to the latest release of 6.7.x at the time of writing it was 6.7.3 this ensures you get all fixes in your branch.

Because branches are worked on by different dev teams just because its fixed in 6.4.0.14 does not mean the same issue is fixed in 6.7.0 so you need to make sure to go to 6.7.3 or what ever is that latest patch in that branch

fmc error after upgrading to 6.7.3 interface modified

You get a health warning that interfaces are modified after upgrade to 6.7.3

SSH into FMC

enter "expert" mode

enter "sudo su"

Run this command

OmniQuery.pl -db mdb -e "select status,category,hex(uuid),body from notification where status=11;"

For each UUID delete the notification

OmniQuery.pl -db mdb -e 'delete from notification where uuid=unhex("XXXXXXXXXXXXXXXXXXXXXXXX");'

OmniQuery.pl -db mdb -e 'delete from notification where uuid=unhex("YYYYYYYYYYYYYYYYYYYYYYYY");'

OmniQuery.pl -db mdb -e 'delete from notification where uuid=unhex("ZZZZZZZZZZZZZZZZZZZZZZZZZZZZ");'

Check again, it should be blank

OmniQuery.pl -db mdb -e "select status,category,hex(uuid),body from notification where status=11;"

Check your FMC interface the alert should have cleared

show patch history on FMC CLI

 SSH to FMC

expert

cat /etc/sf/patch_history

This is useful for seeing hotfixes applied as they don't show in the version number in the web interface

Azure to Cisco Firepower FTD S2S VPN issues

The issue

Azure’s IKEv1 VPN is “policy based” by default (crypto map)

Azure’s IKEv2 VPN is “route based” by default (VTI / tunnel interface / routes)

Cisco FTD side IKEv2 VPN is “policy based” (crypto map)

My FTD was running version 6.6.1 which doesn’t support the VTI interfaces needed for route based VPN. VTI support is added in version 6.7 but that version also requires 32GB of RAM and it also deletes old DH groups liks DH group 2. All 3rd parties would need to be contacted. All S2S VPN’s with 3rd parties would need to be updated. That is a significant amount of work that would need to be co-ordinated and would need sign off and OOH work etc.

Why it works sometimes

When Azure side is trying to initiate the traffic. It’s trying to use IKEv2 route based which won’t work. However when the lifetime is reached and the VPN re-keys if the Cisco side initiates the traffic with IKEv2 policy based the azure side will accept that connection. That is why it works sometimes and does not work other times. It just depends on which side tries to bring the VPN up first after it has gone down from lifetime expiry.

Some possible fixes:

1 – Change the VPN to the old IKEv1 policy based VPN. This should work but might have implications for security audits etc. 

2 – Change azure side to be policy based, and responder only. Cisco TAC said there is a checkbox to make the VPN policy based and responder only. Azure side will need to go into powershell and manually add traffic selectors

The TAC engineer said a support ticket with Azure may be required to set this up. 

Cisco side will need to setup a script to constantly ping something on the Azure side. This will keep the Cisco side initiating the VPN.

3 – Upgrade Firepower to 6.7. May need RAM upgrade. Will need to contact all 3rd parties which have a site to side and co-ordinate updating all the VPN settings.

Possible quick fix/work around:

Setup the ping –t from the Cisco side to the azure side

Clear down the VPNs (affects all S2S VPN’s)

Do this a few times until we can bring the VPN between Azure <-> DLR backup with DLR side as initiator

Daniel can you give me a host to ping on the Azure side (10.5.0.0 255.255.255.0), I don’t think it even needs to respond but just something I can use to generate traffic to match the VPN.

Get firepower sensor (sfr) details from ASA CLI

Gives sensor IP and manager IP (FMC)

show modules sfr detail

backup on FMC

link below explaining the backup for FMC and FTD,

https://www.cisco.com/c/en/us/td/docs/security/firepower/60/configuration/guide/fpmc-config-guide-v60/Backup_and_Restore.html

http://www.network-node.com/blog/2019/3/27/150-copying-backing-up-and-restoring-ftd-device-configuration

High unmanaged disk usage on /ngfw cisco FMC/FTD error

After upgrade to 6.7 I got this error in the FMC health section:

High unmanaged disk usage on /ngfw cisco 

Going into the CLI is appeared there was space.
It looks like a bug CSCvc03899. 
Some old install files left behind. 

You need to remove them but you won't be able to roll back to that version. My system was stable and I had taken a backup of the FMC and managed device on 6.7 and no intention to roll back to any previous version anyway.

From CLISH (>) run cleanup-revert
> upgrade cleanup-revert
It is not possible to revert back to the previous version once the revert version is deleted.
Are you sure you want to proceed (yes/no)? yes

Go into expert mode and then (sudo su -)

Find old version files
FTD#cd /ngfw/Volume
# du -hs * | sort -rh
7.3G 6.4.0 ---> old version
6.9G root1
4.1G lib
116K home
0 root

Delete old version files Delete 6.4.0 file as below:

/ngfw/Volume# rm -rf 6.4.0
# du -hs * | sort -rh
6.9G root1
4.1G lib
116K home
0 root

cisco firepower FTD 2110 initial config

 First steps

• Laptop / console cable, network cables, racking bolts and tools
  
• Have mgmt IP assigned for it. Will also needs IP's for inside/outside other interfaces etc.
  
• Unbox and put ears on
• Record serial numbers (there is a tab at the front interfaces)
• Mount in racks and boot up
• Plug into console port with usb to serial cable going to laptop
• Mgmt interface should be plugged into VLAN where it can reach the FMC.

Default username and password

username: admin

Password: Admin123

You'll want to change that 

connect ftd
show user 

configure user password admin 

 

Add another user

You may want to add another user as a back up

configure user add myusername

Configure  management IP

• configure network ipv4 manual 192.168.100.50 255.255.255.0 192.168.100.254
• 192.168.100.50 = MGMT IP of FT
•  192.168.100.254 = GW IP
• ping system 192.168.100.254
  

Configure route (if needed)

configure network static-routes ipv4 add eth0 192.168.10.0 255.255.255.0 192.168.1.1

Add FTD to FMC

Login into to FMC web interface

Devices -> Device management -> add

Fill in IP of FTD and a key like "cisco"

Now go back to FTD cli

show managers (should be none set)

configure manager add [IP-ofFMC] cisco

Wait for it to complete

Now go back to FMC. Give some time for the new device to settle (i think auto policy deploy happens). You can now upgrade the FTD to the same software as your other FTDs.

Overview -> Dashboard -> Status

Once upgraded your FTD is ready to be configured and policy deployed. You will need to patch any interfaces like inside/outside to the correct devices/VLANs. You should make sure you have access to SSH into the mgmt interface for troubleshooting. Most management and config is done from the FMC web interfaces from here on.

cisco firepower flex config gothcha

 I was trying to apply some flex config but it wasn't appearing in the CLI.

After opening a case with TAC he showed that we need to edit the objects in

Device -> Flexconfig

Now we can deploy the policy to make the changes.

He also mentioned we should use AD authentication method

Also we found from the debug the mapping of our group should be as follows

CN=AD_GROUP_NAME,OU=VPN,OU=Groups,DC=CUSTOMER,DC=COM

download packet capture (pcap) file from FMC / FTD / firepower

connect to the sensor of the FTD

use "system support diagnostic-cli" to go into ASA CLI

setup your capture as normal and capture your traffic.

Once complete "copy /pcap capture: disk0:"

now type exit twice to get out of ASA CLI

type "expert"

cd to "/mnt/disk0"

cp MYCAP.pcap /ngfw/var/common

On the FMC web interface

Devices -> hammer + wrech icon -> advanced 

Go into advanced troubleshooting -> File download

Enter MYCAP.pcap and click download.

uploading firepower TS files to cisco faster with Customer eXperience Drive (CXD)

One of the big pains dealing with firepower is TAC will ask for TS files a lot. They take a long time to generate then you have to return to the FMC to download them, sometimes they can be 1gig or so and finally you need to upload them to the case. Cisco have made it a bit faster with cxd.

When you open you case with cisco, make note of the case number. Click the button to generate the token.

Log into the CLI of your FMC via ssh
expert
sudo su
curl -k https://cxd.cisco.com/public/ctfr/firepower.py | python - -c [CASE#] -t [TOKEN] --auto-upload &

Do the same on the active FTD as well.

You can move on with your day and the TS files with automatically generate and upload to the case.

You need to have DNS resolution working.

On sensor CLI 

Make sure the gateway and DNS servers are setup and working in the FTD CLI.

show networking

Configure the IP and GW

configure network ipv4 manual 192.168.100.50 255.255.255.0 192.168.100.254

nslookup cxd.cisco.com 192.168.100.53

Configure working DNS servers

configure network dns servers 192.168.100.53,192.168.100.53

The "ASA" parts DNS is handled from the GUI Devices -> platform settings -> DNS

Also look under system -> config -> Management interfaces

Enable DNS

Add a group 

Assign to interfaces (inside,outside)

wildcard cert on firepower FTD

In this case the wildcard was installed on a windows server (exchange)

I opened mmc
added the certs snap in
Found the wildcard cert
Exported it with the private key (set a password)
Exported PFX

In firepower went to objects -> PKI -> cert enroll
Selected import from PCKS12 files

Now go to devices certficates -> add

Now go to devices -> VPN -> Remote access

Edit the anyconnect profile
Access interfaces tab
Change the two entries SSL and IKEv2 and select the new cert

Save + Deploy

error message about silo drain on FTD

> system support silo-drain

And look the option for "connection events"

Then run the following commands as root (expert sudo su):

#pmtool RestartById SFDataCorrelator

#pmtool RestartById diskmanager

download pcap file that was created in the FTD CLI

You might setup a capture inside
system support diagnostic-cli

copy /pcap capture:cap-name

now go into normal FTD cli
expert mode
cd /ngfw/mnt/disk0/
cp cap-name.pcap /ngfw/var/common/

Now you can download inside.pcap from the web gui
Devices -> Device MGMT -> Troubleshoot icon on FTD -> Advanced Troubleshooting

Its also possible to copy off with the copy command to scp/tftp

disable http2 on cisco firepower FTD/FMC for ssl decryption

I tired to setup SSL decryption following the documentation from Cisco.

I was getting an error in the browser ERR_SSL_VERSION_INTERFERENCE

This is because the FTD tells the web server that it supports http2 which it actually doesn't so you have to disable that and use http1.1

From Cisco TAC:

Here is the command regarding disabling HTTPv2.0 on firepower:
> system support ssl-client-hello-tuning extensions_remove 16,13172
Then you need to restart snort using following command on expert mode, this will cause network outage for a few seconds
>expert
# sudo pmtool restartbytype snort

firepower FTD resources

https://communities.cisco.com/docs/DOC-30977

https://communities.cisco.com/community/partner/security