Collecting logs from a juniper ssg firewall

Turn on logging in putty

set console page 0

get tech

set console page 22

If trouble shooting a VPN

set console page 0

get tech
get ike cookie
get sa
get route
get event include x.x.x.x (peer address)
get event type 536

set console page 22

Cisco NAT changes from version 8.3+

There is a short explanation of post-8.3 NAT implementation here:
https://supportforums.cisco.com/docs/DOC-12690#ACL_Changes

There is a very detailed document where you can find examples of every kind of nat in old and new syntax here:
http://www.cisco.com/en/US/docs/security/asa/asa83/upgrading/migrating.html

There is a video here also:
https://supportforums.cisco.com/docs/DOC-12324

If you are migrating from pre 8.3 to post 8.3 you may need to convert your VPN, ACL and NAT configurations :(

Further Cisco VPN Studies

Configuring a VPN

Creating a VPN has several parts. Some are optional depending on how you configure your firewall. The two main components are phase 1 and phase 2.

Phase 1 (isakmp)

Is to create a secure foundation to start phase 2.

Phase 2 (ipsec)

This is the secure tunnel used for the actual data transfer.

Set up phase 1

Enable isakmp 

crypto isakmp enable

Create isakmp polices

crypto isakmp polocy 10 

encryption aes 128

hashing sha

group 2

authentication pre-share

lifetime 28800

Isakmp policies are your phase 1 policies. You will need to connect to several 3rd parties. They won't all use the same phase 1 settings. You will need to create several phase 1 policies to accommodate all of your VPNs. When attempting to build a VPN your fw will go through the list of policies from top to bottom until it finds a policy that matches exactly. If it doesn't find one the VPN will not come up. The lifetime is how long the phase1 tunnel will stay up before it is torn down, new keys are are generated and the phase 1 tunnel is created again. 28800 is 8 hours.

Set up the pre-shared key 

crypto isakmp key 0 P@55w0rd

The other side needs that password, exchange it over the phone. It should be a good password. Common cause of VPNs not coming up is a pre-shared key mismatch, someone has made a typo.

Configure the isakmp identity (optional)

This is not required unless you are using hostnames. Only useful if your public IP changes.

crypto isakmp identity 200.50.200.230  

Set up Phase 2

Create your ipsec transform set 

crypto ipsec transform-set esp-aes-esp-sha

esp-aes 128

esp-sha-hmac

security-association lifetime kilobytes 20000

Much like the isakmp policies. You will have to set up several transform sets for use with different VPNs. You can set the lifetime in seconds or data. When the limit is reached the phase 2 tunnel will be rebuilt. 

Match traffic to be encrypted (ACL)

To do this you need to create an access list. You may need to create mirrored ACLs to allow the traffic back from the other side.

ip access-list extended traffic_to_encrypt_over_vpn1

permit ip source 192.168.10.0 0.0.0.255 destination 10.50.1.0 0.0.0.255

Create the crypto map

This puts your phase 1 and phase 2 configurations together

crypto map VPN1_Map 10 ipsec-isakmp match address traffic_to_encrypt_over_vpn1

set peer 200.50.50.73

no set pfs

set transform-set esp-aes-esp-sha

crypto map [map_name] [secuence_number] [vpn_type]

The peer address is the IP of the other firewalls outside interface

PFS is more secure but adds overhead

Apply the crypto map to the outside interface

interface gi0/1

 crypto map VPN1_Map

Don't forget the nonat statement 

There are two main approaches here. You can create nat and nonat ACLs and use those. Alternatively you can just create a nat ACL and add a deny where you don't want an address to be NAT'd

deny 192.168.10.0 0.0.0.255 10.50.1.0 0.0.0.255

This deny's that network from being NAT'd when attempting to connect to the network on the other side of the VPN. Remember you need to nonat networks attempt to return pings etc. Think about this if you have mirrored ACLs on either side.

NAT traversal
If both firewalls are behind a NAT you will need to enable NAT-T.
crypto isakmp nat-traversal 20

Testing

Display active phase 1 security associations

sh crypto isakmp sa 

QM_IDLE is good, its done negotiating phase 1

Display active phase 2 security associations

sh crypto ipsec sa 

There is a lot more information here

Check local and remote addresses are correct

You want to see encaps and decaps increasing

You don't want to see send/receive errors increasing

Force VPNs to be rebuilt 

clear crypto isakmp

new firewalls let you select which VPN, older ones just rebuild them all

Creating VPNs with the Cisco SDM

Due to their complexity a lot of users use the SDM wizard to create VPNs. However the SDM adds junk and gives entry names that will be hard to understand when reading back later. This will make troubleshooting much harder. If you are going to use the SDM at least use it in this way. There is a button in the SDM where you can create your VPN with a wizard but instead of installing it on the firewall you can view the commands the SDM was going to run. Here you can go in and change names to something sensible, use the description command. Copy the commands and run them on the firewall manually yourself. This way your config won't get messed up and you can become familiar with the commands required.

Working with 3rd parties

Most of the time you will be setting up a VPN with a third party. You won't have any control over their firewall. In these cases its a good idea to create a VPN template with the settings you use, your peer address, your contact details etc. Send the template to the 3rd party get them to fill it in and send it back.  Exchange the pre-shared key over the phone. You can go ahead and set up the VPN with the settings provided. It should come up, if not you should have each other contact details for troubleshooting.

The first thing is to attempt to bring up a VPN that allows any service and a weak pre-shared key like 1234567890. Confirm phase 1 settings, confirm phase 2 settings. Get them to send screen shots. Once you get the VPN up you can work on restricting access and don't forget to install a strong pre shared key.

checking cisco devices for errors

Quick command to see if there are any errors on any interfaces
# sh interface | i error
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 0 interface resets
10 input errors, 0 CRC, 0 frame, 10 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 0 interface resets
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 0 interface resets
142 input errors, 0 CRC, 0 frame, 142 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 0 interface resets
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 0 interface resets
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 0 interface resets

We can see there are errors on two interfaces lets find the one with 10 errors.
# sh interface | b 10 input errors
10 input errors, 0 CRC, 0 frame, 10 overrun, 0 ignored, 0 abort
4116571872 packets output, 3222361986 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/128)
output queue (curr/max blocks): hardware (0/128) software (0/29)
interface ethernet2 "dmz-mgmt" is up, line protocol is up

With this command we see the next interface is ethernet2, so the problem interface is ethernet1. Running the commands "sh interface" and "sh ip interface brief" can be useful for finding the interface you are looking for. Collisions usually suggest a speed/duplex mismatch. Common practice is to set auto for desktops but configure it for servers, network equipment etc.

This command will give you more detail on the interface
# sh interface ethernet1
interface ethernet1 "dmz-mon" is up, line protocol is up
  Hardware is i82559 ethernet, address is 0012.438a.f237
  IP address 172.19.131.1, subnet mask 255.255.255.0
  MTU 1500 bytes, BW 100000 Kbit full duplex
        291592760 packets input, 1207931955 bytes, 0 no buffer
        Received 475697 broadcasts, 0 runts, 0 giants
        10 input errors, 0 CRC, 0 frame, 10 overrun, 0 ignored, 0 abort
        4116575267 packets output, 3224176751 bytes, 0 underruns
        0 output errors, 0 collisions, 0 interface resets
        0 babbles, 0 late collisions, 0 deferred
        0 lost carrier, 0 no carrier
        input queue (curr/max blocks): hardware (128/128) software (0/128)
        output queue (curr/max blocks): hardware (0/128) software (0/29)

At this point we should investigate what is connected to the other end of the interface. You need to investigate the errors that appear in the syslog. Check that the speed and duplex are configured correctly on each end. It can help to start graphing the errors so you can pin point when the errors are happening. It can help you narrow it down. The clear counters command can be used to clear counters when errors have been resolved. If the errors are occuring in realtime you can clear counters and watch the rate at which the errors are increasing.

You should also check that the clock is in sync and that ntp is configured.

Good link on linux bonding

http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding

hyper-v studies

Installing and configuring host and parent settings

Before adding the hyper-v role

Before adding the hyper-v role you need to spec your VM hosts. How many VMs will you need etc. How much RAM CPU, storage etc.

You can see a full list of supported VM guest OS's here, make sure you are supported:
http://technet.microsoft.com/en-us/library/cc794868(WS.10).aspx
Just because its not supported doesn't mean that it won't work but if you have an issue and need to call microsoft they will not support it.

Hyper-v integration services
Always install the hyper-v integration tools where possible. FYI when you upgrade the kernel on linux systems you need to update the hyper-v integrations tools also. I have a blog post about doing this.

BIOS settings for the Hyper-V hosts
You need to turn on the following options in the BIOS on the hyper-v hosts. Sometimes it is enabled by default but still needs to be checked.

Hardware assisted virtualization
AMD: AMD-V
Intel: VT

Hardware enabled data execution prevention
AMD: NX no execute bit
Intel: XD execute disable bit

You can get itainium CPUs to work but its pointless, you need to use a 3rd party hypervisor and they are end of life.

Adding the hyper-v role

A role is a primary function of the server. Features enchance the roles on the servers. Hyper-v is a role, failover clustering is a feature. To add the hyper-v role

GUI:
Use server manager -> add roles -> add the hyper-v role

Command line:
DISM /online /enable-feature /featurename:Microsoft-Hyper-V
start /w ocsetup Microsoft-Hyper-V

Powershell:
Add-WindowsFeature Hyper-v

Hyper-v server
Hyper-v server is a special OS just for hyper-v hosts. It doesn't have other roles but does have the failover clustering feature. Hyper-v server is free but you don't get any licenses for the guest OS's. Hyper-v on windows data center edition you get unlimited guest OS's. The hyper-v role is turned on by default. There is a text based interface on hyper-v server which you should used to set up remote management and then manage the server from your desktop.

Enabling hyper-v by SCVMM
As soon as you add a host to SCVMM it will check that the hyper-v role is added, if not it will add it.

Securing the hyper-v hosts
Isolate the host and guest networks
Use server core (smaller attack surface, less patches)
Server core can also be used for webservers in the DMZ etc
Bitlocker doesn't work with clustered disks keep your hosts in a physically secure enviornment

Enable remote management

Remoting with Hyper-v Manger
Right click -> connect to server

Remoting with RDP
start -> run -> mstsc
Conect to hosts or guest VM's

Remote Server Admin Tools
http://www.microsoft.com/en-us/download/details.aspx?id=7887
Once installed you can add hyper-v, failover clustering and more to administative tools

Remoting with RD connection manager
http://www.microsoft.com/en-us/download/details.aspx?id=21101
You can build a tree of host -> guests
Gives you a little thumbnail of each console you can click on each one to connect

Deploying the VMM agent with SCVMM
Default ports are 80 (management) and 443 (data)
Some users change these defaults for security reasons
DMZ servers need a local install
DMZ service is given a randomized name

Configuring firewall rules
Usually installing hyper-v the firewall is automatically configured
netsh advfirewall show rule name=all dir=in > fwrules.txt
notepad fwrules.txt
On server core use SConfig tool
Clustering the VMM library requires 'Remote Volume Management' fw setting enabled on all nodes

VMM Firewall ports
WinRM 80
SMB 443
DCOM 135
TDS 1433  (Tabular data stream)
WCF 8100 (VMM self service portal web server to VMM server)
HTTPS 443
BITS 443
VMRC 5900
RDP to vm hosts 2179
RDP to vms 3389
SFTP 22

Configure virtual networks and VLAN security

Virtual network manager
There are three network types:

• External (Everything, other physical servers on the network)
• Internal (VMs on the same host and the VM host)
• Private (Only VMs on the same host can communicate, good for testing)

You are only allowed 1 external network per physical NIC on the host.
Internal networks connects VMs on the same host, VMs can also connect to the VM host.
Private networks enable VMs on the same host to communitcate only.

You can enable VLAN tagging. If you have 4 NICs you'll probably keep one for management, but you can completly isolate the other 3 networks with VLAN tagging.

Configuring MAC addresses
VM hosts have a pool of MAC addresses which it automatically assigns to VMs as they are created.
VMs are automatically assigned a dynamic MAC addresses by default.
Within a host they are checked for conflicts, but they are not checked between hosts which don't talk to each other. It's rare but a MAC conflict is possible. If you are using SCVMM it will check for conflicts on all hosts.

VLANs can be used for security. It can only be used with External and Internal networks. Most of the time you will use it with external networks and assign it to a physical NIC. It can be a good idea to rename your NICs so it's obvious what they are used for.

If you only have one NIC or limited NICs on your host you can check a box under allow management operating system to share this network adapter. This allows the host OS to do VLAN tagging and management on the single NIC.

I have seen issues with linux servers where the MAC address is hard coded on the server but for some reason it is migrated to another host and assigned a new MAC address which causes an issue. To resolve we needed to use static MACs.

Use static MAC addresses for DHCP
Use MAC address spoofing for NLB

Configure storage

Planing for disks and storage
Hyper-V hosts can use:
Direct attached storage (DAS) disks are in the hosts
Storage area networks (SAN)

• Required for failover clustering so all VM hosts can access a disk
• Host clustering: Fibre Channel, FCoE, Serial Attach SCSI (SAS), iSCSI
• Guest Clustering: iSCSI

VMs require storage for:

• Virtual hard disk files (VHD)
• Snapshots (AVHD)
• Failover clustering
• Application data files

Fibre channel - Fast and expensive, working with fibre cables/connectors can be painful
iSCSI - Cheap but not as fast, uses ethernet, simple setup.

Hyper-v currently does not have a method (virtualized HBA that suppots fibre channel) to attach a VM to a fibre channel disk. No way to communicate VM -> hypervisor -> VM protocol -> physical storage device. If you need to do this you have to use iSCSI.

iSCSI will never be as fast as fibre channel. However performance is decent for most small medium implementations. Make sure you are using switches (and cables) that are fit for purpose gigabit ethernet and jumbo frame. You may need to configure the NICs on your VM hosts to use jumbo frame.

VM storage
Default locations
VHDs: C:\Users\Public\Documents\Hyper-V\virtual hard disks
VMs (config files): C:\ProgramData\Microsoft\Windows\Hyper-V

• Virtual Machines (XML file)
• Snapshots (.avhd)

Considerations
Performance
Hard drive space
Security
Shared storage for failover clustering

Multipath I/O (MPIO)

• Multiple read/write paths from the VM to the storage
• Provides redundant failover and load balancing support for disks or LUNs
• Supports bandwidth aggregation
• Distribute I/O transactions across multiple adapters
• It is a windows server feature which can be added (look for latest updates / hotfixes)

Launch the console
start -> run -> mpiocpl

MPIO Devices
Lists devices and allows add/removing of new devices

Discover multi paths
Allows management of device instances and to add devices IDs for fibre channel devices

DSM Install
Install/Uninstall vendor or 3rd party device specific modules (DSM) the DSM could come from dell etc to allow MPIO to work with their storage array

Configuration Snapshot
Allows capture a snapshot of the current MPIO configuration

iSCSI
Cheap simple storage solution.
Support for failover clustering
Required for guest failover clustering
Uses the existing IP network.
Can be a storage array or DAS on a server by using the MS iSCSI target.
You can download the MS iSCSI target here:
http://www.microsoft.com/en-us/download/details.aspx?id=19867
Many of the skills from working with networking, ethernet, switches etc transfer to iSCSI. Its just the SCSI protocol over IP.

iSCSI target
This sits on the device (SAN or server) where the storage is. SANs have the target build in.

iSCSI Initiator
Initiator connects to iSCSI target (the target must be configured already)
Should use a dedicated NIC (it's a good idea to rename NICs iSCSI1 etc)
Can use any iSCSI target

Setting up simple connections
iSCSI target
Setup the MS iSCSI target on a server
You will see the disks that are presented in iSCSI target (admin tools)

iSCSI Initiator
I find its best not to use the quick connect button (the wrong nic may be selected) use the connect button, select MS iSCSI initiator, the NIC to use and the IP of the target.

• Target: Create virtual disk
• Initiator: Request access to disks
• Target: Accept access request from initiators
• Initiators: Refresh configuration to check connection
• Initiators: Login to the target (enable automatic reconnections)
• Servers: Initialize, format and bring disks online (disk management)
• Now you can use these disks for your VMs or cluster

Executing iscsicli.exe commands
iscsicli is the CLI for iSCSI (needed for server core)
http://blogs.msdn.com/b/san/archive/2008/07/27/iscsi-initiator-command-line-reference-and-server-core-configuration.aspx


Configuring child/guest settings

Hyper-v manager
Hyper-v manager can be found in start -> administrative tools
You may need to install it via the RSAT rools if using a desktop.
You can connect to remote hyper-v hosts servers right click -> connecto to server

Hyper-v settings
Can be accessed by right clicking on a host and selecting hyper-v settings
Default locations for virtual hard disks, virtual machine config files
NUMA Spanning
User related settings like keyboard/mouse and credentials

Virtaul network manager
View the types of networks available
Add networks
View the mac address pool

Type 2 hypervisor
First type of hypervisor that came out
Guest OS VM's -> Hypervisor -> Host OS -> Hardware
Examples: Virtual PC & server, VMware workstation, KVM

Type 1 hyperviror
Guest OS VMs -> Hypervisor -> hardware
Examples: Hyper-v, Xen, VMware ESX

When you enable the hyper-v role on a server, the hypervisor imposes itself between the hardware and the OS. Kernel generally runs at ring 0 in the CPU. Hyper-v hypervisor runs at -1.

Classes of type 1 hypervisor
Monolithic (VMware ESX)
Management OS and Guest OS's run above the hypervisor
Drivers run at the hypervisor level
VMware is responsible for maintaining the drivers
Hardware compatibility list is critical when deploying ESX servers

Microkernel (Hyper-V)
Management OS partition, Guest OS child partition, virtualization stack and drivers run above the hypervisor
The drivers are the drivers supported by the OS
Drivers run within guests
Guest OS's access the drivers over the VMbus
Larger selection of drivers, easier to get updates

VM settings
Right click on a VM
BIOS
Memory
CPU
Hard drive
Ethernet
Com ports
Floppy disk drive
Must be IDE hard drive for boot
Other disks can be SCSI but the boot drive must be IDE
Integration Services (SCSI controllers, syntetic NICs)
Managemet options (what to do when VM is started / stopped)

Hard disk types

• Fixed disk - If you create an 80GB disk it consumes 80GB of disk space on your storage. Can provide better performance for applications with high levels of disk activity.
• Dynamically expanding disk - Grows as your data usage grows, if you create a 80GB disk is will only use 256KB and will grow as your data usage on the virtal disk grows.
• Differencing disk - Point it at a disk already created and say that will be the base disk. That disk will become readonly and the differencing disk and all changes are in the differencing disk. Same idea as snapshots. Can be useful if you create a base install OS and create a differencing disk. It can also reduse the total file size. Users can get a shared base disk but they get their own differencing disk. If there is an issue with the base disk it can affect many users.

The issue mentioned above is the reason why MS do not recommend to use snapshots as a method for backing up production systems.

Storage options for virtual machines
You can have 2 IDE controllers
Each controller can have 2 devices
That's a total of 4 IDE devices per VM




Managing and monitorin virtual enviornments

Ensuring high availability and recoverability

Performing migration

Configuring remote desktop Role Services Infrastructure

Active directory domain controllers can be virtualised but you cannot use snapshots, it is advised that you have some physical DCs also. Make sure you have full backsups with system state running. Make sure you have NTP setup and working. There seems to be a wide difference of opinion here. Some people say do not virtualise them others do, and some people mix virtual and physical.
Exchange can be virtualised but only certain versions on certain OS's are supported by MS so check that before attempting it. Also unified communications is not supported virtualised.

You get a kernel panic after doing a kernel update when using hyper-v integration services aka linuxic

It appears the hyper-v integration tools have no way to detect that the kernel has been updated. If you update your kernel the linuxic tools will be stuck on the old version and it will cause a kernel panic after you reboot. You need to select the previous kernel and boot into that to fix it.

Run "uname –r" to get the current kernel

Run the following commands to find which files are using uname to retrieve the kernel version

• cd /opt/linuxic
• grep –iR “uname –r” *

Edit the files with vi and replace "uname –r" with the following

shell echo "2.6.18-194.26.1.el5PAE" in the make files (your kernel version will be different)

echo "2.6.18-194.26.1.el5PAE" in the perl scripts  (your kernel version will be different)

Then run the script below. The script will detect the latest kernel (from /etc/grub.conf) version and upgate the linuxic files

 

date=`date +"%d%m%Y"`;

cp -R /opt/linuxic /opt/linuxic."$date";
cd /opt/linuxic;

grep -m2 title /etc/grub.conf | awk '{print $7}' | sed -e 's/(//g;s/)//g' | xargs > kernels.txt

while read newkernel oldkernel
do
grep -ilR "echo \"$oldkernel\"" * | xargs sed -i 's/'$oldkernel'/'$newkernel'/g';
done < 'kernels.txt'

 

  

If Linux Integration services are not installed then:

• Download the latest linux integration services for hyper-v
• Login to SVCMM
• Right click on virtual machine and select properties
• Select DVD/CD drive in the menu, on the right select the image file option
• Select the LinuxIC file, select the share option also
• Click OK and go back to console
• Run “mkdir /mnt/cdrom; mount /dev/cdrom /mnt/cdrom”
• Mkdir /opt/linuxic
• Cp /mnt/cdrom/* /opt/linuxic
• Cd /opt/linuxic
• Make
• Run the fix above and set up the script to update the kernel version in linuxic