Configuring Active / Active Failover in Cisco ASA

Active / Active failover method is supported only in Multiple Context mode. This Active / Active words here are kind of misleading as even though both the ASAs are forwarding traffic, they are not forwarding the same traffic, meaning 1 context is active for a 1 ASA while the other context is active for the other ASA only. So 1st of all we need to get contexts ready before the failover configuration.

Going to create 2 contexts, IT and SALES. 

SW1 and SW3 are carrying IT traffic while SW2 and SW4 is carrying SALES traffic.

Let's start by converting ASAs to multiple context mode by mode multiple command on ASAs and bringing up the interfaces from system context.

Configuring ASA1;

context IT

  allocate-interface Ethernet1 

  allocate-interface Ethernet3 

  config-url disk0:/IT.cfg

context SALES

  allocate-interface Ethernet2 

  allocate-interface Ethernet4 

  config-url disk0:/SALES.cfg

Now it's the time to configure Failover..

failover lan unit primary

failover lan interface FO Ethernet0

failover link FO Ethernet0

failover interface ip FO 10.10.10.1 255.255.255.0 standby 10.10.10.2

Configuring failover groups

failover group 1

  primary

  preempt

failover group 2

  secondary

  preempt

Join the contexts to failover groups

context IT

  join-failover-group 1

context SALES

  join-failover-group 2

And on Secondary ASA;

failover lan unit secondary

failover lan interface FO Ethernet0

failover link FO Ethernet0

failover interface ip FO 10.10.10.1 255.255.255.0 standby 10.10.10.2

finally hit failover on both the ASAs..

prompt command will change the system name accordingly to identify where you are in CLI.

use something like hostname context priority state  in system context.

You can configure all the other things from now on just like you would do in normal ASAs but remember to do them on currently active ASA for the context you need to configure. As an example, configure interface IP for SALES context should be done in ASA2 now.

Failover show commands like show failover will be helpful in config verification.

Configuring Active / Standby Failover in Cisco ASA

This is the most widely used failover configuration in Cisco ASA deployments. 

There is a special connectivity for control plane of this failover called failover link. In the diagram it is the connectivity between eth0 interfaces. 

There are 2 things to be shared between the ASAs, configuration and the stateful information. You can use 2 separate interfaces for this or you can use 1 for both which is the most popular deployment.

In this setup, I am also using 1 link for both the purposes. Also this link is used to monitor each other's availability by the ASAs.

We will make ASA1 the primary and ASA2 the secondary. 

On ASA1;

failover lan unit primary

failover lan interface FO Ethernet0

failover link FO Ethernet0

failover interface ip FO 10.10.10.1 255.255.255.0 standby 10.10.10.2

failover

On ASA2;

failover lan unit secondary

failover lan interface FO Ethernet0

failover link FO Ethernet0

failover interface ip FO 10.10.10.1 255.255.255.0 standby 10.10.10.2

failover

failover lan interface command is there to define the config syncing interface where as failover link command is to define the stateful information syncing interface. 'FO' is just a name given.

If you are defining another seperate interface for stateful information sync up, use the floowing command;

failover link SF g0/3

SF is just a name given.

Now you can configure anything on the primary active firewall and will be synced up to the secondary.

Additional Note:-

You can configure prompt hostname command to identify the current active device.

Following are some verification commands,

show failover

show failover history

show failover state

failover exec standby <any exec level command> can be used in configure mode of primary ASA to execute commands in standby unit.

Security Contexts in Cisco ASA

Security Contexts are just virtual firewalls within same physical ASA hardware box. 

Licenses must be there to begin with, verify using show license features command and current mode using show mode command. show mode will display current mode as single mode.

Start with clearing configuration by config mode command clear configure all.

Change the mode to Multiple Context by the command mode multiple. ASA will go through a reboot.

Once you are in the Multiple Context mode, you are booted by default into a default context called system context. This context is there just to define your other contexts.

Also there is another default context called admin context. It is there just to access system context but you can use it for your normal purposes as well.

You can make your own context to be the admin context later. The * mark is there to identify the admin context.

Now I am creating 2 contexts called IT and SALES and allocate interfaces. This is done in system context.

context IT

  allocate-interface e0

  allocate-interface e1

  config-url flash:IT.cfg

context FINANCE

  allocate-interface e2

  allocate-interface e3

  config-url flash:SALES.cfg

Note that a config file needed to be created because a separate config file must be maintained for each Context.

Changing Security Mode can be done using changeto context command. You can change to any context via admin context and you can change to admin context via other non admin contexts. But you cannot change to non admin contexts or system context from non admin contexts.

After changing to Contexts you can configure just as you would do in a normal firewall. Now you have 2 virtual ASA firewalls in same box.

Let's change IT context as the admin context by the command admin-context IT.

Resource Allocation

This can be done through Class Maps in the following simple way.

class GOLD

 limit-resource conns 100000

 limit-resource Xlates 50000

class SILVER

 limit-resource conns 75000

 limit-resource Xlates 25000

Classes were defined and values were set for most resource utilizing features in ASA, connections and NAT translations.

Now let's assign those classes to Contexts,

context SALES

  member GOLD

context FINANCE

  member SILVER

If you want you can allocate same interface to other contexts. As an example Ethernet 0 is allocated to both IT context and SALES contexts.

write memory all will save all configuration made on all contexts at once.

Clustering Cisco ASAs using Spanned Mode

This feature came to ASA after version 9.x. This is the only method of configuring redundancy with more than 2 Cisco ASAs if required.

There are 2 methods in ASA clustering, one is Individual Interface Mode and the Other one is Spanned Mode. This post is about most widely used Spanned Mode.  

CCL is the Cluster Control Link which governs the configuration sharing within ASAs.

SW1's VLAN 1 interface IP is 192.168.10.1 and SW2's VLAN 1 interface IP is 203.115.50.1. INSIDE and OUTSIDE interfaces per ASA is as shown in the diagram and port channels are to be configured.

Within the cluster, 1 ASA must act as the Master and all others are Slaves. Here I am configuring ASA1 as the master,

cluster interface-mode spanned

cluster group PERIMETER

  local-unit ASA1

  cluster-interface e0 ip 10.10.10.1 255.255.255.0

  priority 5

  enable

On ASA2;

cluster interface-mode spanned

cluster group PERIMETER

  local-unit ASA2

  cluster-interface e0 ip 10.10.10.2 255.255.255.0

  priority 10

  enable

show cluster info and show cluster history are verification commands.

Cluster configuration part is done, but in order to flow traffic with load balancing, port-channels must needed to be configured.

These switch port channels are the real load balancers here. Let's do this with LACP configuration.

On SW1;

interface range e0/0-1

  channel-group 1 mode active

On SW2;

interface range e0/0-1

  channel-group 2 mode active

On ASA1; (configuration is now done on Master box only, Slaves will replicate)

interface e1

  channel-group 1 mode active

  no shut

interface e2

  channel-group 2 mode active

  no shut

int po1

  port-channel span-cluster

  mac-address aaaa.bbbb.1111

  nameif INSIDE

  security-level 100

  ip address 192.168.10.1 255.255.255.0

int po2

  port-channel span-cluster

  mac-address aaaa.bbbb.2222

  nameif OUTSIDE

  security-level 0

  ip address 203.115.50.1 255.255.255.0

Additional Note:-

If one session entered from ASA1 and returned from ASA2, ASA2 will forward the returning traffic to ASA1 via CCL. So unlike in other redundant methods like Active/Standby or Active/Active in ASA, CCL link is used for data traffic forwarding as well.

Troubleshooting Tips for NAT Related Issues in Cisco ASA

Here are some tips I found while troubleshooting ASA NAT issues. 

Verifying Configuration

First thing is that the Auto-NAT is not showing in the show run output of Cisco ASA under object network configuration which is used to configure it.

As an example, I did the following Static Auto NAT config;

Well, it should be there in the running config, but not where I configured..

It is there in the running config, but in a different place, Pipe nat or scroll all the config, you will find it.

View NAT Translation Information

There are few commands to see NAT translations in Cisco ASA.

show nat command will only be a summary, better command will be show nat detail, well for me show nat is a useless command as show nat detail covers it all.

Untranslate hits means in the above example, the traffic is coming from OUTSIDE. Translate hits means coming from DMZ.

Dynamic NAT & PAT Pool Related Issues

show nat pool command can be used to tackle PAT issues like PAT pool exhaustion issues.

Allocated 1 means, one port from 1024-65535 pool is used.

Another useful command is show xlate, specially for verifying PAT translations.

You can bind show xlate command with a local address or a global address for more granular output.

NAT Divert Issues.

NAT is used to figure out the Egress interface for a traffic flow before even the route lookups/PBRs.

So if the interface pair is misconfigured, the entire flow will be forwarded to a wrong interface dropping the traffic.

object network WEB

   host 192.168.10.10

   nat (dmz,outside) source static 203.115.50.10

Above configuration tells the ASA that, when 192.168.10.10 tries to move from DMZ to OUTSIDE, it should be translated to 203.115.50.10 and when something hits 203.115.50.10 on OUTSIDE interface, it should be untranslated back to 192.168.10.10 and forward to DMZ interface. 

If the DMZ interface is misconfigured here, the traffic is dropped.

show nat divert-table can be useful to find such issues.

NAT Concepts in Cisco Firewalls (ASA & FTD)

There are many misconceptions about NAT out there. I would like to write something I learned, figured out about NAT concepts in Cisco Firewalls. 

Source NAT means Your NAT

In Cisco world Source NAT word is misleading. "Source" here really means "Yours". 

You are configuring your Firewall to translate your IP addresses which is in the LAN throughout the entire session for both forward and return traffics. This Source NAT (aka Yours IP translation) will be used 99% in real world.

Source NAT really means Your IP is getting NATted

On the other hand "Destination NAT" which really means translating remote address will be needed very rarely. It is not really used anymore in real world. Think of an example where you need to translate remote IP for your purposes.

Auto NAT vs Manual NAT

In Cisco Firewalls, you can perform NAT via 2 methods. Auto NAT which is also called Object NAT is something you configure within a network object itself where Manual NAT which is also called Policy NAT / Twice NAT is performed based on a traffic criteria (Source/Destination IP).

Types of NAT

Following are the different types of NAT which are used for different requirements.

Static NAT

Static PAT (Port Forwarding)

Dynamic NAT

Dynamic PAT

We will take a look at how the above NAT types can be configured using Auto-NAT method as it s the simplest method in ASA.

Static NAT

Works bi-directional by default. Generally used to translate public servers statically. So this is used by clients to access your servers. The flow is mostly initiated from OUTSIDE towards DMZ in real world.

Ex:- 

Map DMZ Web server IP 192.168.50.10 to Outside public IP 203.115.50.5

Map DMZ Email server IP 192.168.50.20 to Outside public IP 203.115.50.6

object network WEB

   host 192.168.50.10

   nat (dmz,outside) static 203.115.50.5

object network EMAIL

   host 192.168.50.20

   nat (dmz,outside) static 203.115.50.6

Static PAT (Port Forwarding)

Same kind of use case like Static NAT instead used when you have only 1 public IP to map several private IPs.

Ex:- 

Map DMZ Web server IP 192.168.50.10 to Outside public IP 203.115.50.5

Map DMZ Email server IP 192.168.50.20 to Outside public IP 203.115.50.6

Map DMZ DNS server IP 192.168.50.30 to Outside public IP 203.115.50.7

object network WEB

   host 192.168.50.10

   nat (dmz,outside) static 203.115.50.5 service tcp 80 80

object network EMAIL

   host 192.168.50.20

   nat (dmz,outside) static 203.115.50.5 service tcp 25 25

object network WEB

   host 192.168.50.30

   nat (dmz,outside) static 203.115.50.5 service udp 53 53

You can change the port numbers if you want to give server access from a different port. 1st port number is the real service port, 2nd one is the mapped port.

Dynamic NAT

This is used to map a pool of private IPs to a pool of public IPs dynamically. IP assignment will happen randomly.

Define the range public IPs

object network PUBLIC-POOL

   range 203.115.50.50 203.115.50.100

Define the internal IP range

object network INTERNAL

   subnet 192.168.10.0 255.255.255.0

   nat (inside,outside) dynamic PUBLIC-POOL

Dynamic PAT

Generally configured for internet access to internal hosts. Will be overloaded to an interface, single IP or even to a pool. If overloaded to a single IP, 64000 simultaneous connections will be provided. If that is not enough, you can configure a PAT pool so that the amount of simultaneous translations will be multiplied by the number of IP addresses in the pool

object network PUBLIC-IP

   host 203.115.50.10

object network INTERNAL-IPS

   subnet 192.168.10.0 255.255.255.0

   nat (inside,outside) dynamic PUBLIC-IP

or 

object network INTERNAL-IPS

   subnet 192.168.10.0 255.255.255.0

   nat (inside,outside) dynamic interface 

Configure using Manual NAT Method

As an example,

You need to translate 192.168.10.10 to 203.115.50.5 when 192.168.10.10 is going to access 203.115.50.10 and at the same time translate 192.168.10.10 to 203.115.50.6 when 192.168.10.10 is going to access 203.115.50.20

Here, the source IP and destination IP of the packet must match to perform the NAT. This can be achieved by Manual NAT via the following configuration.

Polices are defined after objects created.

object network S

   host 192.168.10.10

object network X1

   host 203.115.50.5

object network X2

   host 203.115.50.6

object network D1

   host 203.115.50.10

object network D2

   host 203.115.50.20

nat (inside,outside) source static S1 X1 destination D1 D1

nat (inside,outside) source static S1 X2 destination D2 D2

You can translate destination service by this too and also you can do this uni-directionally even for static NATs if you want. You can perform any type of NAT which you could do with Auto NAT including Static NAT, Static PAT, Dynamic NAT, Dynamic PAT, Destination NAT etc and even put a dynamic policy to be executed 1st by changing the policy order with simple modification to the command.

NAT Order of Operation

There are 3 Sections in ASA configuration. 

Section 1 will be executed 1st while section 3 will be the last. Section 1 is comprised of Manual NAT and Section 2 is comprised with Auto NAT. Section 3 is for Manual NATs again if you want to execute some of them after Auto NAT rules.

Within Section 2, Static NAT rules will be executed before Dynamic NAT rules.

Manual NAT will be executed by policy order which you can change but Auto NAT will be executed with the longest match 1st theory

show nat command will display the NAT table in order.

Troubleshooting Packet Drops in Cisco ASA for Non-VPN Traffic

This packet flow is something I created based on the labs I did to understand how ASA works. This will help you to troubleshoot packet drops in ASA for non VPN traffic.

RX Ring Buffer Overrun

RX Ring is the software input queue which holds the packets for processing. If the RX Ring is filled, the traffic will be dropped before even processing. This can be seen by show interface <interface-name> command like the following.

Overrun counter will indicate how many packets were dropped due to RX Ring exhaustion. Here the "hardware (0/0)" is just because I use a ASAv for lab, but for real hardware it will show some value.

Existing Connection Check

Connection table can be viewed from show conn command.

Above snap shows an existing telnet connection originated from INSIDE interface source 192.168.10.10. If a packet which belongs to same traffic comes to the INSIDE interface, ASA will forward it to directly to NAT Translations.

If the packet is a TCP SYN or a UDP, it will be forwarded to NAT Untranslations and move on.

If there is no existing connection and if the packet is a non-TCP SYN, it will be dropped with a syslog similar to the following:

ASA-6-106015: Deny TCP (no connection) from 192.168.10.10 /32256 to 203.115.50.10 /23 flags PSH ACK on interface INSIDE

NAT Untranslation 

Packet it untranslated before ACL check; this is the reason we put ACLs to your private addresses instead of your public addresses.

The packet drop can occur due to the destination address translations and diverting to the wrong interface.

ACL Check

First packet in flow is checked against ACLs. This is why you can see fewer hits in show access-list command.

Denied packets are dropped and logged like the following:

%ASA-4-106023: Deny tcp src OUTSIDE:203.115.50.10/14379 dst INSIDE:192.168.10.10/24 by access-group "OUTLIST"

Allowed packets will be logged like the following:

%ASA-7-609001: Built local-host OUTSIDE:203.115.50.10

%ASA-7-609001: Built local-host INSIDE:192.168.10.10

%ASA-6-302013: Built inbound TCP connection 87 for OUTSIDE:203.115.50.10/18653 (203.115.50.10/18653) to INSIDE:192.168.10.10/23 (192.168.10.10/23)

NAT Translation

This is the place to run the NAT translation rules. This is done before routing so that the route lookups can be performed for translated addresses.

I don't see any reason to the traffic to be dropped here. Cisco ASA software older than version 8.2 needed the NAT to be performed mandatorily but now it is not the case. So for older ASAs, there could have been traffic drops here, but not now. 

L3 Route Lookup

Egress interface is determined 1st by NAT rules or existing connection entries. If couldn't find a macth, global routing table will be consulted.

If the L3 route lookup failed, you will get a syslog similar to following.

%ASA-6-110003: Routing failed to locate next hop for TCP from INSIDE 192.168.10.10/18653 to OUTSIDE:203.115.50.20/23

L2 Address Lookup

If layer 2 lookup fails, you will not see a syslog. show arp will not show required entry and debug arp will indicate if we are not receiving ARP reply. 

TX Ring Buffer Underrun

Same as the Buffer Overrun, if the output queue if full, the buffer underrun counter will increase and packets will be dropped.

Clearing the Confusion of CLI & GUI Admins of Cisco ISE

Cisco ISE is an application developed on top of Cisco ADE-OS (Application Development Engine).  Because of that, it cannot be accessed through it's CLI. When we log in to the so called ISE CLI, it's actually the ADE-OS CLI where we can configure interface IPs, routing etc. Also we use that interface  to see the status of the Cisco ISE application by the following famous command, show application status ise.

So the CLI admin account has nothing to do with the ISE application admin account.

This CLI admin account password is asked to change during the initial login to the device. If you want to change it later on, use the following command,

password

Confusion comes with the account name "admin" is being same for the ISE default application but with a password you need to set additionally with the following command,

application reset-passwd ise admin

Use this admin account/password to log in to the ISE through Web UI.