Cisco VPN with OpenSSL Certificates

Having realised the impracticality of a DMVPN with 10+ spokes (and dynamic spoke-to-spoke shortcut tunnels) using pre-shared keys for authentication, I decided to look into using certificates. This is something I had never done, and it appears this can be pretty straightforward, especially using the Cisco IOS CA server.

However... for whatever reason, I decided I wanted to use OpenSSL to do all my certificate issuing. So here is how I did it.

The basis of the way it works is simple: every router has a certificate, which it uses to authenticate to every other router and you don't need a pre-shared key for each router-to-router tunnel. This is all validated by having all the certificates issued by a single central Certification Authority. So, if Router A and Router B try to establish a VPN and both sides have a certificate signed by the same CA, they can use the keys associated with these certificates to establish the tunnel. *Much* easier than a lot of PSKs.

I'll illustrate this with a basic VPN setup, rather than DMVPN. I'll assume that you already understand basic site-to-site IPSEC tunnels, but here's roughly the configuration I used. Two routers with 10.0.0.0/30 for WAN addressing, and I'll use 192.168.0.0/30 for the tunnel addressing:

crypto isakmp policy 10
 encr 3des
 hash md5
 group 2
 authentication rsa-sig
 lifetime 28800

crypto isakmp identity hostname

crypto ipsec transform-set Basic esp-3des esp-md5-hmac
 mode transport

crypto ipsec profile SecTunnel
 set transform-set Basic
 
interface Tunnel0
 ip address 192.168.0.1 255.255.255.252
 tunnel source FastEthernet0/0
 tunnel destination 10.0.0.2
 tunnel protection ipsec profile SecTunnel

interface FastEthernet0/0
 ip address 10.0.0.1 255.255.255.252

This is pretty straightforward, although note the use of "authentication rsa-sig" in the ISAKMP policy! You also have to make absolutely sure your clocks are synchronised. I recommend running NTP. This is something often overlooked in a basic Cisco lab.

So, now we need to generate a CA certificate. Unlike for web servers and the like, you don't need to generate a CA and then an Intermediate certificate - you just generate a single certificate that will then sign all router certificates.

openssl req -x509 -newkey rsa:4096 -keyout vpnca.key -days 365 -out vpnca.crt

Enter a password when prompted, you'll need it later. Also fill in basic details. In this instance they don't matter too much.

$ openssl req -x509 -newkey rsa:4096 -keyout vpnca.key -days 365 -out vpnca.crt
Generating a 4096 bit RSA private key
...................................++
writing new private key to 'vpnca.key'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:GB
State or Province Name (full name) [Some-State]:London
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Useful Company Ltd
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:testca.mydomain.com
Email Address []:
This will create your key (vpnca.key) and your certificate (vpnca.crt).

Now you should create your keys on the router:

ip domain-name <your domain>
crypto key generate rsa modulus 2048 general-keys

MyRouter(config)#crypto key generate rsa modulus 1024 general-keys
The name for the keys will be: MyRouter.mydomain.com

% The key modulus size is 1024 bits
% Generating 1024 bit RSA keys, keys will be non-exportable...[OK]
This has created the keys we need. Next you need to create and authenticate your CA Trustpoint on the router:

Create:
MyRouter(config)#crypto ca trustpoint CARoot
MyRouter(ca-trustpoint)#enrollment terminal
MyRouter(ca-trustpoint)#revocation-check none
Authenticate. Paste the contents of vpnca.crt in when prompted, minus the header and footer:

MyRouter(config)#crypto ca authenticate CARoot

Enter the base 64 encoded CA certificate.
End with a blank line or the word "quit" on a line by itself

MIIFhTCCA22gAwIBAgIJAKdOkXFRpOkLMA0GCSqGSIb3DQEBCwUAMFkxCzAJBgNV
--- snip ---
RmApUW1+BZYoab6VsgRrabfPx5e5Gxp7EA==

Certificate has the following attributes:
       Fingerprint MD5: 1B63EEB6 2B9A75C1 1115BF2C 5630C5A9
      Fingerprint SHA1: 3D90F95A 89B608B2 695A2FAD CECD5857 6BD02636

% Do you accept this certificate? [yes/no]: yes
Trustpoint CA certificate accepted.
% Certificate successfully imported

This will set up your trustpoint. Now you should generate a Certificate Signing Request:

MyRouter(config)#crypto ca enroll CARoot
% Start certificate enrollment ..

% The subject name in the certificate will include: MyRouter.mydomain.com
% Include the router serial number in the subject name? [yes/no]: no
% Include an IP address in the subject name? [no]: no
Display Certificate Request to terminal? [yes/no]: yes
Certificate Request follows:

MIIBhjCB8AIBADAmMSQwIgYJKoZIhvcNAQkCFhVNeVJvdXRlci5teWRvbWFpbi5j
---snip---
NMVFrfGZM5zIDg==

---End - This line not part of the certificate request---
Copy the Base64-encoded CSR into your clipboard and then paste into a file on your server for SSL, call it "<hostname>.csr". Make sure it has the appropriate PEM header/footer:
-----BEGIN CERTIFICATE REQUEST-----
MIIBhjCB8AIBADAmMSQwIgYJKoZIhvcNAQkCFhVNeVJvdXRlci5teWRvbWFpbi5j
---snip---
NMVFrfGZM5zIDg==
-----END CERTIFICATE REQUEST-----

Now we sign the certificate - enter the CA key password from earlier when prompted:

$ openssl x509 -req -days 365 -CA vpnca.crt -CAkey vpnca.key -CAcreateserial -in MyRouter.csr -out MyRouter.crt
Signature ok
subject=/unstructuredName=MyRouter.mydomain.com
Getting CA Private Key
Enter pass phrase for vpnca.key:

Now we look inside the MyRouter.crt file produced and copy the Base64 portion of the certificate. Then we can go back to our router and import it:

MyRouter(config)#crypto ca import CARoot certificate

Enter the base 64 encoded certificate.
End with a blank line or the word "quit" on a line by itself

MIIDdzCCAV8CCQD37zWqJTK0NjANBgkqhkiG9w0BAQsFADBZMQswCQYDVQQGEwJH
---snip---
heaHO96lAMJ6QnjUMtcVpi0UwEHTeCPWC/96

% Router Certificate successfully imported


Your certificate is now imported!

If you now repeat this exact same process on your other router, your VPN should be ready. It may not come straight up, you may need to clear SAs to have it reinitialise. Congratulations, you are now using RSA Signature Authentication instead of Pre-Shared Key!

Uptime... not recommended

Cisco Internetwork Operating System Software
IOS (tm) C3750 Software (C3750-I9-M), Version 12.2(20)SE4, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2005 by cisco Systems, Inc.
Compiled Sun 09-Jan-05 00:09 by antonino
Image text-base: 0x00003000, data-base: 0x0099748C

ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.1(14r)EA1a, RELEASE SOFTWARE (fc1)

3750-sw1 uptime is 10 years, 6 weeks, 6 days, 2 minutes
System returned to ROM by power-on
System restarted at 12:16:53 UTC Wed Sep 28 2005

Definitely needs an upgrade at some point....

Cisco Packet Captures and Perl Net::Pcap::Easy

So today I once again was dealing with a packet capture taken from a Cisco router using the "monitor capture buffer/monitor capture point" commands. I wanted to build a Perl script to do some basic parsing and sanity checking of the data.

Unfortunately, my simple pcap library of choice in Perl, Net::Pcap::Easy, doesn't like Cisco packet captures:

ERROR Unhandled data link type: RAW at /usr/local/share/perl/5.18.2/Net/Pcap/Easy.pm line 122.

This is very easy to fix, although a bit ugly. The lines of code in Net::Pcap::Easy which handle frames with no Ethernet header is shown below:

        # For non-ethernet data link types, construct a
        # fake ethernet header from the data available.
        my ($ether, $type);
        if ($linktype == Net::Pcap::DLT_EN10MB) {
            $ether = NetPacket::Ethernet->decode($packet);
            $type = $ether->{type};

        } elsif ($linktype == Net::Pcap::DLT_LINUX_SLL) {
            use bytes;
            $type = unpack("n", substr($packet, 2+2+2+8, 2));
            $ether = NetPacket::Ethernet->decode(
                    pack("h24 n", "0" x 24, $type) . substr($packet, 16));
            no bytes;

        } else {
            die "ERROR Unhandled data link type: " .
                Net::Pcap::datalink_val_to_name($linktype);
        }

As you can see, it generates the error for the unhandled packet type. Since I don't have the data at the Ethernet level, my tools wouldn't be using that data, so literally anything will do, as long as it doesn't crash the higher level packet dissectors.

So... I added a really simple extra option to the if statement:

        } elsif ($linktype == Net::Pcap::DLT_RAW) {
                #Steve's attempt to handle RAW
            $ether = NetPacket::Ethernet->decode(pack('H*','0001020304050005040302010800').$packet);
            $type = $ether->{type};


Literally, this adds a header that says data is from 00:01:02:03:04:05 and to 00:05:04:03:02:01, and is of type 0x0800, IP. This fix worked first time, which surprised me a little!

Hope that helps!

Cisco ezVPN with FreeRADIUS

It's been a while since I posted, but I have come across another comparatively poorly-documented part of the networking world. I struggled to find the answers to this, so here is a writeup that may help!

So, the idea is that I want to use ezVPN for remote sites dialling into a central point. However, rather than having my users configured locally on the hub router, I want to use RADIUS. However, I don't want to pay to use Cisco ACS server or Windows Server, so FreeRADIUS is the obvious open-source option.

Hub Router Configuration

The configuration on the router is fairly simple and well-documented. You will need an appropriate IOS with appropriate crypto functionality. In my case, I am using a Cisco CSR1000V in the cloud to test this. Below are the relevant parts of my configuration:

aaa group server radius VPN-RADIUS
 server-private <RADIUS Server IP> key <RADIUS Shared Secret>
 server-private <RADIUS Server IP> auth-port 1812
 server-private <RADIUS Server IP> acct-port 1813
 ip vrf forwarding Mgmt-intf
 ip radius source-interface GigabitEthernet0

!
aaa authentication login VPN-Users group VPN-RADIUS
aaa authorization network VPN-Users group VPN-RADIUS
!
crypto isakmp policy 10
 encr 3des
 authentication pre-share
 group 2
!
crypto isakmp policy 20
 encr aes
 authentication pre-share
 group 2
!
crypto isakmp client configuration group RemoteUsers
 key <pre-shared-key>
 acl 110
 save-password
!
crypto isakmp profile VPN
   match identity group RemoteUsers
   client authentication list VPN-Users
   isakmp authorization list VPN-Users
   client configuration address respond
   virtual-template 2

!
crypto ipsec transform-set ezVPN-Transform esp-3des esp-sha-hmac
 mode tunnel
!
crypto ipsec profile ezVPN
 set transform-set ezVPN-Transform
 set isakmp-profile VPN
!
crypto dynamic-map Remote-Map 10
 set transform-set ezVPN-Transform
 set isakmp-profile VPN
 reverse-route
!
crypto map VPN 65535 ipsec-isakmp dynamic Remote-Map
!
interface Virtual-Template2 type tunnel
 ip unnumbered GigabitEthernet2
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile ezVPN
!
access-list 110 permit ip 172.16.0.0 0.15.255.255 any
!
interface Gigabit1
 ip address <WAN Address> <Netmask>
 crypto map VPN


This is a lot of configuration, and some of this may not be necessary. I need to go back and clean it out. In particular, the ISAKMP Client Configuration group will actually be provided by RADIUS, so likely we can remove this part.

The majority of the above configuration is easily found with some Google searches and is well documented. However, the next part, the RADIUS part, is not.

FreeRADIUS Configuration

First of all you need to get a server set up and FreeRADIUS installed. I won't elaborate further, since this is also well documented elsewhere. For reference, in my setup I used Ubuntu Server 12.04LTS.

clients.conf

The clients.conf file is simply used to identify expected RADIUS clients that may talk to us, and set the shared-secret for them. Below is a sample file:
client <Cisco device RADIUS source IP> {
        secret          = <Shared Secret>
        nastype         = cisco
        shortname       = ezVPN-Server
}

This now enables our Cisco router to talk to the RADIUS server. Check that you know what the source-address is on the Cisco - it can be defined within the Server Group config.

 

 users

The users file defines the different users available. This can also be done with an SQL database. However, the SQL part is well documented so I will not cover it here. First we have to set up the VPN Group user definition - this is the main part that I struggled with. Here is my example config:
RemoteUsers  Cleartext-Password := "cisco"
        Tunnel-Type = "ESP",
        Tunnel-Password = "<pre-shared-key>",
        Cisco-AVPair := "ipsec:tunnel-type=ESP",
        Cisco-AVPair += "ipsec:key-exchange=IKE",
        Cisco-AVPair += "ipsec:save-password=1",
        Cisco-AVPair += "ipsec:inacl=110"
So, to explain some of the fields:
- "RemoteUsers" - this will be whatever your VPN group is
- Cleartext-Password := "cisco" - This is required - the router sends a request with the group name and the password of "cisco".
- Tunnel-Type - This is another required item.
- Tunnel-Password - This should be your actual VPN group key.

Now we move on to the "Cisco-AVPair" entries. These are Cisco Attribute-Value pairs. So basically this is information that gets passed back to the router as parameters for the connection.

• ipsec:tunnel-type=ESP - Required - stating we will use ESP
• ipsec:key-exchange=IKE - Required - stating we will use IKE
• ipsec:save-password=1 - Optional - Since my setup is for branch-offices, I want to have the password pre-configured in the router without people needing to keep entering it. Therefore I have to permit the password to be saved locally.
• ipsec:inacl=110 - Optional - This is for split-tunnel VPNs. It specifies which ACL on my hub router should be used to define the split networking list.

You may have already realised, this is exactly the same data as can be included statically on the Cisco router, as below:
crypto isakmp client configuration group RemoteUsers
 key <pre-shared-key>    ! Tunnel-Password = "<pre-shared-key>"
 acl 110                 ! Cisco-AVPair += "ipsec:inacl=110"
 save-password           ! Cisco-AVPair += "ipsec:save-password=1"

For further reading about Cisco Attribute-Value pairs, have a look at this page: https://supportforums.cisco.com/document/58091/exploring-remote-access-vpn-easy-vpn-cisco-router-cisco-secure-access-control-server

For our actual VPN users, the entries are much simpler. We just need to add user/password pairs to the users file:

User1     Cleartext-Password := "<password>"

Once this is done, you should have a working ezVPN setup with FreeRADIUS server.

IGMP Query Solicitation

Once again, I've come back to IGMP/Multicast/STP. This is a topic that, while reasonably well documented, can be complex to find explanations of.

In this case, I saw a network where a switch was running many (lots, like 300+) VLANs and we were regularly seeing traffic from the same MAC in all of those VLANs. This behaviour was intruiging, so I did some digging.

The switch in question was an Allied Telesyn x610 switch. Looking at the packets in Wireshark, it turns out they all originate from a switch and they are IGMP Query Solicitation. So...

What is IGMP Query Solicitation?

So you remember back in one of my earliest posts, where we had IGMP and Spanning-Tree interacting in such a way that we had floods of traffic on the network? This is very much related. In that scenario we were seeing that our switches were flooding traffic to all ports whenever there was a Spanning-Tree topology change. This "IGMP Query Solicitation" is another behaviour that can occur at exactly the same time, when a topology change is seen. As well as flooding the multicast traffic to ensure it reaches the correct destination, the switch can also send an "IGMP Query Solicitation" message to everybody. This message essentially "resets" IGMP by prompting the querier to immediately send out a General Query. This, in turn, means that all clients will re-affirm their interest in the appropriate multicasting groups by sending a Membership Report. Therefore, the IGMP snoopers (switches) will then be able to rebuild their snooping databases and once again send traffic to all the right places. So it provides a way of restoring order to your network after Spanning-Tree announces a Topology Change.

So do I want this behavior? How do I turn it off?

Yes. It's good. It helps your network to get back to normal after any changes in topology. In our case there was a device that was not coping with seeing the same MAC in 300+ VLANs at the same time. However, there may be legitimate reasons to turn this on or off. In the Cisco switches I tested, this was disabled by default. In order for it to function, it required:

• An IGMP querier active in a VLAN (any VLAN with no querier got no query solicitation)
• IGMP Query Solicitation enabled ("ip igmp snooping tcn query solicit")

However, in the Allied Telesyn x610 switch that I was using, this behaviour is enabled by default. In addition, IGMP snooping is enabled by default. This is sort of good because if you don't need it, it is unlikely to cause harm, yet not having it when you need it can cause your network to flood.

In the case of the AT switches, there are two options, the default is that only the STP root-bridge will send the query solicitation packets. You can turn that off, or you can optionally enable it for all switches, not just the root. Commands are:

(no) ip igmp snooping tcn query solicit root
(no) ip igmp snooping tcn query solicit

The variation in implementations and defaults across switch manufacturers is interesting. Allied Telesyn seem to be protecting people, whereas Cisco are assuming you should know what you're doing at least a little bit!

As always, I hope this has been of use to somebody!

Cisco NAT Payload Inspection

I came across an interesting problem today with our public-facing DNS server. For various reasons that I can't actually justify, we have some DNS A records with private IP addresses in our main public-facing NS. Somebody noticed today that those weren't working any more and that DNS requests for these addresses would time out. For example, with apologies for the potentially confusing anonymisation of my company's names and addresses:

steve@linuxbox:~$ dig myname.mydomain.com @ns1.mydomain.com

; <<>> DiG 9.8.1-P1 <<>> myname.mydomain.com @ns1.mydomain.com
;; global options: +cmd
;; connection timed out; no servers could be reached

Whereas, a request for a name with a public IP would work:

steve@linuxbox:~$ dig mypublicname.mydomain.com @ns1.mydomain.com
 
; <<>> DiG 9.8.1-P1 <<>> mypublicname.mydomain.com @ns1.mydomain.com
;; global options: +cmd
;; Got answer:
--snip--
mypublicname.mydomain.com.       86400   IN      A       4.2.2.2

So... what is happening?

As I often do, my first step into looking was a packet capture. I captured whilst doing first a "public" request and then a "private" request. The result was as below:

So, for a "normal", "public" request, the response is fine. For the "private" request, the server replies correctly, but our NAT router is sending an ICMP Destination Unreachable and the response never reaches the client. So, this immediately says that the router is doing some level of inspection at Layer 5 or above and electing to drop our DNS response, which we don't want.

A quick bit of Googling later and I found this incredibly helpful article: https://plone.lucidsolutions.co.nz/networking/cisco/ios/a-workaround-for-nat-rewriting-dns-packets

This makes complete sense. So, I added the no-payload option to my NAT statement and immediately, all my DNS requests started working!

Samsung Galaxy Player 50 (YP-G50EW) Hard Reset

So, I found it very difficult to find instructions for doing a hard reset on an early model Samsung Galaxy Player (model YP-G50EW). There are various key codes that are reported, but the one I found to work is:

Hold down Volume Up, Volume Down and the Home/Menu button, then hold Power until the Samsung logo appears. The release the power button, but keep the other buttons held down.

Now you are in the Android Recovery menu and you can continue with normal reset operations!

As always, hope this helps somebody!