Friday, 12 September 2014

Cisco ezVPN with FreeRADIUS

It's been a while since I posted, but I have come across another comparatively poorly-documented part of the networking world. I struggled to find the answers to this, so here is a writeup that may help!

So, the idea is that I want to use ezVPN for remote sites dialling into a central point. However, rather than having my users configured locally on the hub router, I want to use RADIUS. However, I don't want to pay to use Cisco ACS server or Windows Server, so FreeRADIUS is the obvious open-source option.

Hub Router Configuration

The configuration on the router is fairly simple and well-documented. You will need an appropriate IOS with appropriate crypto functionality. In my case, I am using a Cisco CSR1000V in the cloud to test this. Below are the relevant parts of my configuration:

aaa group server radius VPN-RADIUS
 server-private <RADIUS Server IP> key <RADIUS Shared Secret>
 server-private
<RADIUS Server IP> auth-port 1812
 server-private
<RADIUS Server IP> acct-port 1813
 ip vrf forwarding Mgmt-intf
 ip radius source-interface GigabitEthernet0


!
aaa authentication login VPN-Users group VPN-RADIUS
aaa authorization network VPN-Users group VPN-RADIUS
!
crypto isakmp policy 10
 encr 3des
 authentication pre-share
 group 2
!
crypto isakmp policy 20
 encr aes
 authentication pre-share
 group 2

!
crypto isakmp client configuration group RemoteUsers
 key <pre-shared-key>
 acl 110
 save-password

!
crypto isakmp profile VPN
   match identity group RemoteUsers

   client authentication list VPN-Users
   isakmp authorization list VPN-Users
   client configuration address respond
   virtual-template 2


!
crypto ipsec transform-set ezVPN-Transform esp-3des esp-sha-hmac
 mode tunnel
!
crypto ipsec profile ezVPN
 set transform-set ezVPN-Transform
 set isakmp-profile VPN
!

crypto dynamic-map Remote-Map 10
 set transform-set ezVPN-Transform
 set isakmp-profile VPN
 reverse-route
!

crypto map VPN 65535 ipsec-isakmp dynamic Remote-Map
!
interface Virtual-Template2 type tunnel
 ip unnumbered GigabitEthernet2
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile ezVPN

!
access-list 110 permit ip 172.16.0.0 0.15.255.255 any
!
interface Gigabit1
 ip address <WAN Address> <Netmask>
 crypto map VPN


This is a lot of configuration, and some of this may not be necessary. I need to go back and clean it out. In particular, the ISAKMP Client Configuration group will actually be provided by RADIUS, so likely we can remove this part.

The majority of the above configuration is easily found with some Google searches and is well documented. However, the next part, the RADIUS part, is not.


FreeRADIUS Configuration

First of all you need to get a server set up and FreeRADIUS installed. I won't elaborate further, since this is also well documented elsewhere. For reference, in my setup I used Ubuntu Server 12.04LTS.

clients.conf

The clients.conf file is simply used to identify expected RADIUS clients that may talk to us, and set the shared-secret for them. Below is a sample file:
client <Cisco device RADIUS source IP> {
        secret          = <Shared Secret>

        nastype         = cisco
        shortname       = ezVPN-Server

}

This now enables our Cisco router to talk to the RADIUS server. Check that you know what the source-address is on the Cisco - it can be defined within the Server Group config.

 

 users


The users file defines the different users available. This can also be done with an SQL database. However, the SQL part is well documented so I will not cover it here. First we have to set up the VPN Group user definition - this is the main part that I struggled with. Here is my example config:
RemoteUsers  Cleartext-Password := "cisco"
        Tunnel-Type = "ESP",
        Tunnel-Password = "<pre-shared-key>",
        Cisco-AVPair := "ipsec:tunnel-type=ESP",
        Cisco-AVPair += "ipsec:key-exchange=IKE",
        Cisco-AVPair += "ipsec:save-password=1",

        Cisco-AVPair += "ipsec:inacl=110"
So, to explain some of the fields:
- "RemoteUsers" - this will be whatever your VPN group is
- Cleartext-Password := "cisco" - This is required - the router sends a request with the group name and the password of "cisco".
- Tunnel-Type - This is another required item.
- Tunnel-Password - This should be your actual VPN group key.

Now we move on to the "Cisco-AVPair" entries. These are Cisco Attribute-Value pairs. So basically this is information that gets passed back to the router as parameters for the connection.

  • ipsec:tunnel-type=ESP - Required - stating we will use ESP
  • ipsec:key-exchange=IKE - Required - stating we will use IKE
  • ipsec:save-password=1 - Optional - Since my setup is for branch-offices, I want to have the password pre-configured in the router without people needing to keep entering it. Therefore I have to permit the password to be saved locally.
  • ipsec:inacl=110 - Optional - This is for split-tunnel VPNs. It specifies which ACL on my hub router should be used to define the split networking list.
You may have already realised, this is exactly the same data as can be included statically on the Cisco router, as below:
crypto isakmp client configuration group RemoteUsers
 key <pre-shared-key>    ! Tunnel-Password = "<pre-shared-key>"
 acl 110                 ! Cisco-AVPair += "ipsec:inacl=110"
 save-password           ! Cisco-AVPair += "ipsec:save-password=1"


For further reading about Cisco Attribute-Value pairs, have a look at this page: https://supportforums.cisco.com/document/58091/exploring-remote-access-vpn-easy-vpn-cisco-router-cisco-secure-access-control-server

For our actual VPN users, the entries are much simpler. We just need to add user/password pairs to the users file:

User1     Cleartext-Password := "<password>"

Once this is done, you should have a working ezVPN setup with FreeRADIUS server.

Monday, 9 June 2014

IGMP Query Solicitation

Once again, I've come back to IGMP/Multicast/STP. This is a topic that, while reasonably well documented, can be complex to find explanations of.

In this case, I saw a network where a switch was running many (lots, like 300+) VLANs and we were regularly seeing traffic from the same MAC in all of those VLANs. This behaviour was intruiging, so I did some digging.


The switch in question was an Allied Telesyn x610 switch. Looking at the packets in Wireshark, it turns out they all originate from a switch and they are IGMP Query Solicitation. So...

What is IGMP Query Solicitation?

So you remember back in one of my earliest posts, where we had IGMP and Spanning-Tree interacting in such a way that we had floods of traffic on the network? This is very much related. In that scenario we were seeing that our switches were flooding traffic to all ports whenever there was a Spanning-Tree topology change. This "IGMP Query Solicitation" is another behaviour that can occur at exactly the same time, when a topology change is seen. As well as flooding the multicast traffic to ensure it reaches the correct destination, the switch can also send an "IGMP Query Solicitation" message to everybody. This message essentially "resets" IGMP by prompting the querier to immediately send out a General Query. This, in turn, means that all clients will re-affirm their interest in the appropriate multicasting groups by sending a Membership Report. Therefore, the IGMP snoopers (switches) will then be able to rebuild their snooping databases and once again send traffic to all the right places. So it provides a way of restoring order to your network after Spanning-Tree announces a Topology Change.

So do I want this behavior? How do I turn it off?

Yes. It's good. It helps your network to get back to normal after any changes in topology. In our case there was a device that was not coping with seeing the same MAC in 300+ VLANs at the same time. However, there may be legitimate reasons to turn this on or off. In the Cisco switches I tested, this was disabled by default. In order for it to function, it required:
  • An IGMP querier active in a VLAN (any VLAN with no querier got no query solicitation)
  • IGMP Query Solicitation enabled ("ip igmp snooping tcn query solicit")
However, in the Allied Telesyn x610 switch that I was using, this behaviour is enabled by default. In addition, IGMP snooping is enabled by default. This is sort of good because if you don't need it, it is unlikely to cause harm, yet not having it when you need it can cause your network to flood.

In the case of the AT switches, there are two options, the default is that only the STP root-bridge will send the query solicitation packets. You can turn that off, or you can optionally enable it for all switches, not just the root. Commands are:
(no) ip igmp snooping tcn query solicit root
(no) ip igmp snooping tcn query solicit
The variation in implementations and defaults across switch manufacturers is interesting. Allied Telesyn seem to be protecting people, whereas Cisco are assuming you should know what you're doing at least a little bit!

As always, I hope this has been of use to somebody!

Thursday, 15 May 2014

Cisco NAT Payload Inspection

I came across an interesting problem today with our public-facing DNS server. For various reasons that I can't actually justify, we have some DNS A records with private IP addresses in our main public-facing NS. Somebody noticed today that those weren't working any more and that DNS requests for these addresses would time out. For example, with apologies for the potentially confusing anonymisation of my company's names and addresses:

steve@linuxbox:~$ dig myname.mydomain.com @ns1.mydomain.com

; <<>> DiG 9.8.1-P1 <<>> myname.mydomain.com @ns1.mydomain.com
;; global options: +cmd
;; connection timed out; no servers could be reached

Whereas, a request for a name with a public IP would work:


steve@linuxbox:~$ dig mypublicname.mydomain.com @ns1.mydomain.com
 

; <<>> DiG 9.8.1-P1 <<>> mypublicname.mydomain.com @ns1.mydomain.com
;; global options: +cmd
;; Got answer:
--snip--

mypublicname.mydomain.com.       86400   IN      A       4.2.2.2

So... what is happening?

As I often do, my first step into looking was a packet capture. I captured whilst doing first a "public" request and then a "private" request. The result was as below:

So, for a "normal", "public" request, the response is fine. For the "private" request, the server replies correctly, but our NAT router is sending an ICMP Destination Unreachable and the response never reaches the client. So, this immediately says that the router is doing some level of inspection at Layer 5 or above and electing to drop our DNS response, which we don't want.

A quick bit of Googling later and I found this incredibly helpful article: https://plone.lucidsolutions.co.nz/networking/cisco/ios/a-workaround-for-nat-rewriting-dns-packets

This makes complete sense. So, I added the no-payload option to my NAT statement and immediately, all my DNS requests started working!


Sunday, 9 June 2013

Samsung Galaxy Player 50 (YP-G50EW) Hard Reset

So, I found it very difficult to find instructions for doing a hard reset on an early model Samsung Galaxy Player (model YP-G50EW). There are various key codes that are reported, but the one I found to work is:

Hold down Volume Up, Volume Down and the Home/Menu button, then hold Power until the Samsung logo appears. The release the power button, but keep the other buttons held down.

Now you are in the Android Recovery menu and you can continue with normal reset operations!

As always, hope this helps somebody!

Friday, 1 February 2013

IRC DOS Bot

So, if you've read my previous post, you will see that I was recently looking into a Denial-of-Service attack. Once I eventually tracked down the host it was coming from, I needed to identify what was sending all the packets.

Thankfully, the owner of the server was very interested in my offer of assistance with their server, so I was able to do more digging!

So, it was a Debian Linux server running sending a lot of source-spoofed packets onto the network. My first step was to get access back to the server remotely, to make life easier. This was simple; two iptables rules to block all packets to our two destination hosts:

iptables -I OUTPUT -d aaa.bbb.37.10 -j DROP
iptables -I OUTPUT -d aaa.bbb.38.140 -j DROP

This quickly stopped the outgoing packets and made the server remotely accessible.

Next up, checking netstat gave no obvious connections to these IPs. However, knowing they are spoofed, this was actually the wrong place to look. The offending process was almost certainly using a raw socket. So:

~# netstat -anp | grep raw
raw     0   0 0.0.0.0:1       0.0.0.0:*     7      2666/dhcpd
raw     0   0 0.0.0.0:255     0.0.0.0:*     7      9060/0]
raw     0   0 0.0.0.0:255     0.0.0.0:*     7      28477/0]


There were three processes with raw sockets open, dhcpd, and two unnamed ones. Knowing that dhcpd was legitimate, the two other processes needed looking at. A quick run of "top" showed me that one of them was consuming *way* too much CPU!

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
9060 root      20   0  1836  300  196 R  101  0.0 642:44.59 migration


It seemed a bit strange that the process was called "migration" as this is a standard Linux process. However, the real migration process would have a low pid as it is a system process started at boot. The other process was also called migration and also had a high pid, but this time without the high cpu. So, I paused both processes, without terminating them, to enable further analysis.

kill -STOP 9060
kill -STOP 28477


Another new trick I learned, which to many of you will be no surprise, is that you can simply and easily access a copy of the running application even if it has removed itself from the disk. Looking in /proc/<pid> I could see:
~# cd /proc/28477
/proc/28477# ls -l
total 0
dr-xr-xr-x 2 root root 0 Jan 28 12:12 attr
-r-------- 1 root root 0 Jan 28 12:12 auxv
-r--r--r-- 1 root root 0 Jan 28 12:12 cgroup
--w------- 1 root root 0 Jan 28 12:12 clear_refs
-r--r--r-- 1 root root 0 Jan 27 01:27 cmdline
-rw-r--r-- 1 root root 0 Jan 28 12:12 coredump_filter
-r--r--r-- 1 root root 0 Jan 28 12:12 cpuset
lrwxrwxrwx 1 root root 0 Jan 28 11:30 cwd -> /usr/sbin/ttyload
-r-------- 1 root root 0 Jan 28 12:12 environ
lrwxrwxrwx 1 root root 0 Jan 26 06:25 exe -> /usr/sbin/ttyload/migration (deleted)
dr-x------ 2 root root 0 Jan 28 11:30 fd
dr-x------ 2 root root 0 Jan 28 11:30 fdinfo
-r-------- 1 root root 0 Jan 28 12:12 io
-r-------- 1 root root 0 Jan 28 12:12 limits
-rw-r--r-- 1 root root 0 Jan 28 12:12 loginuid
-r--r--r-- 1 root root 0 Jan 28 11:30 maps
-rw------- 1 root root 0 Jan 28 12:12 mem
-r--r--r-- 1 root root 0 Jan 28 12:12 mountinfo
-r--r--r-- 1 root root 0 Jan 28 12:12 mounts
-r-------- 1 root root 0 Jan 28 12:12 mountstats
dr-xr-xr-x 8 root root 0 Jan 28 12:12 net
-rw-r--r-- 1 root root 0 Jan 28 12:12 oom_adj
-r--r--r-- 1 root root 0 Jan 28 12:12 oom_score
-r--r--r-- 1 root root 0 Jan 28 12:12 pagemap
-r--r--r-- 1 root root 0 Jan 28 12:12 personality
lrwxrwxrwx 1 root root 0 Jan 28 11:30 root -> /
-rw-r--r-- 1 root root 0 Jan 28 12:12 sched
-r--r--r-- 1 root root 0 Jan 28 12:12 sessionid
-r--r--r-- 1 root root 0 Jan 28 12:12 smaps
-r--r--r-- 1 root root 0 Jan 28 12:12 stack
-r--r--r-- 1 root root 0 Jan 27 01:26 stat
-r--r--r-- 1 root root 0 Jan 27 01:26 statm
-r--r--r-- 1 root root 0 Jan 27 01:27 status
-r--r--r-- 1 root root 0 Jan 28 12:12 syscall
dr-xr-xr-x 3 root root 0 Jan 28 11:43 task
-r--r--r-- 1 root root 0 Jan 28 12:12 wchan


So the "exe" symlink pointed to the original file on disk, but told me that it had now been deleted. However, the file was still in memory, so I was able to copy it and start looking at it!

/proc/28477# cat exe > /tmp/thing
/proc/28477# file /tmp/thing
/tmp/thing: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped


Looking further at what strings it contained (IP address hidden!):

# strings /tmp/thing
--snip--
aa.bb.ccc.251
ircd.exampledomain.net
privmsg %s :CHECKSUM <on/off>
privmsg %s :Checksum has been turned on.
privmsg %s :Checksum has been turned off.
privmsg %s :DNS <host>
privmsg %s :Resolving %s...
privmsg %s :Unable to resolve.
privmsg %s :Resolved to %s.
privmsg %s :%d.%d.%d.%d
privmsg %s :%d.%d.%d.%d - %d.%d.%d.%d
privmsg %s :White Knight 2012
privmsg %s :NICK <nick>
privmsg %s :Nick cannot be larger than 50 characters.
NICK %s
privmsg %s :Removed all spoofs
privmsg %s :What kind of subnet address is that? Do something like: 169.40
privmsg %s :BYSIN <target> <port>
privmsg %s :Packeting %s.
privmsg %s :SLICE <destination> <lowport> <highport>
privmsg %s :Error in SLICE2: Low > high.
privmsg %s :PAN <target> <lowport> <highport>
privmsg %s :Talking %s ...
privmsg %s :This knight accepts the following commands via ctcp (with a #):
privmsg %s :NICK <nick>                                      = Changes the nick of the knight
privmsg %s :GETSPOOF                                         = Gets the current spoofing
privmsg %s :SPOOFS <subnet>                                  = Changes spoofing to a subnet
privmsg %s :DNS <host>                                       = DNSs a host
privmsg %s :CHECKSUM <on/off>                                = Turns checksum on or off
privmsg %s :IRC <command>                                    = Sends this command to the server
privmsg %s :SH <command>                                     = Executes a command
privmsg %s :KILLALL                                          = Kills all current packeting
privmsg %s :KILL                                             = Kills the knight
privmsg %s :DISABLE                                          = Disables all packeting from this knight
privmsg %s :ENABLE                                           = Enables all packeting from this knight
privmsg %s :VERSION                                          = Requests version of knight
privmsg %s :HELP                                             = Displays this
kill -9 %d;kill -9 0
privmsg %s :Killing pid %d.
--snip--


So it was fairly obviously an IRC bot, with the main goal of sending spoofed DOS attacks. It also had the ability for the controller to execute arbitrary shell commands, which in this case would be run with root privilege. Therefore, it could be used to insert further malware.

I have since been playing a little with this bot in a sandbox, but that is for a later post.

Hope you found that interesting, it was a bit of a rushed post. I hope you find it an encouragement to do some of your own malware analysis!

DOS Attack

So recently I had the interesting experience of helping track down a Denial-of-Service attack coming from a network. Thankfully due to the egress filtering policies, the attack didn't make it onto the Internet.

The problem was first discovered when a Netflow collector started having resource problems. Due to the massive number of SYN packets in the DoS attack, Netflow was creating a new flow in memory for each new SYN packet, and having to remember about it until the flow timed out. Not good! After having figured out what was going on, I was able to capture a sample of traffic and start working out where it was coming from.

Here is a small sample, with the destinations partially obscured:

25771 11:48:32.584857 136.102.253.14 -> aaa.bbb.38.140 TCP 51515 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25772 11:48:32.584866 110.93.51.90 -> aaa.bbb.38.140 TCP 27433 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25773 11:48:32.584868 177.88.230.116 -> aaa.bbb.37.10 TCP 3177 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25774 11:48:32.584876 130.136.233.47 -> aaa.bbb.37.10 TCP 21115 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25775 11:48:32.584889 174.230.64.48 -> aaa.bbb.37.10 TCP 27701 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25776 11:48:32.584896 73.217.102.69 -> aaa.bbb.38.140 TCP 43969 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25777 11:48:32.584902 221.233.146.91 -> aaa.bbb.37.10 TCP 42246 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25778 11:48:32.584910 123.83.83.9 -> aaa.bbb.38.140 TCP 40649 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25779 11:48:32.584915 140.101.194.71 -> aaa.bbb.37.10 TCP 1129 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25780 11:48:32.584922 193.82.83.24 -> aaa.bbb.38.140 TCP 39930 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25781 11:48:32.584929 177.36.15.25 -> aaa.bbb.37.10 TCP 43185 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25782 11:48:32.584936 148.154.218.100 -> aaa.bbb.38.140 TCP 4283 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25783 11:48:32.584942 140.41.59.48 -> aaa.bbb.37.10 TCP 47192 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25784 11:48:32.584950 89.134.150.35 -> aaa.bbb.38.140 TCP 25345 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25785 11:48:32.584955 207.236.20.32 -> aaa.bbb.37.10 TCP 22479 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25786 11:48:32.584963 240.16.136.1 -> aaa.bbb.37.10 TCP 44095 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25787 11:48:32.584970 70.40.185.17 -> aaa.bbb.38.140 TCP 46254 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25788 11:48:32.584975 82.195.26.42 -> aaa.bbb.37.10 TCP 61179 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25789 11:48:32.584983 130.184.250.86 -> aaa.bbb.38.140 TCP 15603 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25790 11:48:32.584989 62.108.248.77 -> aaa.bbb.37.10 TCP 61996 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25791 11:48:32.584996 244.193.231.42 -> aaa.bbb.38.140 TCP 27477 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25792 11:48:32.585003 254.119.67.64 -> aaa.bbb.37.10 TCP 46181 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25793 11:48:32.585009 171.252.91.108 -> aaa.bbb.38.140 TCP 22252 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25794 11:48:32.585016 61.79.106.102 -> aaa.bbb.37.10 TCP 41133 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25795 11:48:32.585023 142.113.79.114 -> aaa.bbb.38.140 TCP 32631 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25796 11:48:32.585029 99.7.239.111 -> aaa.bbb.37.10 TCP 29999 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25797 11:48:32.585037 5.107.140.57 -> aaa.bbb.38.140 TCP 2415 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25798 11:48:32.585043 16.197.131.47 -> aaa.bbb.37.10 TCP 21038 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25799 11:48:32.585050 109.10.184.48 -> aaa.bbb.38.140 TCP 41172 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25800 11:48:32.585057 183.145.182.22 -> aaa.bbb.37.10 TCP 7899 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25801 11:48:32.585063 116.183.190.106 -> aaa.bbb.38.140 TCP 18717 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25802 11:48:32.585070 52.217.83.29 -> aaa.bbb.37.10 TCP 32629 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25803 11:48:32.585077 31.215.117.74 -> aaa.bbb.38.140 TCP 19412 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25804 11:48:32.585084 48.113.61.11 -> aaa.bbb.37.10 TCP 25432 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25805 11:48:32.585090 161.46.43.79 -> aaa.bbb.38.140 TCP 35808 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25806 11:48:32.585097 232.129.112.88 -> aaa.bbb.37.10 TCP 1616 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25807 11:48:32.585103 152.25.15.45 -> aaa.bbb.38.140 TCP 10276 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25808 11:48:32.585110 133.225.67.50 -> aaa.bbb.37.10 TCP 43895 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25809 11:48:32.585116 19.124.117.101 -> aaa.bbb.38.140 TCP 52242 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25810 11:48:32.585123 134.83.167.21 -> aaa.bbb.37.10 TCP 16292 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25811 11:48:32.585129 228.41.163.17 -> aaa.bbb.38.140 TCP 8024 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25812 11:48:32.585135 53.115.220.107 -> aaa.bbb.37.10 TCP 46765 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25813 11:48:32.585142 142.7.225.83 -> aaa.bbb.38.140 TCP 32227 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25814 11:48:32.585151 68.104.42.36 -> aaa.bbb.37.10 TCP 63759 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25815 11:48:32.585156 142.113.105.51 -> aaa.bbb.38.140 TCP 23125 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460
25816 11:48:32.585169 163.167.156.8 -> aaa.bbb.38.140 TCP 37433 > 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1460


So, a very large number of packets! The main things to notice are that they are coming from a massive variety of source addresses (none of which are valid for a packet going out of this network) and that they are destined for only two hosts. These two hosts are the targets, and the packets are being sent with spoofed source addresses, which makes tracking down the actual source quite challenging!

I was able to track down the source of the flows by adding a simple ACL (blocking packets to the two desinations) to different routers in the network, and seeing which ones scored the most hits. For example, this ACL was applied:

ip access-list extended Temp
 deny ip any host aaa.bbb.37.10
 permit ip any any

and then when reviewed a little while later:

# show access-list
Extended IP access list Temp
    10 deny ip any host aaa.bbb.37.10 (2029998454 matches)
    20 permit ip any any (42805196 matches)


Using this method I was quickly able to identify roughly where in the network the packets were coming from, but, to cut a long story short, it took a while to identify the source router.

From there, it was a case of identifying the sending host on the LAN. This can be done with mac-address accounting:

Configure:
(config-if)# ip accounting mac-address input
Check:
#sh interface F0/0 mac-accounting
FastEthernet0/0
  Input  (509 free)
    00aa.ccf2.7c51(153):  5 packets, 411 bytes, last: 1380ms ago
    00aa.ddcd.4a27(163):  1 packets, 103 bytes, last: 9236ms ago
    00bb.ee85.2569(192):  6 packets, 1573 bytes, last: 4580ms ago
                  Total:  12 packets, 2087 bytes 


Unfortunately the above output was an afterthought, and I don't have the "live" version. However, it quickly gave me the MAC of the offending host.

Due to the vast traffic flood, I had to wait until the next day before I could get physically on the console of the hacked PC and do some more digging!

More to follow very soon...

Thursday, 13 September 2012

Irritating Cisco bug

Took me a long time to figure this one out... On a Cisco router, specifically Cisco 2900 running early versions of IOS 15.2, you may be completely baffled by access-lists on your interfaces failing to work! If you do a "debug ip access-list data-plane" you see the following error:
Sep 13 09:42:45: IPACL-DP: Pkt Matched against EPM list, Action: Deny
Sep 13 09:42:45: IPACL-DP: Pkt matched punt/drop it
Sep 13 09:42:45: IPACL-DP: Pkt is dropped in cef path: interface GigabitEthernet0/1 inbound direction

If this is the case for you, it's a bug in IOS. See here. Bug is number CSCtt19027. Cisco say:
ACL applied on serial or Gi interface drops all packets with permit any
Symptoms: When ACL is applied to the serial interface or Gigabit interface,
ping failure seen even though the permit statement is there.

Conditions: The symptom is observed when ACL is configured on the serial
interface or Gigabit interface.

Workaround: Enable EPM by installing the security license.

Further Problem Description: This is seen with those images where EPM is not
supported and because of that an EPM call always gives a return value
as "deny" due to registry call.
Hope this helps someone, because when I searched, I couldn't get any hits for "Matched against EPM list" at all!