Cisco NAT Payload Inspection

I came across an interesting problem today with our public-facing DNS server. For various reasons that I can't actually justify, we have some DNS A records with private IP addresses in our main public-facing NS. Somebody noticed today that those weren't working any more and that DNS requests for these addresses would time out. For example, with apologies for the potentially confusing anonymisation of my company's names and addresses:

steve@linuxbox:~$ dig myname.mydomain.com @ns1.mydomain.com

; <<>> DiG 9.8.1-P1 <<>> myname.mydomain.com @ns1.mydomain.com
;; global options: +cmd
;; connection timed out; no servers could be reached

Whereas, a request for a name with a public IP would work:

steve@linuxbox:~$ dig mypublicname.mydomain.com @ns1.mydomain.com
 
; <<>> DiG 9.8.1-P1 <<>> mypublicname.mydomain.com @ns1.mydomain.com
;; global options: +cmd
;; Got answer:
--snip--
mypublicname.mydomain.com.       86400   IN      A       4.2.2.2

So... what is happening?

As I often do, my first step into looking was a packet capture. I captured whilst doing first a "public" request and then a "private" request. The result was as below:

So, for a "normal", "public" request, the response is fine. For the "private" request, the server replies correctly, but our NAT router is sending an ICMP Destination Unreachable and the response never reaches the client. So, this immediately says that the router is doing some level of inspection at Layer 5 or above and electing to drop our DNS response, which we don't want.

A quick bit of Googling later and I found this incredibly helpful article: https://plone.lucidsolutions.co.nz/networking/cisco/ios/a-workaround-for-nat-rewriting-dns-packets

This makes complete sense. So, I added the no-payload option to my NAT statement and immediately, all my DNS requests started working!