Security

We had some issues at work recently with intermittent e-mail’s from a third party not arriving in our Google App’s mailboxes (we’ve now migrated to using Google App’s for Business as opposed to using Exchange exclusively).

This was a big problem for us as these e-mail’s are an important source for news stories.

I immediately suspected spam/DNS server issues due to my previous experience trouble-shooting our Ironport issue two years ago.

This time however the problem lay with the third parties DNS not ours.

Here are the steps I took to troubleshoot then fix the problem:

Symptoms:

The issue was first reported when one of our news reporters noted some e-mails were not arriving – sent in from this news sources distribution list.

Some e-mail not all were being dropped – this is an important factor as we’ll see below.

Troubleshooting steps:

1. I realised straight away the problem wasn’t related to any e-mail filters in the reporters inbox or SMTP blacklists since approx only 20% of the mails were not arriving. Nonetheless I ran the check’s necessary and found nothing causing mail to be filtering into the users bin.

2. I checked in on another colleague who’s mail domain address is different from the news reporter that detected the problem (We use multiple domain names for separate titles and business teams). That colleague indicated they were missing the exact same e-mails that the first reporter detected missing.

3. The distribution list in question is used by a number of other news organisations so I sent a mail out to another news organisation’s IT dept to ask if they’d noticed those particular e-mails missing – they hadn’t.

4. I then asked an ordinary user at the news sources site, (I’m calling her [email protected] here) to send me an e-mail so I could evaluate the incoming e-mail headers for anything out of place. The e-mail she sent arrived fine but the information I found in her mail header provided the light bulb moment when taken into account with the rest of the information gathered. I found two key entries in [email protected]‘s mail header:

Received-SPF: fail (google.com: domain of [email protected] does not designate 137.191.225.35 as permitted sender) client-ip=137.191.225.35

Authentication-Results: mx.google.com; 

spf=hardfail (google.com: domain of [email protected] does not designate 137.191.225.35 as permitted sender) smtp.mail=137.191.225.35;

It was time to brush up on SPF or Sender Policy Framework

SPF basically boils down to a DNS entry that indicates which SMTP servers are permitted to send e-mail on behalf of a mail domain. This prevents spammers from getting e-mail’s into a users inbox since spam prevention e-mail gateways like Ironport and Postini will perform a DNS verification check each time a particular SMTP server tries to deliver an e-mail purporting to be from a particular e-mail address (domain).

As an example say a spammer tried to deliver an e-mail to [email protected], by spoofing an innocent individuals e-mail address (let’s say [email protected]) and used their spamming SMTP server to try to send [email protected] a spam e-mail. The Ironport/Postini/Spam Gateway when it’s first contacted examines the mailing domain indicated in the e-mail address of the message – say “hotmail.com” and the accompanying IP address of the spamming SMTP server.

Ironport/Postini will then contact the DNS server (which should always be accessible on the internet) for hotmail.com  to check that DNS Server’s SPF record. If it see’s that the IP addresses listed in the SPF record doesn’t match the IP address of the SMTP server that just tried to deliver an e-mail purporting to be from [email protected] it will drop that e-mail delivery attempt to [email protected], thereby preventing the spam reaching [email protected]‘s inbox. According to hotmail.com‘s DNS SPF record the spamming SMTP server trying to deliver a message purporting to be from [email protected] is not authorized…

Now back to our problem – So our news organisations Postini gateway was designating the IP address of our third parties SMTP server 137.191.225.35 as not being permitted to send e-mail on behalf of that user.

I had two problems with that –

1. First why did the SPF fail?

2. The SPF failed but [email protected]‘s e-mail got through anyway.

It was time to probe our news sources (xxxx.ie‘s) DNS configuration. In the process of doing so I discovered it was possible to expose the SPF configuration for DNS server’s available on the internet using this tool

So I plugged in the IP Address of the 137.191.225.35 SMTP server designated not authorized in [email protected]‘s e-mail header.

The Beveridge SPF Test tool then exposed the following SPF configuration for the xxxx.ie domain (some IP addresses changed for security purposes):

v=spf1 mx ip4:137.191.xxx.x1 ip4:137.191.xxx.x2 ip4:137.191.xxx.x3 -all [TTL=86400]

But I also noticed subsequent SPF test’s also returned a different SPF configuration below:

v=spf1 mx ip4:137.191.xxx.x1 ip4:137.191.xxx.x2 ip4:137.191.xxx.x3 ip4:137.191.225.35 ip4:137.191.xxx.x5 -all

So what was I looking at?

Basically one of more DNS servers for the domain xxxx.ie which our Postini gateway was performing SPF checks against had an incorrect/out of date SPF configuration. The second SPF configuration was the correct one as it listed 5 SMTP servers that were authorized to send mail on behalf of the xxxx.ie domain

The first incorrect SPF also should not have contained the  “[TTL=86400]” reference, as this doesn’t conform to SPF standards.

Cross-referencing the IP address of the 137.191.225.35 SMTP Server in the e-mail header I found that 137.191.225.35 matched a missing IP address in the incorrect SPF record. 137.191.225.35 was present in the second SPF record but not the first.

So our Postini was hitting the DNS server with the incorrect SPF configuration about 20% of the time and therefore dropping 20% of the e-mails sent because it was determining that 137.191.225.35 wasn’t authorized to send e-mail on behalf of xxxx.ie 20% of the time.

I’d figured out the issue was was due to an incomplete SPF record on one xxxx.ie DNS server but why did the e-mail from [email protected] deliver though it was given an SPF failure while some e-mails from the distribution list [email protected] were not being at all delivered?

My guess is that our Postini gateway even though it flagged [email protected] as being an SPF failure – SPF checking for Google/Postini is evidently not that strict. There must be additional spam filtering criteria in place on our Postini that flags the [email protected] e-mails based on message content, that together with the results returned from the problem DNS server flagged the [email protected] as being spam 20% of the time. Our Postini services are provided by an outsourced company so I don’t have access to the specific spam filtering criteria to verify this.

Fix:

I contacted the IT Security department manager for the xxxx.ie, who confirmed my findings and isolated the problem DNS server. The DNS server in question was not under his dept’s direct control but was managed by another dept – I’m guessing for redundancy purposes.

He logged a change to have the SPF details applied to the problem DNS server which should eliminate our problems receiving mail from any @xxxx.ie addresses in the future.

Hopefully this has given you an insight to the complex world of spam filtering….

I’ve been testing Microsoft’s Direct Access at work.

In my opinion it’s a true VPN killer, if you only need to deal with Windows Technology in your workplace.

Direct Access uses certificate/IPSEC based encryption/authentication to authorise a remote user and allow them to access a corporate network. The key thing here though is that the remote user doesn’t interact with any software to authenticate. The user’s authenticated without any action on their part – all they need to do is insert their username and password at the login screen much like you would if you were logging onto a terminal in the office.

No more fumbling around with a VPN dongle – if you have a broadband connection you can login and access network resources at the office from home.

I have noticed some problems using Direct Access with a Vodafone 3G/HSDPA connection though so I’d like to share what I did to get the service to work because it can be a pain in the butt to trouble shoot and the advanced diagnostic logs can be hard to decipher.

Firstly make sure you’re using the latest version of your Vodafone Mobile Broadband software (formerly called Vodafone Mobile Connect). I use version 10.2.302. Type “Vodafone Mobile Broadband Software” into Google and your first hit should take you to a page with the latest version. It’s important you get the latest version installed – I initially tried Vodafone Mobile Connect 9.4.6.20539 and couldn’t get Direct Access working no matter what.

The second part of the puzzle is 6to4. I had to completely disable this virtual adapter to force Direct Access to make a connection via Teredo. My suspicion (and this only seems to be a problem with mobile broadband connections) is that Direct Access doesn’t auto-configure the 6to4 adapter address properly and as a result Direct Access doesn’t fail over to any other connection type if it can’t communicate with the 6to4 adapter.

So start a CMD prompt with “Run as administrator” – as below. It’s important you start the CMD prompt with “Run as administrator” as a domain user even with local admin privileges will not be sufficient.

Once you have the CMD prompt up use the following command to disable 6to4:

netsh interface 6to4 set state disabled

And Direct Access should miraculously connect

The following command will take you back to where you started:

netsh interface 6to4 set state enabled

 

 

 

 

Have fun – I hope this helps you out of any Direct Access problems you may be having

I had problems with a virus recently making posts to my Facebook profile of all things. When I first noticed the problem I started going through in my head what I did in the past couple of days to expose my laptop to malware. The cause was simply down to being lax updating my AVG antivirus.  Always, Always, Always keep your anti-virus up to date

So I got home after my work day and set to work killing this virus. First came MalwareBytes which detected two malware and one trojan.

It was then time for an AVG update. After one definition update I ran the updater a second time and more definitions were installed (I’d been very lax)

All was well until I restarted the laptop. Which is when my problems started. The laptop wouldn’t boot back into the OS.

I could detect a bluescreen but couldn’t see what the error message was.

If this happens to you boot into F8 (Safe Mode) then choose the option to “Disable automatic restart on system failure” – this will give you a chance to see what’s causing the blue screen.

In my case it was an Unmountable_Boot_Volume.

I can only put this down to the virus corrupting my MBR when Malware Bytes removed it. Nasty…

So I set about booting into my WinXP setup cd to access the recovery console and run the fixmbr cmd to repair my hard disks Master Boot Record. This had no effect however. Fixboot was a non starter also.

It was at this point that I began to think I’d need something more powerful. My second thought was Microsoft must have a fix for this?

CHKDSK – chkdsk really? The program that runs when your PC restarts after it’s power plug caught your foot? Yes – That chkdsk….

http://support.microsoft.com/kb/555302

chkdsk driveletter: /f

In my case chkdsk f: /f

Damn if it didn’t work! Here’s what I got back….

I’ll never underestimate chkdsk again…

The following scenario highlights the importance of having regular server status reporting in place for unexpected events.

For a couple of months at irregular intervals we had users complain that some mails (not all) they’d been expecting from external users were arriving days later or were not arriving at all.

I’d been examining the headers of delayed e-mails when they did arrive and had found that external mail servers delivering e-mails to ours were taking a number of hours or days to do so. Most of the delayed e-mails “seemed” to originate from Eircom (an ISP based in Ireland) so I chalked the problem down to an issue on Eircoms side. Our Security team agreed. This was an oversight on my part however which I’ll explain later…

As I examined more closely I was able to pull more delayed e-mails from external users whose e-mail were not routed through Eircom so logically this meant the problem wasn’t specifically with Eircom.

As a test I had some of the users that reported delays e-mail my employers e-mail address whilst CCing in my personal gmail address. Eventually the issue reproduced itself.

Mails arrived in my Gmail Inbox within a couple of minutes whilst e-mail destined for my employers e-mail address arrived a number of hours later. In the end I was able to reproduce the issue with the help of 4 seperate senders.

Convincing the Security team was a problem however as everything looked fine on our end. They had a hard time accepting the issue was on our side even though I could reproduce the problem with four seperate senders. Four seperate external mail servers behaving in the same way at the same time? and the problem is not on our end?

They needed more evidence…

So the next port of call was Eircom – They were able to provide the following logs. IP addresses and e-mail addresses removed:

2010-07-13 17:22:19.744009500 starting delivery 2146375: msg 3317173 to remote [email protected]
2010-07-13 17:28:59.581941500 starting delivery 2147602: msg 3317173 to remote [email protected]
2010-07-13 17:49:00.077173500 starting delivery 2150674: msg 3317173 to remote [email protected]
2010-07-13 18:22:19.009197500 starting delivery 2158998: msg 3317173 to remote [email protected]
2010-07-13 19:08:59.078851500 starting delivery 2171856: msg 3317173 to remote [email protected]
2010-07-13 20:08:59.270208500 starting delivery 2188502: msg 3317173 to remote [email protected]
2010-07-13 21:22:20.122931500 starting delivery 2203875: msg 3317173 to remote [email protected]
2010-07-13 22:48:59.138197500 starting delivery 2224308: msg 3317173 to remote [email protected]
2010-07-14 00:28:59.064738500 starting delivery 2244997: msg 3317173 to remote [email protected]
2010-07-14 02:22:19.239298500 starting delivery 2261581: msg 3317173 to remote [email protected]
2010-07-14 04:28:59.989719500 starting delivery 2279211: msg 3317173 to remote [email protected]
2010-07-14 06:48:59.004682500 starting delivery 2299239: msg 3317173 to remote [email protected]
2010-07-14 09:22:19.021063500 starting delivery 2342483: msg 3317173 to remote [email protected]
2010-07-14 12:08:59.033052500 starting delivery 2428140: msg 3317173 to remote [email protected]
2010-07-13 17:22:19.744009500 starting delivery 2146375: msg 3317173 to remote [email protected]

2010-07-13 17:24:09.861329500 delivery 2146375: deferral: Connected_to_<IPAddress>_but_sender_was_rejected./Remote_host_said:_451_#4.1.8_Domain

_of_sender_address_<3rdparty@theircompany.ie>_does_not_resolve/

From the above you can see that Eircoms gateway had attempted to deliver the mail in question numerous times over two days but was being rejected because the senders e-mail address could not be resolved.

So I asked the Security team to investigate if any work had been carried out on our DNS infrastructure (or any failures) on the 13^th that would prevent the Ironport performing a DNS lookup.

They found that one of the DNS servers which our Ironport server was actively using for sender verification was restarting intermittently causing our Ironport to drop connection attempts when the external mail server attempted delivery. 

They didn’t have status reporting in place for the DNS servers to report any unexpected events.

Moral of the story – don’t rely on e-mail headers to judge mail delivery attempts (they only indicate successful connections) and make sure you have status reporting in place.