Hacked server (question about sendmail configuration)

quamenzullo

quamenzullo

Hello everyone,


I have a server with FreeBSD 10.1 that has been hacked recently.

My guess is the attacker "entered" the server using a weakness of the old cms I was using for one website. (This is my fault, I have waited too long and didn't change the cms despite the absence of updates since ages). The www/ directory but also several other directories in the root of the website have been deleted. This is all accessible to the www user, so no wonder. The website also hosted a phishing webpage.

At around the same time, the server started to send spam. So it seems likely both events are related, even if it's not completely sure.

I wonder if it is possible for an attacker, as www user, to modify something in the sendmail configuration to let it relay spam? Or is it necessary to gain root privileges to be able to do that?

In order to know if the sendmail configuration has been changed, I compared the contents of /etc/mail/freebsd.mc and /etc/mail/sendmail.cf on the hacked server to the ones from a fresh 10.2 install (no 10.1 available around here).

The freebsd.mc files are identical (except for the version number of FreeBSD in the comment).

But the sendmail.cf files have several differences:

Code: 
$ diff --suppress-common-lines sendmail.cf_raw sendmail.cf_hacked   
1,2d0
<
<
14c12
< # $FreeBSD: releng/10.2/contrib/sendmail/cf/m4/cfhead.m4 285303 2015-07-09 05:23:17Z gshapiro $
---
> # $FreeBSD: releng/10.1/contrib/sendmail/cf/m4/cfhead.m4 266692 2014-05-26 15:28:28Z gshapiro $
36c34
< #####  $FreeBSD: releng/10.2/etc/sendmail/freebsd.mc 285304 2015-07-09 05:25:47Z gshapiro $  #####
---
> #####  $FreeBSD: releng/10.1/etc/sendmail/freebsd.mc 266698 2014-05-26 15:42:39Z gshapiro $  #####
179c177
< DZ8.15.2
---
> DZ8.14.9
267,269d264
< # use compressed IPv6 address format?
< #O UseCompressedIPv6Addresses
<
321,323d315
< # maximum time in queue before retry (if > 0; only for exponential delay)
< #O MaxQueueAge
<
574,579d565
< # SSL cipherlist
< #O CipherList
< # server side SSL options
< #O ServerSSLOptions
< # client side SSL options
< #O ClientSSLOptions
603,604d588
< # fingerprint algorithm (digest) to use for the presented cert
< #O CertFingerprintAlgorithm
612,613d595
<
<
661c643
< #
---
> #

725d706
<
861d841
<
924,925d903
<
<
1211,1214d1188
< R<@> < $* @ [IPv6:0:0:0:0:0:0:0:1] >
<  $: < ? $&{client_name} > < $1 @ [IPv6:0:0:0:0:0:0:0:1] >
< R<@> < $* @ [IPv6:::1] >
<  $: < ? $&{client_name} > < $1 @ [IPv6:::1] >
1366d1339
< RIPv6:0:0:0:0:0:0:0:1 $@ RELAY  originated locally
1677,1678d1649
<
<
1716c1687
< #
---
> #

1725c1696
< #
---
> #

1878d1848
<


Several lines seem to be comments, but there's something about "RELAY" showing up in the config file from the fresh 10.2. I don't understand what changes it makes (moreover, I cannot understand how the same freebsd.mc files can be compiled into two different sendmail.cf outputs), but could that part have been deleted from the 10.1 server to let it relay e-mails?
 
Most times I've seen spam sent via a hacked website, they haven't been using the server remotely as a relay. They have been sending requests to a script on the website. Sometimes they create a new file with a random name, sometimes they completely overwrite a file from the website, and sometimes they try and hide the code in an existing file without actually changing the original functionality of that file.

They then just send post requests to that script containing what they want to send. The script sends via localhost, which is normally allowed to relay by default.

I've not yet seen a hack where they specifically enable relaying in the mail server. This would be far more likely to fail as they can't guarantee how the mail server is configured, and in the majority of cases port 25 would be blocked for incoming connections on a web server anyway. Most websites have email functionality enabled however, so once they've hacked the website it's trivial to upload a script that uses the language's built in mail functions.

Just to add: this is the most common reason for hacking a website as far as my experience goes. They're usually not interested in actually messing with the website. They want to upload something that lets them send remote commands, either to send spam via your web server, or to use as part of a DDoS attack.
 
quamenzullo said:
I wonder if it is possible for an attacker, as www user, to modify something in the sendmail configuration to let it relay spam?
Click to expand...
No, that's not possible.

Or is it necessary to gain root privileges to be able to do that?
Click to expand...
They would need root, yes.

But they do NOT require root to send out email. Even the most basic outgoing connections to port 25 are allowed by anyone (including the www user).

The way they "relay" email is by using a simple PHP script that's running on your server. The PHP script will connect to a destination domain by itself, there's no need for a working local MTA. This is one of the reasons why you should firewall outgoing connections from your webserver.
 
Please use -u for unified diff output. It is much easier to read.

As far as that server, there is no guarantee they have not found and used some remote root exploit. Install a new system, and only transfer data to it.
 
Most likely, the attacker managed to inject a spam bot. No access to sendmail is necessary for that. Until you clean your server, a quick solution would be to block all outbound traffic in your firewall and allow outbound mail only via an authorized relay host with authentication.
After the server is cleaned, you might want to keep that rule.
 
wblock@ said:
As far as that server, there is no guarantee they have not found and used some remote root exploit.
Click to expand...
That would be a local root exploit. There are no remote root exploits that I know of (at least not on any of the supported versions). There have been some on ancient versions but there hasn't been any on any recent versions.

Usually "they" don't even bother. If they can get a foothold by injecting a shell or PHP script they'll have all the access they need. One of the techniques I've seen involved uploading a perl script to /tmp, starting it and then delete the file itself. That makes it really hard to find. They'll also employ tricks to make the command line look like a regular Apache process in any ps(1) output. They'll usually make it look like a process on Linux, which makes it a little easier to spot on FreeBSD as the process looks slightly different compared to the regular Apache processes on.

In any case, you'd be surprised at what a limited account like www is still able to do. It can open ports (above 1024) and can make connections to outside systems. Pretty much everything you need for relaying spam, attack other systems and whatnot.

Install a new system, and only transfer data to it.
Click to expand...
That's always a good idea if the system has been compromised. Make sure to plug the hole they used to get in though, or else you'll have the same problem within a couple days.
 
Thanks for all these interesting posts!!

I am installing everything new on a new machine and am learning how to setup jails to have more security on the new server. I think the weakness was in this old cms I used for one website (cannot be absolutely sure though, I don't know how to be sure, where to find hints, if it's possible at least...). The website will be remade entirely with another cms.

Many thanks for the quality of this excellent forum!
 
Also, I put hereafter the result of diff -u. (I wished I could put the result of diff -y --suppress-common-lines, what is extremely clear to read).
I still wonder why these two files are different, and if the missing lines in the raw file induce anything insecure.

Code: 
$ diff -u sendmail.cf_raw sendmail.cf_hacked
--- sendmail.cf_raw  2016-01-09 08:51:04.932848045 +0100
+++ sendmail.cf_hacked  2016-01-09 08:24:25.460079946 +0100
@@ -1,5 +1,3 @@
-
-
 #
 # Copyright (c) 1998-2004, 2009, 2010 Proofpoint, Inc. and its suppliers.
 #  All rights reserved.
@@ -11,7 +9,7 @@
 # forth in the LICENSE file which can be found at the top level of
 # the sendmail distribution.
 #
-# $FreeBSD: releng/10.2/contrib/sendmail/cf/m4/cfhead.m4 285303 2015-07-09 05:23:17Z gshapiro $
+# $FreeBSD: releng/10.1/contrib/sendmail/cf/m4/cfhead.m4 266692 2014-05-26 15:28:28Z gshapiro $
 #
 ######################################################################
@@ -33,7 +31,7 @@
 #####  $Id: cfhead.m4,v 8.122 2013-11-22 20:51:13 ca Exp $  #####
 #####  $Id: cf.m4,v 8.33 2013-11-22 20:51:13 ca Exp $  #####
-#####  $FreeBSD: releng/10.2/etc/sendmail/freebsd.mc 285304 2015-07-09 05:25:47Z gshapiro $  #####
+#####  $FreeBSD: releng/10.1/etc/sendmail/freebsd.mc 266698 2014-05-26 15:42:39Z gshapiro $  #####
 #####  $Id: freebsd6.m4,v 1.2 2013-11-22 20:51:15 ca Exp $  #####
@@ -176,7 +174,7 @@
 Kvirtuser hash -o /etc/mail/virtusertable
 # Configuration version number
-DZ8.15.2
+DZ8.14.9
 ###############
@@ -264,9 +262,6 @@
 # use Errors-To: header?
 O UseErrorsTo=False
-# use compressed IPv6 address format?
-#O UseCompressedIPv6Addresses
-
 # log level
 O LogLevel=9
@@ -318,9 +313,6 @@
 # minimum time in queue before retry
 #O MinQueueAge=30m
-# maximum time in queue before retry (if > 0; only for exponential delay)
-#O MaxQueueAge
-
 # how many jobs can you process in the queue?
 #O MaxQueueRunSize=0
@@ -571,12 +563,6 @@
 # SMTP STARTTLS server options
 #O TLSSrvOptions
-# SSL cipherlist
-#O CipherList
-# server side SSL options
-#O ServerSSLOptions
-# client side SSL options
-#O ClientSSLOptions
 # Input mail filters
 #O InputMailFilters
@@ -600,8 +586,6 @@
 O DHParameters=/etc/mail/certs/dh.param
 # Random data source (required for systems without /dev/urandom under OpenSSL)
 #O RandFile
-# fingerprint algorithm (digest) to use for the presented cert
-#O CertFingerprintAlgorithm
 # Maximum number of "useless" commands before slowing down
 #O MaxNOOPCommands=20
@@ -609,8 +593,6 @@
 # Name to use for EHLO (defaults to $j)
 #O HeloName
-
-
 ############################
 # QUEUE GROUP DEFINITIONS  #
 ############################
@@ -658,7 +640,7 @@
 H?M?Resent-Message-Id: <$t.$i@$j>
 H?M?Message-Id: <$t.$i@$j>
-#
+#
 ######################################################################
 ######################################################################
 #####
@@ -722,7 +704,6 @@
 # if we have % signs, take the rightmost one
 R$* % $*  $1 @ $2  First make them all @s.
 R$* @ $* @ $*  $1 % $2 @ $3  Undo all but the last.
-
 R$* @ $*  $@ $>Canonify2 $1 < @ $2 >  Insert < > and finish
 # else we must be a local name
@@ -858,7 +839,6 @@
  $@ $>Parse0 $>canonify $1 $2 $3 ...@*LOCAL* -> ...
 R$* < @ *LOCAL* >  $: $1
-
 #
 #  Parse1 -- the bottom half of ruleset 0.
 #
@@ -921,8 +901,6 @@
 R$=L  $#local $: @ $1  special local names
 R$+  $#local $: $1  regular local names
-
-
 ###########################################################################
 ###  Ruleset 5 -- special rewriting after aliases have been expanded  ###
 ###########################################################################
@@ -1208,10 +1186,6 @@
 R<@> < $* @ localhost >  $: < ? $&{client_name} > < $1 @ localhost >
 R<@> < $* @ [127.0.0.1] >
  $: < ? $&{client_name} > < $1 @ [127.0.0.1] >
-R<@> < $* @ [IPv6:0:0:0:0:0:0:0:1] >
-  $: < ? $&{client_name} > < $1 @ [IPv6:0:0:0:0:0:0:0:1] >
-R<@> < $* @ [IPv6:::1] >
-  $: < ? $&{client_name} > < $1 @ [IPv6:::1] >
 R<@> < $* @ localhost.$m >
  $: < ? $&{client_name} > < $1 @ localhost.$m >
 R<@> < $* @ localhost.UUCP >
@@ -1363,7 +1337,6 @@
 R$@  $@ RELAY  originated locally
 R0  $@ RELAY  originated locally
 R127.0.0.1  $@ RELAY  originated locally
-RIPv6:0:0:0:0:0:0:0:1  $@ RELAY  originated locally
 RIPv6:::1  $@ RELAY  originated locally
 R$=R $*  $@ RELAY  relayable IP address
 R$*  $: $>A <$1> <?> <+ Connect> <$1>
@@ -1674,8 +1647,6 @@
 R$-:$-:$-  $: $2
-
-
 ######################################################################
 ###  RelayTLS: allow relaying based on TLS authentication
 ###
@@ -1713,7 +1684,7 @@
-#
+#

 ######################################################################
 ######################################################################
 #####
@@ -1722,7 +1693,7 @@
 ######################################################################
 ######################################################################
-#
+#

 ######################################################################
 ######################################################################
 #####
@@ -1876,4 +1847,3 @@
  T=DNS/RFC822/SMTP,
  A=TCP $h
-
 
I didn't look at all of it, but the diff looks like what it says it is. You had 10.2 and are now trying to restore 10.1 over it. Or it could be that the hackers had a 10.2 sendmail.cf, but that is unlikely.
 
Ok, so, this kind of difference is normal, after all?

I did understand from the documentation here and there (steps 6 and 7) that the compilation of /etc/mail/freebsd.mc produces /etc/mail/sendmail.cf. (Maybe I'm wrong). (By the way, /usr/src/contrib/sendmail/cf/README does not exist on my 10.2 install, neither on the 10.1). That's why I was wondering why the same mc files do not produce the same cf files.

Now, seing this on the hacked server:
Code: 
rescue-bsd# ls -alhG /mnt/etc/mail/*.mc
-rw-r--r--  1 root  wheel  4.4K Feb 17  2015 /mnt/etc/mail/freebsd.mc
-r--r--r--  1 root  wheel  898B Feb 17  2015 /mnt/etc/mail/freebsd.submit.mc
rescue-bsd# ls -alhG /mnt/etc/mail/*.cf
-rw-r--r--  1 root  wheel  57K Feb 17  2015 /mnt/etc/mail/freebsd.cf
-r--r--r--  1 root  wheel  40K Feb 17  2015 /mnt/etc/mail/freebsd.submit.cf
-rw-r--r--  1 root  wheel  57K Feb 17  2015 /mnt/etc/mail/sendmail.cf
-r--r--r--  1 root  wheel  40K Feb 17  2015 /mnt/etc/mail/submit.cf
rescue-bsd#
I think that maybe /etc/mail/sendmail.cf is compiled from something else? Although the 7th step here leaves me puzzled about what file produces what:
7. Finally, run make(1) while in /etc/mail. That will run the new .mc and create a .cf named either freebsd.cf or the name used for the local .mc. Then, run make install restart, which will copy the file to sendmail.cf, and properly restart Sendmail
Click to expand...

Anyway, changing the cf file would require to restart sendmail, what requires root privileges. Unfortunately, when I discovered that sendmail was sending spam, I stopped it. I checked but don't remember its uptime.

So I guess there are now only very little means to know if a root access has been gained or not; even if it seems completely unlikely, from what I understand from the posts above.
 
quamenzullo said:
So I guess there are now only very little means to know if a root access has been gained or not; even if it seems completely unlikely, from what I understand from the posts above.
Click to expand...
Because you can't tell if they did or didn't have root access you can't trust anything anymore. Hence the suggestion to wipe the machine and start over as that's the only way to be sure there aren't any backdoors left.
 
Another hint about the hacked server. The nginx-access*.log files are writable only by root, but there's a huge gap in the dates.

nginx-access.log.2.bz2 ends with

180.**.**.* - - [07/Nov/2015:00:57:38 +0100] "GET / HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
Nov 7 01:00:00 <hostname> newsyslog[9968]: logfile turned over due to size>100K
Click to expand...

and nginx-access.log.1.bz2 starts with

Nov 7 01:00:00 <hostname> newsyslog[9968]: logfile turned over due to size>100K
79.**.***.** - - [23/Dec/2015:01:03:11 +0100] "GET / HTTP/1.1" 404 168 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:42.0) Gecko/20100101 Firefox/42.0"
Click to expand...

Doesn't it mean there has been a root access? (Or is it still possible to alter the content of these log files as www user?)
 
Last edited:
I don't know how it works (and no idea where to find the answer, apart from studying the source code maybe? can't find it googling). nginx is configured to "act as" the www user, but I run it as root (is this a stupid mistake?).

I don't know neither if it's possible for nginx to only append or also remove entries from its log files. Nor if its permissions would let it write to other log files or not. (And how?)
 
No version of Sendmail I've ever used on FreeBSD would ever allow relaying mail from any of my MX's unless it was purposefully allowed. Stock installs do not permit it. With the exception of a couple of Sendmail macros, the only way Sendmail doesn't scream when www attempts to relay mail, is if it (www) was added to /etc/mail/trusted-users. Have a look, is www, or what ever user is running your web server, located in that file? Is there anybody in that file you don't recognize? Speaking of recognizing; have you run vipw(8) after the attack? No? Why not? How better, to discover the extent of the attack? How about your nightly logs? You know, the "security run"? It would have told you explicitly what changed on your system. Also, have a look in /var/backups. It's there, in case things go wrong -- you know, like getting hacked? I could go on, and on. But I've at least covered some of the more obvious ones. Sorry to come down so hard. Nothing personal. :)

Point being; FreeBSD doesn't want you to get hacked, and contains a great many stop-gaps, from some very experienced people. So you pretty much need to try and get hacked, to get around them. :)

--Chris
 
So, here is what I found out (sorry for the delay):

1. There is no /etc/mail/trusted-users. Neither on the hacked server, nor on a fresh installed FreeBSD. (Is it abnormal?)
2. I didn't know vipw(). I don't know how to use it to know more about the (possible) attack: there seem to be nothing wrong in this file, no unknown user at least.
3. The security output doesn't show anything wrong on the day the spam started (and nothing wrong in the days before). graphics/png is mentionned as vulnerable since less than one day.

Code: 
Subject: *****.org daily security run output


Checking setuid files and devices:

Checking negative group permissions:

Checking for uids of 0:
root 0
toor 0

Checking for passwordless accounts:

Checking login.conf permissions:

*****.org login failures:

*****.org refused connections:

Checking for packages with security vulnerabilities:
Database fetched: Thu Dec 17 03:14:29 CET 2015
png-1.6.19

-- End of security output --

Couldn't find anything wrong in the other security outputs. The daily run outputs let me find when the first spam has been sent.

Still looks like someone used some php script on the old-updated website. Ssh was configured to accept connections using authorized ssh keys anyway (no password connections, no root connections).

Now, I tried to follow the hints you gave to me, and hope to have done the best with them, but maybe I've missed something?
 
You must log in or register to reply here.
Back
Top