Possible server intrusion

Tristram · Aug 30, 2013

For a few months now I've been running FreeBSD on a general purpose household server, shared between myself and my roommates. Recently, though, I noticed something worrisome:

Code:

#sockstat -4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
ftp      ftpd       11315 0  tcp4   192.168.1.80:21       192.168.1.99:56050
ftp      ftpd       11315 1  tcp4   192.168.1.80:21       192.168.1.99:56050
torrent_control python2.71160 3 tcp4 *:8112               *:*
root     ftpd       1121  6  tcp4   *:21                  *:*
root     sendmail   1082  4  tcp4   127.0.0.1:25          *:*
root     sshd       1079  4  tcp4   *:25094               *:*
torrent_control python2.71067 9 tcp4 127.0.0.1:58846      *:*
torrent_control python2.71067 10 udp4 *:60965             *:*
torrent_control python2.71067 11 udp4 *:14415             *:*
torrent_control python2.71067 12 udp4 *:6771              *:*
torrent_control python2.71067 15 udp4 192.168.1.80:6771   *:*
torrent_control python2.71067 16 udp4 192.168.1.80:63742  *:*
dlna     minidlnad  1051  5  udp4   *:1900                *:*
dlna     minidlnad  1051  6  tcp4   *:8200                *:*
dlna     minidlnad  1051  7  udp4   192.168.1.80:22899    *:*
root     syslogd    961   7  udp4   *:514                 *:*
?        ?          ?     ?  tcp4   192.168.1.80:46445    81.17.19.117:65531
?        ?          ?     ?  tcp4   192.168.1.80:42007    81.17.19.117:65531
?        ?          ?     ?  tcp4   192.168.1.80:52315    96.252.188.5:40993
?        ?          ?     ?  tcp4   192.168.1.80:42640    14.201.155.217:50965
?        ?          ?     ?  tcp4   192.168.1.80:13073    76.120.18.153:22037
?        ?          ?     ?  tcp4   192.168.1.80:40680    76.120.18.153:22037
?        ?          ?     ?  tcp4   192.168.1.80:49013    14.201.155.217:50965
?        ?          ?     ?  tcp4   192.168.1.80:62308    76.120.18.153:22037
?        ?          ?     ?  tcp4   192.168.1.80:31896    14.201.155.217:50965

Specifically the last 9 lines are troubling me. The computer has SSH, FTP, torrent (Deluge), and DLNA servers running, along with Deluge's web interface, and only Deluge should ever be connected to the outside world. I checked the torrent list and nothing seemed to be active, though, so I'm not sure what those other connections could be or why they don't have a command, user, PID, or FD listed. I hope I'm wrong, but I'm concerned that this may be indicative of some sort of security breach. Does anyone have any insight as to what else, if anything, could cause this? Is there any way to get more information on those connections?

The server is running FreeBSD 9.1-RELEASE-p5.

SirDice · Aug 30, 2013

Not sure what's causing it but I have seen it on a few servers too. As far as I can tell it's harmless. It's not a security breach, so you can keep breathing

devildetail · Aug 30, 2013

In your place, I would try to monitor the sockets in a freshly booted system with the DLNA and torrent services stopped. After that, you can start the services, one at a time, with an interval of one or two minutes; during this interval you can monitor the sockets and see what's going on.

J65nko · Aug 30, 2013

My first thought was that those last nine connections are probably passive FTP data connections. Before transferring data to the client, the server suggests the client to initiate a FTP data connection using a specific port. The server chooses this port from this range.

Code:

[cmd=#]sysctl -a | grep range\.hi[/cmd]
net.inet.ip.portrange.hilast: 65535
net.inet.ip.portrange.hifirst: 49152

There are two open FTP data connections:

Code:

USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
ftp      ftpd       11315 0  tcp4   192.168.1.80:[highlight]21[/highlight]       192.168.1.99:56050
ftp      ftpd       11315 1  tcp4   192.168.1.80:[highlight]21[/highlight]       192.168.1.99:56050

But the 192.168.1.99 foreign address does not match any of those nine connections. So I wonder too

Pasting those last nine connections into temp file aap:

Code:

[cmd=%] awk '{print $7,"\t", $6}' aap | sort[/cmd]
14.201.155.217:50965     192.168.1.80:31896
14.201.155.217:50965     192.168.1.80:42640
14.201.155.217:50965     192.168.1.80:49013
76.120.18.153:22037      192.168.1.80:13073
76.120.18.153:22037      192.168.1.80:40680
76.120.18.153:22037      192.168.1.80:62308
81.17.19.117:65531       192.168.1.80:42007
81.17.19.117:65531       192.168.1.80:46445
96.252.188.5:40993       192.168.1.80:52315

[cmd=%]awk '{print $7,"\t", $6}' aap >aap2[/cmd]
[cmd=%]awk -F ":" '{print $1}' aap2 | sort -u[/cmd]
14.201.155.217
76.120.18.153
81.17.19.117
96.252.188.5

[cmd=%] for X in $(cat aap3) ; do printf "IP: $X : " ; dig +short -x ${X}; echo; done[/cmd]
IP: 14.201.155.217 : 14-201-155-217.static.tpgi.com.au.

IP: 76.120.18.153 : c-76-120-18-153.hsd1.co.comcast.net.

IP: 81.17.19.117 : 
IP: 96.252.188.5 : pool-96-252-188-5.tampfl.fios.verizon.net.

If your DLNA is only supposed to work on the local network then why do you have these minidlnad services running on all interfaces, including the public interface?

Code:

dlna     minidlnad  1051  5  udp4   [highlight]*:1900[/highlight]                *:*
dlna     minidlnad  1051  6  tcp4   [highlight]*:8200[/highlight]                *:*

adripillo · Aug 30, 2013

Tristram said:
I hope I'm wrong, but I'm concerned that this may be indicative of some sort of security breach. Does anyone have any insight as to what else, if anything, could cause this? Is there any way to get more information on those connections?

Look at mine:

Code:

root     nmbd       2226  13 udp4   16.1.1.115:138        *:*
root     nmbd       2226  14 udp4   16.255.255.255:138    *:*
root     cupsd      2206  8  tcp4   127.0.0.1:631         *:*
root     cupsd      2206  10 udp4   *:631                 *:*
root     ntpd       2183  20 udp4   *:123                 *:*
root     ntpd       2183  22 udp4   16.1.1.115:123        *:*
root     ntpd       2183  26 udp4   127.0.0.1:123         *:*
?        ?          ?     ?  tcp4   16.1.1.115:21957      16.1.1.113:3128
?        ?          ?     ?  tcp4   16.1.1.115:21961      16.1.1.113:3128

The 16.1.1.113 is the proxy computer and 16.1.1.115 is my computer. So, I really do not know what they are doing on those two ports 21957 and 21961 but I think (and hope) it is not a big problem.

junovitch@ · Aug 30, 2013

According to http://whatismyipaddress.com

81.17.19.117 - Corporate IP owned by Private Layer INC in Switzerland
96.252.188.5 - Verizon FIOS in New Port Richey, Florida, US
14.201.155.217 - TPG Broadband Internet in Mornington, Australia
76.120.18.153 - Comcast Cable in Denver, Colorado, US

My money is on BitTorrent. Have you tried stopping it completely and seeing what happens? It's interface may not be telling the whole story.

junovitch@ · Aug 30, 2013

adripillo said:

Look mine:

Code:

root     nmbd       2226  13 udp4   16.1.1.115:138        *:*
root     nmbd       2226  14 udp4   16.255.255.255:138    *:*
root     cupsd      2206  8  tcp4   127.0.0.1:631         *:*
root     cupsd      2206  10 udp4   *:631                 *:*
root     ntpd       2183  20 udp4   *:123                 *:*
root     ntpd       2183  22 udp4   16.1.1.115:123        *:*
root     ntpd       2183  26 udp4   127.0.0.1:123         *:*
?        ?          ?     ?  tcp4   16.1.1.115:21957      16.1.1.113:3128
?        ?          ?     ?  tcp4   16.1.1.115:21961      16.1.1.113:3128

The 16.1.1.113 is the proxy computer and 16.1.1.115 is my computer. So, I really do not know what are doing on that 2 ports 21957 and 21961 but I think (and hope) is not a big problem.

Your local system picked a random high port to connect to what appears to be the default port of a Squid proxy. That random number is nothing to worry about and will be different every time your computer opens a connection to that proxy.

Tristram · Aug 30, 2013

devildetail said:
In your place, I would try to monitor the sockets in a freshly booted system with the DLNA and torrent services stopped. After that, you can start the services, one at a time, with an interval of one or two minutes; during this interval you can monitor the sockets and see what's going on.

I just tried this and it helped narrow things down quite a bit. After starting everything except Deluge, things were in a relatively stable state with only the expected ports open. When I started Deluge, though, the odd "? ? ? ..." entries started appearing. After running netstat a few times, I noticed something. On the first run, this line was present:

Code:

torrent_control python2.71833 12 tcp4 192.168.1.80:27400  93.152.182.112:51410

and after running netstat again, the previous line was gone and this one was at the bottom:

Code:

?        ?          ?     ?  tcp4   192.168.1.80:27400    93.152.182.112:51410

Looking over the output of successive commands, it seems that when it starts Deluge connects to a number of foreign IP addresses (which is to be expected) and then drops most of the connections, but some of the "dropped" connections remain in the form shown above. I guess it makes sense that the user/command/etc. are blank for those entries, as the thread bound to the port has probably died, but I'm not sure why they're still open at all. Does anyone know why this happens, or if there any potential side effects? Even after stopping deluged three lines of that form remained, seemingly indefinitely.

J65nko said:
If your DLNA is only supposed to work on the local network then why do you have these minidlnad services running on all interfaces, including the public interface?

minidlnad doesn't seem to have any way of limiting who connects to it via config files, so I limit access through IPFW. Also, the network is behind a router firewall which only forwards ports 6881-6891.

Thank you for your help, everyone.

ronaldlees · Aug 30, 2013

J65nko said:
% for X in $(cat aap3) ; do printf "IP: $X : " ; dig +short -x ${X}; echo; done

IP: 14.201.155.217 : 14-201-155-217.static.tpgi.com.au.
IP: 76.120.18.153 : c-76-120-18-153.hsd1.co.comcast.net.
IP: 81.17.19.117 :
IP: 96.252.188.5 : pool-96-252-188-5.tampfl.fios.verizon.net.[/code]

I too am usually unnerved by unexplained connections - especially ones that seem to resolve to residential addresses (as appears to be the case with some of the indicated "dig -x" addresses). I'm inclined to agree that BitTorrent is involved...

throAU · Sep 9, 2013

This brings up a bit of an issue: Is it possible to identify the process ID of a socket's owner? I can't see anything in the netstat man page.

Windows has been able to do this since Windows 2000. I think it is totally reasonable and expected to be able to determine what process is generating or responding to network connections, no?

And yes, I'd put my money on BitTorrent.

ShelLuser · Sep 9, 2013

throAU said:
This brings up a bit of an issue: Is it possible to identify the process ID of a socket's owner? I can't see anything in the netstat man page.

I don't know about netstat, but sockstat(1) shows this information by default.

nslay · Sep 9, 2013

It happens for me with some NFS services:

Code:

?        ?          ?     ?  udp4   *:*                   *:*
?        ?          ?     ?  udp4   *:2049                *:*
?        ?          ?     ?  tcp4   *:778                 *:*
?        ?          ?     ?  udp4   *:955                 *:*

The second line is nfsd(8) with the remaining lines being lockd(8).

However, I'm especially confused by sockstat(1) where it notes this:

If a socket is not associated with any file descriptor, the first four columns have no meaning.

I'm guessing it's related to that. So, what does that mean? Is that even possible? Can you use a socket without a file descriptor?

This seems like a bug. Maybe something isn't being copied by fork(2)?

ShelLuser · Sep 9, 2013

nslay said:
So, what does that mean? Is that even possible? Can you use a socket without a file descriptor?

This seems like a bug. Maybe something isn't being copied by fork(2)?

Well, I'm no expert on this, but still curious enough to see for myself. And for sure, it is possible and without any signs of foul play.

For example:

Code:

# sockstat -4
?        ?          ?     ?  tcp4   127.0.0.1:47037       127.0.0.1:10023
?        ?          ?     ?  tcp4   xxx.xxx.xxx.xxx:80    xxx.xxx.xxx.xxx:47908
?        ?          ?     ?  udp4   *:2049                *:*

I started to look into port 80 and searched the remote IP address in my Apache logfiles, and what do you know:

Code:

[Sun May 26 00:18:13 2013] [info] [client xx.xx.xx.xx] (57)Socket is not connec
ted: core_output_filter: writing data to the network

I also noticed normal connections on other websites. So then I started to look into this error message, and that also shed some light on this, see for example this mailing list message. In short: people connecting to your server and cancelling requests.

So then I looked into sockstat again (ever since I started using FreeBSD and learned about sockstat I never touched netstat again) and what do you know:

Code:

?        ?          ?     ?  tcp4   myipaddress:80      yy.yy.yy.yy:56825
?        ?          ?     ?  tcp4   myipaddress:25      zz.zz.zz.zz:49278
?        ?          ?     ?  udp4   *:2049                *:*

All of a sudden the previous port 80 entry is gone and replaced by these two (all at the time of writing this message, writing doesn't take too long; looking things up does). So then I checked out this remote IP address associated with port 80 too and guess what? No error in my Apache logfile. Only a legit entry in the access logfiles.

The port 25 address turned out to be blocked by portgrey (greylist filter; only accepts incoming e-mail at a second connection which needs to be 5 minutes later than the first).

Then there is the port 2049 entry. I could trace that one back to this sockstat entry:

Code:

root     nfsd       853   5  tcp4   *:2049                *:*

Which makes some sense to me since I have been experimenting with NFS on this machine. I can't rule out that some connection went stale.

The others basically come and go. The only thing which puzzles me sometimes is this:

Code:

?        ?          ?     ?  stream -> ??
?        ?          ?     ?  stream -> ??
?        ?          ?     ?  stream -> ??
?        ?          ?     ?  stream -> ??

I can trace the stream entry back to several programs though (postfix, dovecot, tmux, perl, even sshd) and all are related with network connections. And when there's a connection this connection can unexpectedly break.

(edit) Also good to know is that this only involves local connections. So you'll only see these when running sockstat -u (or sockstat).

So my conclusion is that this is certainly not a bug but merely a feature. In the sense that sockstat doesn't merely show you what you'd like to see, it goes all the way and also shows stale connections.

Just my 2 cents though mind you.

ShelLuser · Sep 9, 2013

Pardon for two messages quickly after each other, but I think this to be intriguing and since it provides something new I figured I'd better not edit my previous message.

See this e-mail on the mailinglist. I quote:

Code:

 > > The sockstat utility checks all file descriptors open by all processes
 > > looking for sockets, while netstat shows all kernel-level sockets. This
 > > may mismatch in many ways: a process may have closed its descriptor but
 > > TCP still needs to maintain some state like TIME_WAIT (as mentioned in
 > > the sockstat(1) man page), multiple descriptors may exist for a single
 > > socket and kernel code (like nlockmgr) may use the socket(9) API
 > > directly so there is no descriptor.

Makes perfect sense to me. Just a shame that this information is no longer present in the sockstat(1) manualpage these days (either that or I overlooked it several times, which could be; I'm on a late lunchbreak).

nslay · Sep 9, 2013

ShelLuser said:
Pardon for 2 messages quickly after each other, but I think this to be intriguing and since it provides something new I figured I'd better not edit my previous message.

See this e-mail on the mailinglist. I quote:

Code:

> > The sockstat utility checks all file descriptors open by all processes > > looking for sockets, while netstat shows all kernel-level sockets. This > > may mismatch in many ways: a process may have closed its descriptor but > > TCP still needs to maintain some state like TIME_WAIT (as mentioned in > > the sockstat(1) man page), multiple descriptors may exist for a single > > socket and kernel code (like nlockmgr) may use the socket(9) API > > directly so there is no descriptor.

Makes perfect sense to me. Just a shame that this information is no longer present in the sockstat(1) manualpage these days (either that or I overlooked it several times, which could be; I'm on a late lunchbreak).

I agree. If the socket is closed and in a TIME_WAIT state, then the behavior follows from sockstat(1). However, nfsd(8) and lockd(8) both have the sockets opened and are listening. I see nothing in the above post or in sockstat(1) that suggests that the first four columns should be meaningless in this case. I think it's reasonable that sockstat(1) produce the expected result in this case. In other words, I still think this is a bug.

By the way, does TIME_WAIT really deserve file tags even though it's not a macro?

ShelLuser · Sep 9, 2013

nslay said:
I think it's reasonable that sockstat(1) produce the expected result in this case. In other words, I still think this is a bug.

Well, this is where my expertise runs out, but I agree; it's either specific behaviour of the NFS daemon or a bug.

nslay said:
By the way, does TIME_WAIT really deserve file tags even though it's not a macro?

For what's it worth; when it comes to using the tags I always ask myself if using them would make the message better readable. If the answer is yes then I use them, no matter if we're talking file paths, macro's, parameters or simply numbers. So in this case my answer would be yes; it makes TIME_WAIT easier readable. But; just my 2 cents there mind you.

Possible server intrusion

Administrator