Solved Logs flooded with messages

Argentum

Argentum

Dear all,

For some time now, I am getting strange error messages in the logs -
Code: 
Jan 14 13:14:32 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 316 to 99 packets/sec
Jan 14 13:25:43 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 136 to 96 packets/sec
Jan 14 13:38:56 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 136 to 108 packets/sec
Jan 14 13:38:57 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 130 to 105 packets/sec
Jan 14 13:44:15 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 124 to 103 packets/sec
Jan 14 13:44:23 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 131 to 105 packets/sec
Jan 14 13:46:46 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 343 to 98 packets/sec
Jan 14 13:46:50 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 224 to 108 packets/sec

These seem to come randomly during a day. I did also run tcpdump with icmp6 capture filter, there seem to be no packets with these timestamps. For example the were no icmp6 packets captpured at Jan 14 13:46:46.

There are no such packet floods in the Wireshark log. Is there anybody who is experiencing this? Also, please advise what to check next to solve this.

Did not look into the logs for a long time, but seems that appeared after upgrade to 14.2. Still, it is hard to believe that this is actually causing this.
 
They may be filtered by the firewall before arriving that's why you don't see them if you tcpdump the interface.

icmp6-type 137 is for Redirect Message

This is example from OpenBSD:

pass in quick on em0 inet6 proto ipv6-icmp from any to
{ ( em0 ), ff02::/16 } icmp6-type { 128, 133, 134, 135, 136, 137 } keep state
Click to expand...
 
VladiBG said:
They may be filtered by the firewall before arriving that's why you don't see them if you tcpdump the interface.

icmp6-type 137 is for Redirect Message
Click to expand...
I am not running pf on that machine currently and ipfw is passing all.
 
VladiBG said:
They may be filtered by the firewall before arriving that's why you don't see them if you tcpdump the interface.

icmp6-type 137 is for Redirect Message

This is example from OpenBSD:
Click to expand...
Started your tcpdump command, but this may take half a day when this happens again. Quiet at the moment

Code: 
root@Testsystem ~# tcpdump -nn -vv -i igb0 "icmp6 && ip6[40] == 137"
tcpdump: listening on igb0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
 
VladiBG said:
For ipfw you can use log interface but if you say that it pass all traffic it shoud be visible also on the tcpdump when capturing the traffic on the interface
Click to expand...
I think so also.

Will come back tomorrow and see if the tcpdump captures someting.
 
Argentum said:
Started your tcpdump command, but this may take half a day when this happens again. Quiet at the moment

Code: 
root@Testsystem ~# tcpdump -nn -vv -i igb0 "icmp6 && ip6[40] == 137"
tcpdump: listening on igb0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
Click to expand...
Now it seems that I have no hits with this rule

Code: 
root@Testsystem ~# tcpdump -nn -vv -i igb0 "icmp6 && ip6[40] == 137"
tcpdump: listening on igb0, link-type EN10MB (Ethernet), snapshot length 262144 bytes

However, in the logs there are many new messages again

Code: 
Jan 15 19:58:59 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 236 to 104 packets/sec
Jan 15 19:59:00 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 371 to 100 packets/sec
Jan 15 19:59:01 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 358 to 104 packets/sec
Jan 15 19:59:02 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 170 to 104 packets/sec
Jan 15 19:59:34 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 392 to 107 packets/sec
Jan 15 19:59:35 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 261 to 105 packets/sec
Jan 15 19:59:36 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 184 to 99 packets/sec
Jan 15 20:00:20 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 115 to 100 packets/sec
Jan 15 20:00:30 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 117 to 96 packets/sec
Jan 15 20:02:43 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 234 to 98 packets/sec
Jan 15 20:02:44 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 179 to 105 packets/sec
Jan 15 20:02:55 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 137 to 102 packets/sec

Some of these are really interesting

Code: 
Jan 15 09:56:27 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 2642 to 92 packets/sec
Jan 15 09:56:28 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 2657 to 106 packets/sec
Jan 15 09:56:28 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 3107 to 101 packets/sec
Jan 15 09:56:30 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 3232 to 106 packets/sec
Jan 15 09:56:34 Testsystem kernel: Limiting ICMPv6 neighbor discovery redirect output from 1500 to 103 packets/sec

How is this even possible that there are over 3000 packages per second coming in?
 
If there's another host with smaller prefix like /48 in the same local network when you try to reach him via the router the router will respond with icmp redirect. You can check with ndp -a but you will need to capture also the icmp packet to see the actual address.
 
As I understand "Limiting ICMPv6 neighbor discovery redirect output" is limiting ICMP6 outgoing packets.
Very strange that you can grab them with tcpdump.

You can try to set net.inet.icmp.icmplim_output=0 with sysctl to silences such messages.
 
A firewall isn't going to stop tcpdump(1) from capturing packages sent to the host. A local firewall isn't going to stop packages from arriving at the interface.

You can have a block in all and still capture all the packets being sent to that interface.
 
VladiBG said:
If there's another host with smaller prefix like /48 in the same local network when you try to reach him via the router the router will respond with icmp redirect. You can check with ndp -a but you will need to capture also the icmp packet to see the actual address.
Click to expand...
Seems that I have no devices smaller than 64.

And weird thing is that I am unable to catch there packages.
 
Limitation check

C: 
/* Finally, do rate limitation check. */
if (icmp6_ratelimit(&oip6->ip6_src, type, code))
   goto freeit;

with such message "Limiting ICMPv6 neighbor discovery redirect output" used in icmp6_error

C: 
/*
 * Generate an error packet of type error in response to bad IP6 packet.
 */
void icmp6_error(struct mbuf *m, int type, int code, int param)

comment before function tells that function generate icmp6 error in response to bad IP6 packet
maybe try to catch IP6 packets in periods when you see such messages?
 
dark.initr0 said:
You can try to set net.inet.icmp.icmplim_output=0 with sysctl to silences such messages.
Click to expand...
Just to note that this is for IPv4. For IPv6 there is net.inet6.icmp6.icmp6lim_output .

Code: 
root@Testsystem ~# sysctl -d net.inet.icmp.icmplim_output
net.inet.icmp.icmplim_output: Enable logging of ICMP response rate limiting
root@Testsystem ~# sysctl -d net.inet6.icmp6.icmp6lim_output:
net.inet6.icmp6.icmp6lim_output: Enable logging of ICMPv6 response rate limiting
 
The issue is solved. Eventually understood that after disabling the Bhyve virtualization environment on this machine, I had left
the net.inet6.ip6.forwarding enabled and also rtadvd running.

Since there was no internal prefixlen 96 virtual network any more, the redirects were triggered.

It is all quiet now in the logs. Thanks for everybody who helped to investigate this issue.

The lesson learned is that after all that was a simple issue, but remained unnoticed by me for some time.
 
You must log in or register to reply here.
Back
Top