Netflow on the Endpoint?

So you probably most commonly think of Netflow as a router feature (where you can monitor chokepoints to identify top talkers), but over the long holiday weekend I've used it as a way to monitor behind crappy closed source SOHO APs that don't allow you to turn off NAT. I've started running netflow on some of the Linux endpoints and just for grins I enabled it on my work laptop. On the various systems I point them at a single Netflow receiver, but on my laptop I obviously point them a local receiver.

On Ubuntu/Debian it is as simple as:

apt-get install flow-tools softflowd fprobe

Softflowd and fprobe both allow you to generate Netflow datagrams to send to a netflow receiver such as flow-tools. In both cases the receivers have single configuration files in /etc/default that allow you to specify the interface to monitor and the address and UDP port of the receiver.

root@e6400:/var/flows/2010/2010-11/2010-11-28# cat /etc/default/softflowd
#
# configuration for softflowd
#
# note: softflowd will not start without an interface configured.

# The interface softflowd listens on.
INTERFACE="eth0"

# Further options for softflowd, see "man softflowd" for details.
# You should at least define a host and a port where the accounting
# datagrams should be sent to, e.g.
# OPTIONS="-n 127.0.0.1:9995"
OPTIONS="-n 127.0.0.1:3333"

root@e6400:/var/flows/2010/2010-11/2010-11-28# cat /etc/default/fprobe
#fprobe default configuration file

INTERFACE="wlan0"
FLOW_COLLECTOR="127.0.0.1:3333"

#fprobe can't distinguish IP packet from other (e.g. ARP)
OTHER_ARGS="-fip"

Since neither of these probes allow you to monitor multiple interfaces I'm having to use both to monitor my wired and wireless interfaces.

Next, I configured flow-tools by editing /etc/flow-tools/flow-capture.conf with a single line:

-w /var/flows -n 275 -N 3 127.0.0.1/127.0.0.1/3333

This stories the netflow data in the /var/flows directory and the receiver listens on 127.0.0.1:3333 which corresponds to what we had above

I found that if the directory isn't present the daemon will fail to start. This error message will show up in the logs but not on the console

When I go into work tomorrow and I plug into my dock, this should do the trick, but we'll see.

The only thing I'm not sure about is whether the daemons will correctly handled a downed interface so I may have to manually start the daemons.

Now you'll see the files are created

root@fe6400:/var/flows/2010/2010-11/2010-11-28# ls -alt | head -20
total 248
drwxr-xr-x 2 root root 4096 2010-11-28 18:10 .
-rw-r--r-- 1 root root 88 2010-11-28 18:10 tmp-v05.2010-11-28.181027-0500
-rw-r--r-- 1 root root 96 2010-11-28 18:10 ft-v05.2010-11-28.180515-0500
-rw-r--r-- 1 root root 96 2010-11-28 18:05 ft-v05.2010-11-28.180001-0500
-rw-r--r-- 1 root root 96 2010-11-28 18:00 ft-v05.2010-11-28.175448-0500
-rw-r--r-- 1 root root 96 2010-11-28 17:54 ft-v05.2010-11-28.174935-0500
-rw-r--r-- 1 root root 96 2010-11-28 17:49 ft-v05.2010-11-28.174422-0500
-rw-r--r-- 1 root root 96 2010-11-28 17:44 ft-v05.2010-11-28.173909-0500
-rw-r--r-- 1 root root 96 2010-11-28 17:39 ft-v05.2010-11-28.173356-0500
-rw-r--r-- 1 root root 96 2010-11-28 17:33 ft-v05.2010-11-28.172843-0500
-rw-r--r-- 1 root root 96 2010-11-28 17:28 ft-v05.2010-11-28.172330-0500
-rw-r--r-- 1 root root 96 2010-11-28 17:23 ft-v05.2010-11-28.171816-0500
-rw-r--r-- 1 root root 96 2010-11-28 17:18 ft-v05.2010-11-28.171304-0500
-rw-r--r-- 1 root root 96 2010-11-28 17:13 ft-v05.2010-11-28.170751-0500
-rw-r--r-- 1 root root 96 2010-11-28 17:07 ft-v05.2010-11-28.170237-0500
-rw-r--r-- 1 root root 96 2010-11-28 17:02 ft-v05.2010-11-28.165725-0500
-rw-r--r-- 1 root root 96 2010-11-28 16:57 ft-v05.2010-11-28.165212-0500
-rw-r--r-- 1 root root 346 2010-11-28 16:52 ft-v05.2010-11-28.164659-0500
-rw-r--r-- 1 root root 806 2010-11-28 16:46 ft-v05.2010-11-28.164146-0500

And most of these are empty. I should adjust the the rotation should it creates smaller files.

But I can see what sort of activity my laptop was up to while I was dealing with my youngest son's terrible in-between-two-and-three during supper.

root@e6400:/var/flows/2010/2010-11/2010-11-28# flow-cat ft-v05.2010-11-28.164146-0500| flow-print

srcIP dstIP prot srcPort dstPort octets packets
172.16.1.1 172.16.1.145 17 67 68 576 1
0.0.0.0 255.255.255.255 17 68 67 656 2
172.16.1.145 192.168.1.1 17 58772 53 61 1
192.168.1.1 172.16.1.145 17 53 34490 100 1
192.168.1.1 172.16.1.145 17 53 38384 51 1
192.168.1.1 172.16.1.145 17 53 39480 51 1
172.16.1.145 192.168.1.1 17 39480 53 51 1
172.16.1.145 192.168.1.1 17 34490 53 61 1
172.16.1.145 192.168.1.1 17 53304 53 60 1
192.168.1.1 172.16.1.145 17 53 34640 100 1
172.16.1.145 192.168.1.1 17 38384 53 51 1
192.168.1.1 172.16.1.145 17 53 58772 100 1
172.16.1.145 192.168.1.1 17 34640 53 61 1
192.168.1.1 172.16.1.145 17 53 38674 76 1
172.16.1.145 224.0.0.251 17 5353 5353 2611 9
172.16.1.145 192.168.1.1 17 38674 53 60 1

Walking through .nessus files with Python xml.etree.ElementTree

Back when I used to teach Tenable's Nessus course I was always surprised how most folks, if to perform additional analysis or manipulation scan results, used Excel to process NBE/NSR files rather than using XML. So I added some simple examples of how to use Python and Ruby to the course and how easy it is you write a simple parser. In my slides I believe I used expat which requires you to build a list/hash of the data you extract as you encounter the start or end of the element. This works and expat or Sax are the primary parsers I've used over the years, although I did recently discover minidom.

If you click on the capture above (blogger doesn't handle XML or code that well) you'll see that after parsing the .nessus file and starting with the top node (I'm not sure why I had to call getroot())
I navigated through the different nodes within the .nessus file starting with Report, ReportHost, and ending with ReportItem where I extracted the port, and plugin id so that when you run the script you get this for all the

192.168.20.3
- 22/tcp|0
- 1241/tcp|0
- 111/tcp|0
- 1243/tcp|0
- 111/tcp|10223
- 59370/tcp|11111
- 111/tcp|11111
- 33145/udp|11111
- 111/udp|11111
- 1241/tcp|22964
- 22/tcp|22964
- 1241/tcp|10863
- 1241/tcp|35291
- general/tcp|12634
- general/tcp|22869
- 59370/tcp|25221

This code snippet isn't terribly useful but it illustrates the API and how it is very straightforward to parse .nessus files.



Some Benchmarks
Besides being much cleaner (IMHO) the nice thing about ElementTree is that there is a C implementation. This is a 2.0 MB file that consists of 4 scans and the scans include the results from only a handful of targets.

On Python 2.5 / Cygwin on my Ideapad S10-2 (Windows XP SP3)

Pure Python
real 0m4.250s
user 0m3.155s
sys 0m0.357s

C Version
real 0m1.422s
user 0m0.405s
sys 0m0.374s

I wanted to do a comparison with Win32 on Python 2.6 on the same system but I was unable to get timeit.exe working from the Windows 2003 Resource Kit.


NOTE: ElementTree is available in Python 2.5 and later and you should be ashamed if you are using anything older than that.

Squid v. DDoS

This is hardly more than worth a tweet but longer than 140 chars but Squid Defense against DDoS caught my eye.

I see two lessons here:

Using Open Source tools to respond to an incident is a crude but powerful technique that can get the job done. Try doing this with commercial products.

Environments that were properly engineered with caching and load balancing and could respond to DDoS wouldn't have to worry about the attacks in the first place.

Are you down with OSCP?

I'm generally weary (not wary!) about anything related to SSL or MITM  (and particularly SSL MITM's) but Traffic for Revoked TLSv1 Certificate is actually pretty interesting drink coffee while only the 1 year old is up and toddling around, catch up on blogs on your new Netbook activity.

And Richard's traffic dissection, reminded its been weeks (months?) since I've fired up Wireshark. Tcpdump, every other day, but Wireshark, not so much lately.

Merry Christmas!

rsyslog vs. syslog-ng

I really like syslog-ng, but I just ran across rsyslog tonight. It built on Debian 4.0 (failed on OpenBSD 4.4) but I didn't get it running yet. Will give it a try.

Rsyslog is an enhanced multi-threaded syslogd. Among others, it offers support for on-demand disk buffering, reliable syslog over TCP, SSL, TLS, and RELP, writing to databases (MySQL, PostgreSQL, Oracle, and many more), email alerting, fully configurable output formats (including high-precision timestamps), the ability to filter on any part of the syslog message, on-the-wire message compression, and the ability to convert text files to syslog. It is a drop-in replacement for stock syslogd and able to work with the same configuration file syntax.

I could care less about TLS Encrypted syslog but some of the other features like Handling a massive syslog database insert rate with Rsyslog look sort of interesting.

Database updates are inherently slow when it comes to storing syslog messages. However, there are a number of applications where it is handy to have the message inside a database. Rsyslog supports native database writing via output plugins. As of this writing, there are plugins available for MySQL an PostgreSQL. Maybe additional plugins have become available by the time you read this. Be sure to check.

In order to successfully write messages to a database backend, the backend must be capable to record messages at the expected average arrival rate. This is the rate if you take all messages that can arrive within a day and divide it by 86400 (the number of seconds per day). Let's say you expect 43,200,000 messages per day. That's an average rate of 500 messages per second (mps). Your database server MUST be able to handle that amount of message per second on a sustained rate. If it doesn't, you either need to add an additional server, lower the number of message - or forget about it.

ipt-netflow

Speaking of Netflow, I just ran across ipt-netflow

Very fast and effective Netflow exporting module for Linux kernel. Designed for Linux router with heavy network load. It is iptables module, but not using conntrack for performance reasons.

And from the docs

===========
= RUNNING =
===========

1. You can load module by insmod like this:
# insmod ipt_NETFLOW.ko destination=127.0.0.1:2055 debug=1

Or if properly installed (make install; depmod) by this:
# modprobe ipt_NETFLOW destination=127.0.0.1:2055

See, you may add options in insmod/modprobe command line, or add
them in /etc/ to modules.conf or modprobe.conf like thus:
options ipt_NETFLOW destination=127.0.0.1:2055

2. Statistics is in /proc/net/stat/ipt_netflow
To view slab statistics: grep ipt_netflow /proc/slabinfo

3. You can view parameters and control them via sysctl, example:
# sysctl -w net.netflow.hashsize=32768

4. Example of directing all traffic into module:
# iptables -A FORWARD -j NETFLOW
# iptables -A INPUT -j NETFLOW
# iptables -A OUTPUT -j NETFLOW

Oh if I had a week to kill, to do a complete bakeoff of Linux and BSD user/kernel space implementations.

Any Netflow probes for OpenBSD 4.4?

So I built a new OpenBSD 4.4 box on real hardware (Optiplex GX-100/128MB) so I could ensure the ratio of end hosts to forwarding devices remains less than one on my home network. The great thing about OpenBSD is they haven't touch the installer in the ten years I've used it and the network install always works like a charm assuming you don't fat finger the mirrors.

Pull down some packages, tweak the pf.conf (but forget to enable IP forwarding in sysctl.conf) fire the last system I built back in 2001 or so (K7 with 1.2GB) and then decided to add netflow. The obvious choice is pfflowd which fails to compile and ports says is broken. Spin my wheels around net/if_pfsync.c, browse the diffs. Hmmm... maybe this is harder, screw up the patch for 4.3 I find from Next (well actually I keep bouncing back and forth) try fprobe. 

No luck. Probably a pthreads issue, this might actually work, though?

Finally softflowd which compiles and appears to work, but for some weird reason I'm not seeing the traffic on the wire although it is definitely recording flows. Netstat shows it is has bound the sockets but not traffic is being generated. No firewall drops. Try disabling PF, nothing. Routing table fine. 

Weird.

RealEyes

Sometimes just browsing Freshmeat can lead to some interesting discoveries like tonight I ran across Realeyes

The Realeyes analysis engine is a C library of functions that maintain state information and analysis results about streams of data. Applications may be built on it to search for complex patterns and then output information about the data or even transform it. It has been tested on several Linux distributions but should run on any Unix system.

The first application that has been developed using the library is a network Intrusion Detection System (IDS). It reassembles sessions (including both halves of a TCP session) from live or captured network traffic and analyzes them for patterns.

The detected records are transferred to a database interface and inserted into a PostgreSQL database. The database also maintains configuration information which can be sent to the IDS hosts for dynamic reconfiguration. The database interface can communicate with one or more hosts.

The user interface is a Java application using the Standard Widget Toolkit from the Eclipse project, which has been tested on several Linux distributions and Microsoft Windows. It is used to administer the application as well as to analyze detected network traffic and create reports for supporting a secure environment.

All Realeyes technologies are licensed under GPLv3 and are originally developed on the GNU/Linux v2.6 operating system.

And there is a blog, too -- because everything must have a blog, right?

Finally Cured My Home NAT Pain

Although in the past I've run into issues (on the job) where NAT (or multiple webs proxy chains) made troubleshooting connectivity issues difficult, closer to home my crappy WRT54Gv5 (VxWorks based, I believe which unlike the older Linux 54g's does not allow you to disable NAT) has not only obscured my wireless activity but made access controls difficult. Although I had a Nessus scanner running on the WLAN segment all the PVS data was obscured by the router.

I'm sure there are probably ways I could use different username/password combinations with Squid to provide limited access to my kids computers and full access to anything else, but I'm a bit lazy and its a painful enough just to keep changing the proxy configs on the laptops my wife uses.

So I was about ready to order an Honest to God AP so I could finally get useful PVS data, until I finally realized I had a perfectly good 851w sitting in the closet. I've always found wireless painfully difficult to configure on this router. I think the problem which I finally overcame last night, was configuring a subinterface on the radio interface which the right VLAN, but that is probably worth a separate blog. With ACLs on the 851 only allowing DNS (the Debian dnsmasq package works great) and TCP/3128 to the firewall. I know can distinguish distinguish my kids browsing habits. Not that it is hard. As you can see from the squidview screenshot my daughter is playing dragontales games on pbskids.org. What's the harm in opening up all the entire .gov domain, anyway?

This week I'll span the 851's WAN port on my 2940 and plop in a PVS sensor and I'll have even better visibility!

Detecting CitectSCADA with Nessus

For the most part, I try not to put product specific content on my blog but given all the bellyaching about CORE-2008-0125 and to avoid "advertising" on SCADASEC-L it is worth noting that this plugin now joins the other 36 SCADA Plugins in the Direct (soon to be renamed Professional) Feed. The majority these were developed by Digital Bond in 2006.

Anatomy of An SSH Brute Force Attempt (In Pictures)

Just for fun I decided to turn on SSH on the dirty interface on my highly secure Debian firewall to see what shows up in LCE.

First I filter on all TCP/22 activity



I'm actually most concerned with the valid logins first so I check them



Whew. Only 2 logins, those are probably OK.I could check and see who they are but I'm too lazy and I'm anxious to get to the invalid logins:



But I want to get a better sense of time. Big spike right has I was recovering from getting the kids ready for school.


I back up to look at the two types of events I'm seeing during this period of attack: failed passwords and invalid users. I actually discovered the other day that there is an SSH error message for failed login attempts where the username is valid but when it doesn't match the address in the AllowUser option of sshd_config. I thought that was cool. Not sure why I'm not seeing these here.



and





Yep, Italy again

inetnum:        141.250.0.0 - 141.250.255.255
netname:        UNIPG-NET
descr:          Universita' degli Studi di Perugia
descr:          Centro Ateneo Servizi Informatici, CASI
country:        IT
admin-c:        OG6-RIPE
tech-c:         FG757-RIPE
status:         ASSIGNED PI
remarks:        Perugia Academic and Research Network
mnt-by:         GARR-LIR
source:         RIPE # Filtered

The Cheap Thrill of Using Enterprise Security Tools to Monitor Your Kid's Surfing Habits

One of the nice things about working for a security vendor is being able to able to run cool product on your home network (like when I brought a Cisco 7120 home!). And given the complexity of my home network (4-5 routing/forwarding devices, 2-3 switches, and 10-20 hosts up at any given time) it's not as much overkill as you might think.

The image below shows a 5-day view of my network events using LCE.

The 2nd row is the "firewall" category which primarily consists of syslog's from my two Cisco 851's. The events drop off 2-3 days ago when I disabled the ACLs and forgot to turn them back on.

Oops.



The last row is "web" which you'll see a spike during the last day.

What happened is I finally qot squid up an running for my son so he could use the Linux box in his room. Yeah normally a bad idea but probably not relevant if have security tools that cost as a midsize car watching your network.

If I drill down on the "web" events and find that they are all from squid logs (access.log) which I'm monitoring with an LCE client.



Most of the denies were to sb.google.com so I went into the whitelist and added that so that should flatten that plot.

Drilling down on the cache hits, I can see many of the thumbnails for one of the game sites I let him go to are being cached:

1209917543.808 7 192.168.2.170 TCP_HIT/200 19581 GET http://media.y8.com/gfx/y8bartender1.jpg - NONE/- image/jpeg
1209917543.870 52 192.168.2.170 TCP_HIT/200 12089 GET http://media.y8.com/gfx/thumb_399.jpg - NONE/- image/jpeg
1209917543.871 4 192.168.2.170 TCP_HIT/200 15471 GET http://media.y8.com/gfx/y8family_restaurant.jpg - NONE/- image/jpeg

And a one of the misses shows the sort of games my 9-year old likes to play.

1209906995.596 154086 192.168.2.170 TCP_MISS/503 1420 GET http://y8.com/tags/Killing - DIRECT/y8.com text/html

Not Alerting on Scheduled Nessus Scans with Snort (Debian-style)

So my (Debian based) firewall routes traffic for the three subnets I have behind it, each which is front-ended by either a Cisco 851 or a Linksys AP. On one of the subnets I have a VM running Tenable Security Center which runs daily Nessus scans from two Nessus scanners.

The first scanner (on my Son's Ubuntu 7.10 Optiplex) scans the non-Linksys devices connected to the AP, both wired and wireless. (I exclude the traffic because even traffic through the APs hangs the WRT54G and sometimes the WET54G (bridge). The second scans its local subnet and the other Cisco-protected subnet. I will add a third Nessus scanner behind this subnet that scans back so that on the non-Linksys segments I have a perspective of scans both inside and outside the Firewall (meaning router ACLs)

But this obviously causes even the default Debian Snort ruleset to alert. So tweaking this knowledge base entry to the pecularities of a Debian (4.0) Snort install, I did the following:

1) Create an /etc/snort/excludes.conf that has BPF (not host scanner-ip) to ignore scans to/from that host

2) Modify the PARAMETERS variable in /etc/defaults/snort

# Parameters for the daemon
# Add any additional parameteres here.
PARAMS="-F /etc/snort/excludes.conf -m 027 -D -d "

3) Restart snort as usual
Simple, almost too simple to even blog about...

Outing PeerTAB

Sometime back when it was still cold (probably in April, I around the time we had that last really wet snowfall of the year, perhaps even when I shot this video of my kids beating up our snowman -- it is amazing how many "snowman beating" videos are in Youtube) I got this wacky idea while stuck in traffic.

So the question I had, was why haven't we seen any global (or even Enterprise-wide) log/traffic/alert sharing tools which use P2P technologies? Why haven't we seen a decentralized Dshield? The answer, several months later, was PeerTAB.

Obviously there is no code yet, but there are a reasonably refined set of requirements. Just to be clear, this is not another log analysis tool, or a SIM. It is basically a [hopefully] thin layer on top of JXTA that would allow stuff like Snort logs, netflow-data (lets say from flow-tools), mod-security logs, or whatever to be contributed and searched across a global (or localized) P2P network. I had some initial success in using JRuby and JXTA but there is a lot more to do, so stay tuned.

And if you have any interest in learning about P2P Networking APIs and have some minimal of level of Java/Ruby competence and want to get involved in the project, let me know, let me know.

BWM-NG + Ruby for Gathering Network Stats

BWM-NG is a great tool for getting real time stats, but by using the CSV output you can capture historical data as well.

# bwm-ng -o csv -t 2500 -c 1 -C,
1186367009,em2,198.09,50.12,248.21,126,498,0.40,0.80,1.19,2,1,0.00,0.00,0,0
1186367009,lo0,0.00,0.00,0.00,0,0,0.00,0.00,0.00,0,0,0.00,0.00,0,0
1186367009,total,198.09,50.12,248.21,126,498,0.40,0.80,1.19,2,1,0.00,0.00,0,0

So to make use of this, we'll need some sort of Time/Date API to convert the etime into something useful as well as the CSV parser.

#!/usr/bin/env ruby
require 'csv'
require 'pp'
require 'date'

bytes={}
packets={}
errors={}
cmd = 'bwm-ng -o csv -t 2500 -c 1 -C,'
p = IO.popen(cmd) do |f|
   f.each_line do |g|
       h = CSV::parse(g).flatten
       $dtg = Time.at(h[0].to_i).to_s
       interface = h[1]
       bytes[interface] =  h[2..4]
       packets[interface] =  h[7..9]
       errors[interface] = h[14..15]
   end
end

puts "Date: #{$dtg}"
bytes.keys.sort.each do |i|
   puts "\nInterface: #{i} (TX/RX/Total)"
   print "Bytes:"
   pp bytes[i]
   print "Packets:"
   pp packets[i]
   print "Errors:"
   pp errors[i]
end

And the output looks like

# ./rbbw.rb
Date: Mon Aug 06 03:35:17 +0000 2007

Interface: em2 (TX/RX/Total)
Bytes:["128.88", "76.37", "205.25"]
Packets:["0.80", "1.19", "1.99"]
Errors:["0", "0"]

Interface: lo0 (TX/RX/Total)
Bytes:["133.65", "133.65", "267.30"]
Packets:["1.59", "1.59", "3.18"]
Errors:["0", "0"]

Interface: total (TX/RX/Total)
Bytes:["262.53", "210.02", "472.55"]
Packets:["2.39", "2.78", "5.17"]
Errors:["0", "0"]

Where these are obviously rate values (per second) for a very boring FreeBSD VM.

What ever happened to CIDF?

Was it SANS 98 that I saw some slides on this? I suppose CIDF was another one of those U.S. Government-sponsored research projects (the kind that small companies salivate over and big contractors squander) that went nowhere, produced nothing tangible? There better have been a real implementation that inspired CISL and not the reverse? Oh, yeah and while you are at it, please tell me the point of RFC 4766?

Which is easier? A pure ruby netflow parser or flow-tools bindings that don't suck?

So I pared down one of the examples in vflow (ruby bindings to flow-tools)

#!/usr/bin/ruby

require 'Vflow'
require 'socket'

TESTFILE=ARGV[0]
def dumpvflowrec(r)
puts "=============="
print "srcaddr  #{IPSocket.getaddress(r.srcaddr)} -> "
puts  "dstaddr  #{IPSocket.getaddress(r.dstaddr)}"
puts "srcport   #{r.srcport} dstport #{r.dstport}"
puts "prot      #{r.prot}    tos #{r.tos}"
end

x = Vflow.new()
x.open(TESTFILE)

count = 0
x.each() {|r|
dumpvflowrec(r)
count += 1
puts count
}

I guess I found the bug the author mentions on the website.

franz-g4:/tmp mdfranz$ ruby vftest.rb ft-v05.2007-04-30.064501-0500
==============
srcaddr  202.97.238.199 -> dstaddr  24.136.7.235

SNIP

==============
srcaddr  24.136.7.235 -> dstaddr  64.233.163.19
srcport   58911 dstport 443
prot      6    tos 0
197
==============
srcaddr  207.172.3.8 -> dstaddr  24.136.7.235
srcport   53 dstport 57738
prot      17    tos 0
198

ruby(674) malloc: *** Deallocation of a pointer not malloced: 0x320110;
This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug

Of course not like I could do any better, I know better than to touch anything in C.

What are the odds the Python flowtools will be more robust.

Pretty good, I reckon.

just rtfm ng_netflow(4)

On FreeBSD 6.2, to use the built in kernel netflow export capability, as long as the ng_ether is loaded, the example at the bottom of ng_netflow(4) should work just fine. However, I did put up a a wiki page on the topic, after googling all over. It will be interesting to see what sort of performance is possible with it

Painless Non-Enterprise Netflow

Tonight I released another ubuntutrinux-core snaphot that include fprobe and a few tools from flow-tools. I'll spare the introduction to Netflow except to comment why this might useful for Linux/Trinux (its obvious for routers!) as opposed to other network monitoring tools.

In terms of data your are getting about what you would get with a port logger such as ippl or other port listeners that log connections from hosts

root@gx620:/tmp# flow-cat biglast | flow-print | head
srcIP dstIP prot srcPort dstPort octets packets
24.136.0.111 239.255.255.250 2 0 0 32 1
24.136.0.189 239.255.255.250 2 0 0 32 1
82.211.81.145 24.136.x.y 17 123 123 76 1
24.136.2.30 239.255.255.250 2 0 0 32 1
24.136.2.67 224.0.0.251 2 0 0 32 1
24.136.2.67 239.255.255.253 2 0 0 32 1
10.48.120.1 224.0.0.1 2 0 0 28 1
24.136.0.163 239.255.67.250 2 0 0 32 1
24.136.19.48 224.0.0.253 2 0 0 32 1

Notice this is mostly multicast cruft on RCN with the exception of NTP traffic to the Ubuntu time source. But big deal. Some of you may remember (back in the day!) a NSWC tool called SHADOW (where Northcutt and Irwin made their claim to fame) that was basically a collection of Perl scripts that managed tcpdump file capture and viewing through a web interface.

Well flow-tools allows you to a lot of the same stuff with much less overhead and all from the command-line

$ flow-cat biglast | flow-stat -f5 -S 1 | head -25

# Args: flow-stat -f5 -S 1
#
#
# port flows octets packets
#
80 2464 12632818 162315
53 1099 214515 3053
1026 414 250208 710
32768 405 131967 766
1027 314 155033 314
123 186 14136 186
443 177 333745 3020
7 175 19075 175
5222 141 36842 370
3408 120 15892 220

Basically you "flow-cat" the saved file to a number of different tools, flow-stat being the most useful for me. Not terribly surprising HTTP is at the top nor the 1026 to my firewall. Damn cable.

$ flow-cat biglast | flow-stat

#
# Fields: Total
# Symbols: Disabled
# Sorting: None
# Name: Overall Summary
#
# Args: flow-stat
#
Total Flows : 13168
Total Octets : 380416991
Total Packets : 517136
Total Time (1/1000 secs) (flows): 222894382
Duration of data (realtime) : 34320
Duration of data (1/1000 secs) : 364934
Average flow time (1/1000 secs) : 16926.9733
Average packet size (octets) : 735.6227
Average flow size (octets) : 28889.5043
Average packets per flow : 39.2722
Average flows / second (flow) : 36.1758
Average flows / second (real) : 0.3837
Average Kbits / second (flow) : 8360.8132
Average Kbits / second (real) : 88.6753

After concatenating all this data into a single file, it only took about about 800k for about 10 hours of traffic.

And how did I kick all this off?

First I ran fprobe on Trinux and made sure it was working by testing it out with EHNT
which is the quickest way (it took me a while to wander through the flow-tools manpages, and the ubuntu startup script (in /etc/init.d) for capturing flows didn't work. I'm using pcap to get this but there is a version of fprobe that can generate flow from iptables.

#fprobe -u nobody collector-ip:collector port

BTW, netflow uses UDP. You can sniff to make sure the flow updates are being sent. And then on the server (you'll want to be more restrictive on the local and remote ports, the 0's)

#flow-capture -w /raid/flows/ 0/0/4444 -S20

which creates the directory hierarchy like:

root@gx620:/raid/flows/2007# ls -alR | less

.:
total 0
drwxr-xr-x 3 root root 72 2007-04-29 12:04 .
drwxr-xr-x 3 root root 72 2007-04-29 12:04 ..
drwxr-xr-x 3 root root 80 2007-04-29 12:04 2007-04

./2007-04:
total 2
drwxr-xr-x 3 root root 80 2007-04-29 12:04 .
drwxr-xr-x 3 root root 72 2007-04-29 12:04 ..
drwxr-xr-x 2 root root 2208 2007-04-29 22:15 2007-04-29

./2007-04/2007-04-29:
total 326
drwxr-xr-x 2 root root 2208 2007-04-29 22:15 .
drwxr-xr-x 3 root root 80 2007-04-29 12:04 ..
-rw-r--r-- 1 root root 1145 2007-04-29 12:15 ft-v05.2007-04-29.121249-0500
-rw-r--r-- 1 root root 4500 2007-04-29 12:30 ft-v05.2007-04-29.121904-0500
-rw-r--r-- 1 root root 2153 2007-04-29 12:45 ft-v05.2007-04-29.123001-0500


Of course there are tons more options (the -S20 just says write a status report to syslog every 20 minutes like:

Apr 29 22:20:00 localhost flow-capture[27718]: STAT: now=1177903200 startup=1177867131 src_ip=192.168.100.1 dst_ip=192.168.169.162 d_ver=5 pkts=2918 flows=13828 lost=1 reset=0 filter_drops=0