Using Hashes Like it is 1999

This week I picked up [what I thought would be] a quick logfile analysis task. Things started out great. I took the time to look at the logfile format and generalized 4-5 different messages (with appropriate regexes to get the data I needed) generated by the security device. Next I extended a basic "logrunner" class I wrote last month for analyzing the debug output from the Intel FreeBSD drivers (basically you do some sysctl's and it dumps some kernel messages to see counters missed, received packets--much better than netstat).

In my logrunner class, you basically can "attach" various simple regex matches and a symbol and you get a nice hash back with the values you want and it hides all the low-level details of matching or handling time stamps, etc. (HINT: If you are mucking with syslog files in Ruby and you are not using the Time API, you are a fool, but I digress).

After a few hours in I thought things were going fine, before some distractions kept me from working on it again until until the next afternoon (I overconfidently estimated this would take about 4 hours from start to finish), so I was in a rush. The initial desire to develop something be a more general purpose tool and that was designed properly was replaced with the brute force, quick hack, get-r-done approach.

I ended up iterating through the output hashes output by the logrunner tool to create more hashes some with the IP address as a key, others with a username as the key. And all of this pointed to at least another hash (or two) so I ended up with something like:

blah[blah][blah] = { 1 => { a => b, c => d }, 3 => { a=> q, d => z } }

This would have been a trivial task except there was no single session identifier (or even username or IP address) on each line that I could tie the various pieces of data together. Then I kept getting confused (and alternating between |k| and |k,v| with my Ruby blocks) it took my longer than I had hoped but I was done in about 7. I had the output I wanted. Went from a few hundred megs of logs to a nice Excel-friendly CSV file. And I thought I was done.

Until Friday afternoon, when I found it some additional data was needed. Extracting the data wasn't a problem (that was done in 5 minutes), but correlating it and getting the report format was. Should I add another hash? Redefine the hashes I'd written? Five o'clock on Friday (with restless hungry kids) is not a time for clarity of thought, but this morning I realized the Ruby I was written was as unreadable as the Perl I used to write back in the day.

Spending the afternoon driving out in the snow which turned to sleet which turned to rain finally beat some sense into me. I mapped out the data on paper (this time) and did right. Came up with 6 simple classes (2 base and 4 sub) to abstract away the hashes and ended up with less than 1/10th of the lines of code in the main loop and a 1/3 of the iterations. Nothing fancy, no Ruby foo, nothing that couldn't be done in Python. And the code is actually readable. The moral of the story? If you are using hashes 4-6 levels deep you have a problem. Stop, step away from the keyboard and come up with a cleaner design. Do it right the first time, you won't regret it. Because quick hacks have a funny way of running on systems for a long, long time.

Sunbrid Recant (or searching for a browser-based iCal Replacement)

So once again I'll recant on on a previous blog.

Sunbird is crap. Events mysteriously get created and can't be deleted. Or get deleted, period. Or something weird happens with the interface. It never crashed, though. But it was annoying enough to stick with iCal.

So what I really want is a entirely browser-based iCal/Google Calendar-like tool (that means Javascript, and I'm not a Javascript programmer) tool that allows me to:

Drag (and eventually drop) activity across a daily schedule for stuff I work on.

Remember/autosuggest project names

Export events to some standard format, iCal or XML, YAML, or whatever

Summarize project activity by week/month/totoal

Yeah this probably could be done in Rails/Django but I don't want that. No databases. No webservers, but still browser based. Am I crazy? Creating a Dojo Calendar has promise, but it requires server side code. What I want is something self contained like TiddlyWiki. Where the data is all stored in the .js and can be moved around and modified.

Since I'm obviously in over my head (what else is new) this is obviously something I wouldn't want to start from scratch, but what next? Should I look at GWT or better yet pyjamas to avoid Java development? Obviously this would be (relatively) trivial to do as a traditional desktop GUI app but that is no fun.
The key is needs to be portable, lightweight, usable off line. Somebody else had to have run across this sort of problem (and solved it) before. We'll see what happens.

Tim Fite: Fun Angry Music for Coding

So this post might have been a candidate for Angry Ruby (which is dedicated to whining about using Ruby) except that Ruby is delightful (in all sincerity) compared to dealing with several thousand lines of Bourne shell scripts that use every known flavor of grep and awk known to man. And you'd think porting them from OpenBSD to FreeBSD would be trivial. Sure. So driving home on Thursday I swore to not port another line of shell and begin the long needed port to language a not invented in the 1970s.

So besides doing a lot late night coding for the 4 days (discovering that Trader Joes energy drinks are as effective as they are economical, 99 cents!) I've started listening to Over the Counter Culture which was mentioned on the Sound Opinions Mid Year Best of 2007 sow I believe It's All Right Here got some play on NPR because my son was signing from the PG-13 version after only hearing it twice. It is quite catchy.

So besides buying Run DMC's Raising Hell (when I was a junior in high school, on a school trip to Manilla, Philippines, no doubt) and casually like some of the Beastie Boys songs that made it onto KROX over the years, I've never been a rap/hip hope listener. Just not me. However, I've really taken to this album. There are 4-5 solid tracks, which is pretty high.

Probably my favorite is In Your Hair which starts (and finishes) with these two stanzas.

A king is not a president
A roof is not a resident
The truth is not self evident
When youth is on the line

The Boss is not the boss of them
The cross is on the cross again
The crim is not a government
A crime is just a crime
The rhymes in

Hay Man are not the predictable sort you'd expect from pop hip hop:

...
Break these palms
Break these pines
Break these laws (?)
Break these lines
Break these looks
Break these lease
Break these hooks
Break this heat
Break these rocks
Break this concrete
...
Without no fuel the fire don't burn bright
Without no mule the car can't turn right
Did you hear that shit?
Let's play more Fite
Because somethings fit to change
in the furnace tonight.

Good stuff. Camouflage and I've been shot are both fairly amusing.

These songs are fairly political (predictably anti-corporate, anti-consumerism, anti-Bush-era America, etc.) but these music, rhymes, lyrics, and far from that ("stop faking butter, start faking lard!") and I figure they'll

Oh yeah the complete album is free to to download. So you have no excuse not to check it out.

Two Linux Kernel Lessons of The Day

So I learned two important things about Linux kernel compilation today:

1. On Edgy (for some kernels like 2.6.16.13, where I'm still trying to get Click running on real hardware) you have to disable gcc stack protection by adding -fno-stack-protector to CFLAGS. I actually didn't figure this out the first time but when I was getting the error trying to build GRUB2 (because GRUB sucks on MacBooks, something about the keyboard working less that 25% of the time) , I figured something was up
   
2. Never, never, never, never try to compile a kernels (well maybe 2.0 might be ok) on the modular drive bay of a Dell Latitude, where I run Linux so as to not disrupt my "Corporate Image" -- Let's just say it took about 3-4 hours on a P4-1.6 with a Gig of Ram. Yikes.
   

But the day wasn't all bad, as I discovered Ruby Dnet and more importantly and interesting blog by Brian Caswell about using DL. Of course I started this trail because Ruby is so lame and doesn't have tons of stuff like Python Netifaces for easily (and cross-platform-ily) getting address information.

Comparing Tempfile API's in Python and Ruby

Much to the detriment of my productivity at work, I've been bouncing back and forth between Python and Ruby. But it makes for some interesting API's comparisons. So I'll continue the motif I started with looking at Ruby and XML-RPC API's by comparing temp files APIs.

First the security background, Insecure temp file usage is a notorious (but typically local) security flaw that can lead arbitrary code execution, , escalation of privilege, and denial of service--commonly as a result of race conditions. Take a search Packetstorm and you'll see a bunch of examples of what not to do.

Ruby
So let's take a look at the tempfile module in the Ruby Standard library:

new(basename, tmpdir=Dir::tmpdir)

Creates a temporary file of mode 0600 in the temporary directory whose name is basename.pid.n and opens with mode "w+". A Tempfile object works just like a File object.

If tmpdir is omitted, the temporary directory is determined by Dir::tmpdir provided by ‘tmpdir.rb’. When $SAFE > 0 and the given tmpdir is tainted, it uses /tmp. (Note that ENV values are tainted by default)

How would you use this?

franz-g4:~ mdfranz$ irb
irb(main):001:0> require 'tempfile'
=> true
irb(main):002:0> t = Tempfile.new("ruby")
=> #
irb(main):003:0> u = Tempfile.new("ruby")

At this point you have a regular old File object you can do what ever you want to with. So the permissions are limited to the process owner and the filename is based on the process id, which depending on the OS may be more or less predictable. But the good news is the file is deleted automatically after the script finishes execution.

franz-g4:/tmp mdfranz$ ls -al ruby*
-rw-------   1 mdfranz  wheel  0 Mar 15 20:36 ruby655.0
-rw-------   1 mdfranz  wheel  0 Mar 15 20:37 ruby655.1

Python
With Python we get an (ostensibly) more secure methods as part of the tempfile module:

mkstemp([suffix[, prefix[, dir[, text]]]])
Creates a temporary file in the most secure manner possible. There are no race conditions in the file's creation, assuming that the platform properly implements the O_EXCL flag for os.open(). The file is readable and writable only by the creating user ID. If the platform uses permission bits to indicate whether a file is executable, the file is executable by no one. The file descriptor is not inherited by child processes.

Unlike TemporaryFile(), the user of mkstemp() is responsible for deleting the temporary file when done with it.

If suffix is specified, the file name will end with that suffix, otherwise there will be no suffix. mkstemp() does not put a dot between the file name and the suffix; if you need one, put it at the beginning of suffix.

If prefix is specified, the file name will begin with that prefix; otherwise, a default prefix is used.

If dir is specified, the file will be created in that directory; otherwise, a default directory is used.

If text is specified, it indicates whether to open the file in binary mode (the default) or text mode. On some platforms, this makes no difference.

mkstemp() returns a tuple containing an OS-level handle to an open file (as would be returned by os.open()) and the absolute pathname of that file, in that order. New in version 2.3.

and for directories

mkdtemp([suffix[, prefix[, dir]]])
Creates a temporary directory in the most secure manner possible. There are no race conditions in the directory's creation. The directory is readable, writable, and searchable only by the creating user ID.

The user of mkdtemp() is responsible for deleting the temporary directory and its contents when done with it.

The prefix, suffix, and dir arguments are the same as for mkstemp().

mkdtemp() returns the absolute pathname of the new directory. New in version 2.3.

An example combining the two:

>>> import tempfile
>>> td = tempfile.mkdtemp()
>>> td
'/tmp/tmpaJ-M4J'
>>> tf = tempfile.mkstemp(dir=td)
>>> tf
(3, '/tmp/tmpaJ-M4J/tmpb7pxE4')

And even after the script executes they are still around

franz-macbook:/tmp mdfranz$ ls -al tmpaJ-M4J/
total 0
drwx------   3 mdfranz  wheel  102 Mar 17 07:08 .
drwxrwxrwt   4 root     wheel  136 Mar 17 07:06 ..
-rw-------   1 mdfranz  wheel    0 Mar 17 07:08 tmpb7pxE4

So unlike the Ruby you have to call open on the pathname + filename you got back from mkstemp/mktemp vs getting a File/IO object back.

Conclusions
As usual, the Python API documentation is more complete (and probably more importantly) consistent and in a single place. Going back and forth between the The Pragmatic Programmer and the the Ruby Standard Library Documentation and the Ruby Class and Library reference is very annoying. While I certainly didn't audit the source for Python or Ruby to determine the difficulty of race conditions and to see whether or not they are "impossible" but the use of a more pseuedo-random value for the filename (and directory name) with Python certainly raises the bar higher for Python. And combining the directory and file functions in Python. The only real downside for the Python API is that you have to clean up temporary files and directories automatically, likely a consequence of the Python API not being "truly and consistenly OO" (a common critique).

Django mysite.settings madness

For fun, I've started writing a small firewall (meaning OpenBSD PF) monitoring webapp using Django, which I actually will be blogging on it real soon now -- but first on something that seemed rather confusing about the framework, and that has tripped up others as well.

In recent (at least 0.95 and 0.95.1 but probably much earlier) you don't actually have to set the environment variable DJANGO_SETTINGS_MODULE if you running the webserver with python manage.py runserver or access a shell with python manage.py shell, but you still have to set this enivironment variable -- or call settings.configure() -- if you are access any of your Django objects from another app.

While this is clearly documented on the web site, it still gives newbies trouble, especially when running Django with mod_python under Apache. In fact a couple of months ago it stumped me for a while,.

However, what was most confusing is that everywhere (in most of the examples) the variable is always defined to mysite.settings but it is ambigous what the "mysite" should be. Is mysite my actuall site (or project) or is it always set to mysite and Django knows automatically. So settings the settings.py that gets imported by your scripts, but is mysite a path name?

If I used the value "mysite.settings", I get the following:

franz-g4:~/Documents/dev/playin/pfwatch mdfranz$ export DJANGO_SETINGS_MODULE=mysite.settings
franz-g4:~/Documents/dev/playin/pfwatch mdfranz$ python shellstuff.py
Traceback (most recent call last):
File "shellstuff.py", line 4, in ?
from monitor.models import Source,Event
File "/Users/mdfranz/Documents/dev/playin/pfwatch/../pfwatch/monitor/models.py", line 1, in ?
from django.db import models
File "/opt/local/lib/python2.4/site-packages/django/db/__init__.py", line 7, in ?
if not settings.DATABASE_ENGINE:
File "/opt/local/lib/python2.4/site-packages/django/conf/__init__.py", line 27, in __getattr__
self._import_settings()
File "/opt/local/lib/python2.4/site-packages/django/conf/__init__.py", line 52, in _import_settings
raise EnvironmentError, "Environment variable %s is undefined." % ENVIRONMENT_VARIABLE
EnvironmentError: Environment variable DJANGO_SETTINGS_MODULE is undefined.

To back up a bit, I have the following files in my project directory:

franz-g4:~/Documents/dev/playin/pfwatch mdfranz$ ls
__init__.py manage.py pfwatch.sql settings.pyc urls.py
__init__.pyc monitor settings.py shellstuff.py urls.pyc
franz-g4:~/Documents/dev/playin/pfwatch mdfranz$

So monitor is an application directory where I have my models defined and shellstuff.py is a script that I want to be able to access my models, meaning my data, which fails if I use mysite.settings. It also fails if I use pfwatch.settings both from within the current directory or if I step up a level.

So how did I finally get it sort of working? The solution is just to set DJANGO_SETTINGS_MODULE to just settings.

Of course this blog entry will make a lot more sense once you read my next one.

From GWT to pyGWT (aka pyjamas)

I'm not sure how I stumbled across pyjamas (but it might have something to do with taking the day off to look after my kids) and it is definitely unclear how I will every use this but--it is basically a Python version of Google Web Toolkit where by you define and implement your UI's in Python (or Java in the case of GWT) which in turn gets translated into JavaScript and HTML which will ultimately be processed by the browser and would then be integrated into the web application [framework] or whatever.

For example, here is the UI for email application built with pyjamas. The directory consists of

franz-macbook:~/Documents/dev/playin/pyjamas-0.1/examples/mail mdfranz$ ls
AboutDialog.py  Mail.html       MailItem.py     Mailboxes.py    TopPanel.py
Contacts.py     Mail.py         MailItems.py    Shortcuts.py    build.sh
Logger.py       MailDetail.py   MailList.py     Tasks.py        output

And you run the build.sh script to start the conversion from Python to Javascript (this is calling the master Builder.py script on your python app, in this case Mail.py)

franz-macbook:~/Documents/dev/playin/pyjamas-0.1/examples/mail mdfranz$ ./build.sh
Building 'Mail.py' to output directory 'output'
Copying: public directory
Copying: Mail.html
Copying: pygwt.js
Copying: Images
Creating: Mail.nocache.html
Creating: IE6.cache.html
Importing pyjslib
Importing Mail
Importing Shortcuts
Importing Tasks
Importing ui
Importing pygwt
Importing DeferredCommand
Importing Timer
Importing History
Importing DOM
Importing Mailboxes
Importing Contacts
Importing Logger
Importing TopPanel
Importing AboutDialog
Importing Window
Importing MailList
Importing MailItems
Importing MailItem
Importing MailDetail
Creating: Opera.cache.html
Creating: OldMoz.cache.html
Creating: Safari.cache.html
Creating: Mozilla.cache.html
Done. You can run your app by opening 'output/Mail.html' in a browser
franz-macbook:~/Documents/dev/playin/pyjamas-0.1/examples/mail mdfranz$

Which then creates the following files in the output directory:

franz-macbook:~/Documents/dev/playin/pyjamas-0.1/examples/mail/output mdfranz$ ls
IE6.cache.html          Mozilla.cache.html      Safari.cache.html       tree_open.gif
Mail.html               OldMoz.cache.html       pygwt.js                tree_white.gif
Mail.nocache.html       Opera.cache.html        tree_closed.gif

As you can see different versions of HTML/Javascript are generated for respective browsers. And what you can't see (because blogger doesn't handle raw HTML very well) is the at the pyqwt.js is what kicks things off.

If this doesn't make sense there is interesting podcast by the Google GWT folks that explains the background on how they problems of JavaScript development (lack of good tools, reliance on browser hacks, browser detection, etc.) led them to come up with the solution to do development in Java and then translate the code into JavaScript.

Ruby, Python, and an XML-RPC Server Arbitrary Shell Command Execution Flaw

So I've been playing around with the Ruby XML-RPC APIs (and finally got them talking with Python, but more on that later) and I was sort of shocked to see that a "nasty security hole" described on the xmlrpc4r site wiki (or whatever it is) by Brian Candler is still around in the most up to date versions of Ruby 1.8.x in OpenBSD 4.0 and Ubuntu Dapper LTS. Furthermore, this particular issue does not appear to be included in any of the vulnerability databases such as NVD, CERT, or Secunia. Nor is it listed on the Ruby site. Although there is a workaround, the vulnerable code is the dominant and most common approach described in tutorials--therefore probably what is used by newbies. The "secure way" is ugly and unnatural.

Since Ruby seem to be about the only HTTP implementation that has HTTP Digest Authentication on the server side, I was thinking about using it (instead of Python, which I obviously prefer) for a small VMware monitoring app but this (not to mention Ruby/Ruby XML-RPC's relatively poor security track record and I know there have been a couple of Python issues as well) has got me reconsidering this approach. Unlike Perl, I've never had any strong opposition to using Ruby and I always considered them roughly equivalent.

The Perl-ish "there is more than one way to do it" certainly can produce less readable and could result is less secure code, both in terms of Ruby itself and apps developed with Ruby. I wouldn't want to come up with any strong conclusions on the relative insecurity of the languages or APIs, but the relative attack surface of the two XML-RPC implementations is worth a peek.

So here is a quick Python script to enumerate available methods on a XML-RPC web service that has introspection enabled.

import xmlrpclib
server = xmlrpclib.Server("http://127.0.0.1:8080")
for method in server.system.listMethods():
 print method

And if we run it against the example XML-RPC server script

require "xmlrpc/server"
s = XMLRPC::Server.new
class MyHandler
 def sumAndDifference(a, b)
     { "sum" => a + b, "difference" => a - b }
 end
end
s.add_introspection
s.add_handler("sample", MyHandler.new)
s.serve

We get a whole lot...

franz-g4:~ mdfranz$ python client.py
system.listMethods
system.methodSignature
system.methodHelp
sample.to_a
sample.respond_to?
sample.type
sample.dclone
sample.sumAndDifference
sample.protected_methods
sample.eql?
sample.instance_variable_set
sample.is_a?
sample.hash
sample.to_s
sample.send
sample.class
sample.tainted?
sample.private_methods
sample.__send__
sample.untaint
sample.id
sample.inspect
sample.instance_eval
sample.clone
sample.public_methods
sample.extend
sample.freeze
sample.display
sample.__id__
sample.method
sample.==
sample.methods
sample.===
sample.nil?
sample.dup
sample.instance_variables
sample.instance_of?
sample.object_id
sample.=~
sample.singleton_methods
sample.equal?
sample.taint
sample.frozen?
sample.instance_variable_get
sample.kind_of?

Compared to a simple Python server script (which actually would have its own code execution issues pre 2.4.1/2.3.5 since I didn't use dispatch)

from SimpleXMLRPCServer import *
class HeyJim(object):
 def __init__(self):
     pass
 def bobo(spaz):
     return "x" * int(spaz)
server = SimpleXMLRPCServer(("localhost",8000))
server.register_instance(HeyJim)
server.register_introspection_functions()
server.serve_forever()

Which only produces...

franz-g4:~ mdfranz$ python client.py
bobo
system.listMethods
system.methodHelp
system.methodSignature

I initially used the register_function instead of register_method, but the results were the same. I wanted it to be a fair comparison. Perhaps one reaons is Ruby's method/attribute sloppiness. It exposes everything. IIRC, Python only allows methods to be exposed on objects.

Jython Applets

This is a very cool way to run Jython -- in your browser!

Hell, even readline works - which is more than I can say for my Mac :(

Nobody must use Java Regular Expressions

I'm having a hell of a time converting some Python regex's to Java (first rule, you have to double escape everything) and I'm still getting a bunch of nasty exceptions, the "Illegal repetition" error in particular :(

Yes, I'm weak, but once you use Visual C# it is hard to go back.

package regextest;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class Main {
  public Main() {
  }
  public static void main(String[] args) {
      // Pattern p = Pattern.compile("^\\[char\\speer[\\d+]_[\\d+]*");
      // packet =  re.compile(r"^char\speer(\d)_(\d+)\[.*{(.+)};$")
      Pattern p = Pattern.compile("^\\[char\\speer(\\d)_(\\d+).*$");
      Pattern p1 = Pattern.compile(".*\\{(.*)\\};");
      Matcher m = p.matcher("[char peer1_2[] = {, 0x03, 0x00, 0x00,
0x31, 0x02 0x5f, 0x53, 0x50, 0x50, 0x31, 0x5f, 0x50, 0x41 };");
      Matcher m1 = p1.matcher("[char peer1_2[] = {, 0x03, 0x00,
0x00, 0x31, 0x02 0x5f, 0x53, 0x50, 0x50, 0x31, 0x5f, 0x50, 0x41 };");

      if ( m.matches()) {
          if ( m.groupCount() > 0 ) {
              System.out.println(m.group(1));
              System.out.println(m.group(2));
          }
      } else {
          System.out.println("No match");
      }

      if (m1.matches()) {
          System.out.println(m1.group(1));
      } else {
          System.out.println("You suck!");
      }
  }
}

Ruby in 2006

It had been quite some time since I looked at Ruby. I went through a brief phase back in 2002, but it was still rough around the edges. There was/is however a very nice interface to libpcap that I started writing some protocol characterization tools which never did get completed or released. With all the hype around AJAX and Ruby on Rails I had to give it another look. Also the new version of Metasploit Framework is written in Ruby.

Among the improvements I've noticed:

• Easy installation on windows
  
• Ruby GEMS - a package adminstration tool simuliar to Gentoo Emerge, Debian Apt -- or yum if it actually worked right.