Showing posts with label Python-n-Ruby. Show all posts
Showing posts with label Python-n-Ruby. Show all posts

Wednesday, January 13, 2010

Avoiding Bracket Hell in MongoDB Queries (Python Style)

To me it wasn't immediately obvious from the MongoDB Advanced Query documentation that you can string together multiple operators to perform existence, membership, and greater/than that tests. And since JSON can get very messy (and long!) and the syntax is slightly different from the Javascript in the documentation, instead of passing JSON directly to the find method of your collection pass a dictionary and assign the various conditions

For example:

myq = {}
myq["batchstamp"] = b # a timestamp
myq["modbus_tcp_reference_num"] = {"$exists": True}
cur = coll.find( myq )

Although it doesn't appear much easier than passing

{'modbus_tcp_reference_num': {'$exists': True}, 'batchstamp': 999999999}

Once start adding additional conditions (themselves which may have dictionaries it is much easier and less error prone. Trust me!


Friday, December 25, 2009

Installing Redmine on Debian Etch



Here is a step by step summary of what I did to get to get Redmine up and running on Debian 5.x (Linux etch55 2.6.24-24-openvz #1 SMP Fri Sep 18 19:57:34 UTC 2009 i686 GNU/Linux)

If you don't know what Redmine is, it is like Trac, but better and on Rails. If you don't know what Trac is you probably wouldn't be interested in Redmine, so you can stop reading.

What it took?

1. Review this Rails on Debian guide. I'm sure there are others, but this is was a good starting point to get the nuances of running Rails apps on Debian, which can be a bit of pain if you are relying on packages. I know real rails folks use OSX but I'm not a real rails guy.

2. Install the necessary Debian packages. This is what I had to do on an OpenVZ VE, so you your packages may differ slightly: ruby ruby-dev irb sqlite3 ri libzlib-ruby libsqlite3-ruby libmysql-ruby mysql-serer mysql-client libopenssl-ruby

3. Install Rubygems the normal way. I installed 1.3.5. I created a symlink for gem1.8 just because.

4. Read the Redmine Installation Guide. Most of what you need to know is there, and I'm not going to repeat what it is there because it should just work, especially if you are familiar with rails or the configuration of rails apps.

5. Install rails and rake. I installed rails 2.1.2 based on the minimum for the 0.8x of redmine. I assume more recent versions of rails will work.

6. Download the stable release of Redmine.

7. You should have already configured mysql-server during the install, but make sure you put your password in the database.yml.

8. Create your database.yml, the session key and run the various rake scripts in the installation guide.

9. Fire up with webrick and login with admin/admin.

What went Wrong?
The main issues I had were related to not installing the right Debian packages. For example you definitely need libopenssl-ruby or the startup scripts will fail. I also screwed up the database.yml.

What Next?
Get git working following the instructions here. But first I need to work on my git skills, since I mostly have used subversion.

Wednesday, December 09, 2009

WebSocket Service Fingerprinting with Curl

Fingerprinting is probably a bit of a stretch, but at least I didn't use the "h" word, but using pywebsocket is probably the easiest way to learn about the protocol.

Startup the server....

franz@mfranz-s10-2:~/Documents/pywebsocket-read-only/src/mod_pywebsocket$ python standalone.py -p 8888 -w ../example/

Then the client...

mfranz@mfranz-s10-2:~/Documents/pywebsocket-read-only/src/example$ python echo_client.py -s 127.0.0.1 -p 8888
Send: Hello
Recv: Hello
Send: 日本
Recv: 日本
Send: Goodbye
Recv: Goodbye

Look at the traffic on the wire with ngrep.
interface: lo (127.0.0.0/255.0.0.0)
####
T 127.0.0.1:44284 -> 127.0.0.1:8888 [AP]
GET /echo HTTP/1.1..
##
T 127.0.0.1:44284 -> 127.0.0.1:8888 [AP]
Upgrade: WebSocket..
##
T 127.0.0.1:44284 -> 127.0.0.1:8888 [AP]
Connection: Upgrade..
##
T 127.0.0.1:44284 -> 127.0.0.1:8888 [AP]
Host: 127.0.0.1:8888..
##
T 127.0.0.1:44284 -> 127.0.0.1:8888 [AP]
Origin: http://localhost/..
##
T 127.0.0.1:44284 -> 127.0.0.1:8888 [AP]
..
##
T 127.0.0.1:8888 -> 127.0.0.1:44284 [AP]
HTTP/1.1 101 Web Socket Protocol Handshake..
##
T 127.0.0.1:8888 -> 127.0.0.1:44284 [AP]
Upgrade: WebSocket..
##
T 127.0.0.1:8888 -> 127.0.0.1:44284 [AP]
Connection: Upgrade..
##
T 127.0.0.1:8888 -> 127.0.0.1:44284 [AP]
WebSocket-Origin:
##
T 127.0.0.1:8888 -> 127.0.0.1:44284 [AP]
http://localhost/
##
T 127.0.0.1:8888 -> 127.0.0.1:44284 [AP]
..
##
T 127.0.0.1:8888 -> 127.0.0.1:44284 [AP]
WebSocket-Location:
##
T 127.0.0.1:8888 -> 127.0.0.1:44284 [AP]
ws://127.0.0.1:8888/echo
##
T 127.0.0.1:8888 -> 127.0.0.1:44284 [AP]
..
##
T 127.0.0.1:8888 -> 127.0.0.1:44284 [AP]
..
##
T 127.0.0.1:44284 -> 127.0.0.1:8888 [AP]
.Hello.
#
T 127.0.0.1:8888 -> 127.0.0.1:44284 [AP]
.Hello.
#
T 127.0.0.1:44284 -> 127.0.0.1:8888 [AP]
........
#
T 127.0.0.1:8888 -> 127.0.0.1:44284 [AP]
........
#
T 127.0.0.1:44284 -> 127.0.0.1:8888 [AP]
.Goodbye.
#
T 127.0.0.1:8888 -> 127.0.0.1:44284 [AP]
.Goodbye.
###

Now with curl, notice the headers that you have to add to get a response. With anything less I got a 404. The origin header can be anything.

mfranz@mfranz-s10-2:~$ curl -v http://127.0.0.1:8888/echo -H "Upgrade: WebSocket" -H "Connection: Upgrade" -H "Origin: http://localhost"


* About to connect() to 127.0.0.1 port 8888 (#0)
* Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
> GET /echo HTTP/1.1
> User-Agent: curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.15
> Host: 127.0.0.1:8888
> Accept: */*
> Upgrade: WebSocket
> Connection: Upgrade
> Origin: http://localhost
>
<>
But if the URI doesn't match you get

mfranz@mfranz-s10-2:~$ curl -v http://127.0.0.1:8888/ -H "Upgrade: WebSocket" -H "Connection: Upgrade" -H "Origin: http://localhost"


* About to connect() to 127.0.0.1 port 8888 (#0)
* Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.15
> Host: 127.0.0.1:8888
> Accept: */*
> Upgrade: WebSocket
> Connection: Upgrade
> Origin: http://localhost
>
* Empty reply from server
* Connection #0 to host 127.0.0.1 left intact
curl: (52) Empty reply from server
* Closing connection #0


Thursday, November 26, 2009

Generating SVG Output (from Graphviz) in your Django App

So accomplishing new coding tasks can be a challenge with interruptions and I've had a lot of interruptions this week but I finally got there. And I'm thankful!

I have an app that is storing data, meaning Django models for the uninitiated. What is in there doesn't matter, but it is something that is conducive to plotting with graphviz. So the starting point is a string that is in the .dot format. I have some code that makes queries to the database and I end up with a string.

So there is a utility function that creates this string...

def make_svg_str:
#blah blah blah snip
dot_string += "}"
p = subprocess.Popen('/usr/bin/dot -Tsvg', shell=True,\
stdin=subprocess.PIPE, stdout=subprocess.PIPE)
(stdout,stderr) = p.communicate(dot_string)
return stdout

So I almost got this right the first time except that I forgot the stdout in Popen() which caused the output to go to stdout (and not be assigned to the string) so I saw the .xml in the dev web server logs.

The graphviz string (dot_string) is being piped to the dot executable and then the function is returning the XML SVG as a string, and is obviously assigned to the stdout variable in the tuple.

Now the tricky part within my views.py.

My first mistake was using the Django CSV docs instead of the PDF docs because the latter is what we need. I also didn't remember that HttpResponse is a file-like object so we can can just write to it once we have the SVG text.

def svg(request):
f = Foo.objects.all()
response = HttpResponse(mimetype='image/svg+xml')
response['Content-Disposition'] = 'filename=somefilename.svg'
response.write(make_svg_str(f))
return response

So this will display your image within your browser (which is what I wanted) instead of downloading file if you the use the "attachment" in the Content-Disposition key.

The name of the game is taking shortcuts that get the job done. I'm using the admin interface to provide a good-enough UI to enter the data and now I'm using Graphviz to visualize that data without having to spend a lot of time writing UIs or nasty JavaScript.

Saturday, October 03, 2009

Walking through .nessus files with Python xml.etree.ElementTree

Back when I used to teach Tenable's Nessus course I was always surprised how most folks, if to perform additional analysis or manipulation scan results, used Excel to process NBE/NSR files rather than using XML. So I added some simple examples of how to use Python and Ruby to the course and how easy it is you write a simple parser. In my slides I believe I used expat which requires you to build a list/hash of the data you extract as you encounter the start or end of the element. This works and expat or Sax are the primary parsers I've used over the years, although I did recently discover minidom.

If you click on the capture above (blogger doesn't handle XML or code that well) you'll see that after parsing the .nessus file and starting with the top node (I'm not sure why I had to call getroot())
I navigated through the different nodes within the .nessus file starting with Report, ReportHost, and ending with ReportItem where I extracted the port, and plugin id so that when you run the script you get this for all the

192.168.20.3
- 22/tcp|0
- 1241/tcp|0
- 111/tcp|0
- 1243/tcp|0
- 111/tcp|10223
- 59370/tcp|11111
- 111/tcp|11111
- 33145/udp|11111
- 111/udp|11111
- 1241/tcp|22964
- 22/tcp|22964
- 1241/tcp|10863
- 1241/tcp|35291
- general/tcp|12634
- general/tcp|22869
- 59370/tcp|25221

This code snippet isn't terribly useful but it illustrates the API and how it is very straightforward to parse .nessus files.



Some Benchmarks
Besides being much cleaner (IMHO) the nice thing about ElementTree is that there is a C implementation. This is a 2.0 MB file that consists of 4 scans and the scans include the results from only a handful of targets.

On Python 2.5 / Cygwin on my Ideapad S10-2 (Windows XP SP3)

Pure Python
real 0m4.250s
user 0m3.155s
sys 0m0.357s

C Version
real 0m1.422s
user 0m0.405s
sys 0m0.374s

I wanted to do a comparison with Win32 on Python 2.6 on the same system but I was unable to get timeit.exe working from the Windows 2003 Resource Kit.


NOTE: ElementTree is available in Python 2.5 and later and you should be ashamed if you are using anything older than that.

Tuesday, January 27, 2009

Is jennydddggeee too hot for you? (or, Automated Twitter Spam Blocking?)



If you are reading this blog, you don't know anyone like this, don't want to know anyone that looks like that -- and certainly don't want either of them following your every move.

So it should be pretty easy to write less than 25 lines of Python using Twyt that automatically removes any followers that have a single post.

But there have to be tools that already do this. Or any Twitter clients that will automatically block spam followers.

Tuesday, December 30, 2008

Why use Python to access your Moodle User Database?

Well, besides that PHP is an absolute shit for brains language and basic stuff like yaml, displaying syntax errors in imported modules and other sane things you would expect after using Python or Ruby just ain't there.

And oh yeah, and it is is butt ugly ($, ->, ::, ?> etc.)

Not only that because because I was able to whip this up cool script with SQLAlchemy (no I'm not using the ORM, just want to avoid MysqlDB)

mfranz@mfranz-s10:~/crap$ cat alctest.py
#!/usr/bin/env python
from sqlalchemy import *
from pprint import pprint
e = create_engine("mysql://moodle:blackboard@127.0.0.1/moodle")
m = MetaData(e)
user_table = Table('mdl_user',m,autoload=True,autoload_with=e)
pprint(user_table.columns.keys())
mfranz@mfranz-s10:~/crap$ ./alctest.py
[u'id',
u'auth',
u'confirmed',
u'policyagreed',
u'deleted',
u'mnethostid',
u'username',
u'password',
u'idnumber',
u'firstname',
u'lastname',
u'email',
u'emailstop',
u'icq',
u'skype',
u'yahoo',
u'aim',
u'msn',
u'phone1',
u'phone2',
u'institution',
u'department',
u'address',
u'city',
u'country',
u'lang',
u'theme',
u'timezone',
u'firstaccess',
u'lastaccess',
u'lastlogin',
u'currentlogin',
u'lastip',
u'secret',
u'picture',
u'url',
u'description',
u'mailformat',
u'maildigest',
u'maildisplay',
u'htmleditor',
u'ajax',
u'autosubscribe',
u'trackforums',
u'timemodified',
u'trustbitmask',
u'imagealt',
u'screenreader']

Now that I've got that off my chest.

So what I was trying to do, since Moodle is PHP (and I'm stuck with Moodle) and we are a PHP shop and I thought I would do the right thing and try to use PHP even though I hate it, know it is evil, etc.

The app is in PHP and there are obviously some higher-level APIs/ for accessing Moodle tables, so it makes sense I should write my scripts in PHP?

And there were.So I started using DML (although I was using Pre-2.0 has awful documentation on the wiki, so I basically had to look at the source, which at least has decent internal documentation) to provide external (meaning not through the Moodle web UI) to the Moodle user database.

But that took way too long. Of course it has been years since I've touched any PHP, so I'll admit that was part of the problem. Mainly, forgetting semi-colons. What kind of insane language requires semi-colons as statement separators?

I was contemplating some a weird hack (which I know works just fine, because I've done it before) of sending YAML over SSH (in lieu of XMLRPC, which is a pain in the ass to secure) but php-syck is completely broken with CentOS and I wasn't able to build the PHP module manually, which I shouldn't have to, anyway.

So the long and short of it. I completed in Python (and my Python is rusty) in an hour what took me 3-4 in PHP so Python it is. Honestly, much of the time could have been saved If PHP had an interactive interpreter like Ruby or Python so could quickly test out the new APIs I was learning, inspect objects, etc.

Saturday, December 13, 2008

XDot Graphviz Viewer

The last time I was using Graphviz I was mostly using the Mac, which had a nice viewer, but XDot seems to be the best option for Linux. It is a single Python script and all the dependencies are available in the repos.

Saturday, September 27, 2008

Cobbler & Func




Having devoured all the debate coverage I ran across Cobbler on Freshmeat the morning:


Cobbler is a Linux installation server that allows for rapid setup of network installation environments. With a simple series of commands, network installs can be configured for PXE, reinstallations, media-based net-installs, and virtualized installs (supporting Xen, qemu, KVM, and VMware Server). Cobbler uses a helper program called 'koan' (which interacts with Cobbler) for reinstallation and virtualization support.


And func, almost makes me want to be responsible for a few hundred *NIX boxes, again.

Friday, August 01, 2008

Lua is Not Perl so It Must Be OK (or maybe not!)

I'd heard of Lua before, but this week I kept on hearing about it and I have nothing better to do while waiting outside my daughter's door hoping, praying, she'll finally get to sleep.

Lua 5.0.3 Copyright (C) 1994-2006 Tecgraf, PUC-Rio
> print "blah"
blah
> print "blah"
blah
> d = [1,3,4]
stdin:1: unexpected symbol near `['
> d = (1,3,4)
stdin:1: `)' expected near `,'
> d = { 1,3,4 }
> d
>>
>> print d
stdin:3: `=' expected near `print'
> print d
stdin:1: `=' expected near `d'
> d
>> puts
stdin:2: `=' expected near `puts'
> print (d)
table: 0x8062df8
> type(d)
>
> print type(d)
stdin:1: `=' expected near `type'
> print (type(d))
table


The parentheses are weird, but I actually sort of like it, it feels sort of Pythonic but I'm sure its not so off to the Tutorials!

A few minutes later.

Yuck, but OO is bloody awful and after doing Ruby anything with __ just feels dirty, even Python.

OK, there is not point in bothering with Lua, but for a minute there it looked interested.

Move on, nothing to see here.

Thursday, May 08, 2008

Port Forwarding on Windows (Python to the rescue)

If there are Windows equivalents of redir I'm not aware of them, but the first proxy I tried on A list of open-source HTTP proxies written in python (a cool site I ran into several years ago) was TCPWatch and it worked great (at least on Python 2.5.1 under cygwin.)

(Truth be told, the first tool I ran across was pinhole in an ASPN recipe but it ran into some issues, possbly thread related)

Saturday, February 16, 2008

Ruby vs. Python (.NET Style)



If Ruby has the lead on the JVM it appears the opposite is true for Python, as Iron Python is miles ahead on .NET (or should I say .DLR/.CLR?). Still trying to build Iron Ruby it about 3-4 CPU-hours in. Of course slow ass Thinkpad-IO which is killing VMWare isn't helping, and the bloat of Visual C# Express 2008 which itself took an hour or two to install over the network.

Sunday, January 13, 2008

BinData for Ruby Fuzzers

Not that I'm into fuzzing binary protocols/file formats anymore. But if I still wasted my time on that sort of nonsense BinData looks useful enough for that sort of thang:


= BinData is a declarative way to read and write structured binary data.

This is a performance release. Execution speed has been doubled and memory usage has been decreased by about 25%.

== What is BinData?

Do you ever find yourself writing code like this?

io = File.open(...)
len = io.read(2).unpack("v")
name = io.read(len)
width, height = io.read(8).unpack("VV")
puts "Rectangle #{name} is #{width} x #{height}"

It's ugly, violates DRY and feels like you're writing Perl, not Ruby.

Here's how you'd write the above using BinData.

class Rectangle < BinData::Struct
. endian :little
. uint16 :len
. string :name, :read_length => :len
. uint32 :width
. uint32 :height
end

io = File.open(...)
r = Rectangle.read(io)
puts "Rectangle #{r.name} is #{r.width} x #{r.height}"

BinData supports signed/unsigned integers, strings, null terminated strings, arrays, choices and user defined structures.

Thursday, January 10, 2008

Wow A Ruby Modbus Implementation

Since we know all the cool kids use Ruby (but the smart kids use Python) so cool kids that want to "play SCADA" can now use RModbus to bring down the power grid!

I didn't play around with this because I think the best Open Source Modbus implementation (for SCADA hacking) is Jamod which I previously with Jython but would now probably use JRuby since I'm no longer smart anymore.

Sunday, December 23, 2007

Aspen: A Python Web Server You Can Get Excited About



A year ago (or at least over Christmas and New Years) I was playing a lot with Django (and reading about WSGI) so its fitting I ran across Aspen.


Aspen is designed around the idea that there are basically two kinds of websites, publications and applications, differentiated by their organization and interface models. A publication website organizes information into individual pages within a hierarchical folder structure that one navigates by browsing. In an application website, on the other hand, data is not organized into hierarchical pages but is dealt with via a non-browsing interface such as a search box.

The HTML version of this documentation is an example of a publication website: a number of hypertext documents organized into sections. If we weren't using LaTeX (or if I knew how to use it better), the sections would probably be encoded in folders. Gmail is a pure application website, one which organizes and presents information non-hierarchically. Most websites, however, are hybrids. That is, within an overall hierarchical organization you will find both individual pages of information as well as applications such as a site search feature, or a threaded discussion forum.

Publication websites are actually a subset of application websites, of course. An application site can use any interface metaphor; a publication is an application that uses the familiar folder/page metaphor to organize and present its information. Therefore, every website is fundamentally an application.

Aspen enables the full range of websites: publications, applications, and hybrids. It uses the filesystem for the hierarchical structure of publication and hybrid websites, and provides a mechanism for including applications within that hierarchy.


Based on the screencast, it looks very cool. Why? It is so un-Ruby: well documented, it supports multiple frameworks (a Python HTTP server that support PHP!) and Conan O'Brien-style talking faces. Hopefully I'll be able to squeeze some time away from baby care to play around with it.

Saturday, December 01, 2007

Using Hashes Like it is 1999

This week I picked up [what I thought would be] a quick logfile analysis task. Things started out great. I took the time to look at the logfile format and generalized 4-5 different messages (with appropriate regexes to get the data I needed) generated by the security device. Next I extended a basic "logrunner" class I wrote last month for analyzing the debug output from the Intel FreeBSD drivers (basically you do some sysctl's and it dumps some kernel messages to see counters missed, received packets--much better than netstat).

In my logrunner class, you basically can "attach" various simple regex matches and a symbol and you get a nice hash back with the values you want and it hides all the low-level details of matching or handling time stamps, etc. (HINT: If you are mucking with syslog files in Ruby and you are not using the Time API, you are a fool, but I digress).

After a few hours in I thought things were going fine, before some distractions kept me from working on it again until until the next afternoon (I overconfidently estimated this would take about 4 hours from start to finish), so I was in a rush. The initial desire to develop something be a more general purpose tool and that was designed properly was replaced with the brute force, quick hack, get-r-done approach.

I ended up iterating through the output hashes output by the logrunner tool to create more hashes some with the IP address as a key, others with a username as the key. And all of this pointed to at least another hash (or two) so I ended up with something like:

blah[blah][blah] = { 1 => { a => b, c => d }, 3 => { a=> q, d => z } }

This would have been a trivial task except there was no single session identifier (or even username or IP address) on each line that I could tie the various pieces of data together. Then I kept getting confused (and alternating between |k| and |k,v| with my Ruby blocks) it took my longer than I had hoped but I was done in about 7. I had the output I wanted. Went from a few hundred megs of logs to a nice Excel-friendly CSV file. And I thought I was done.

Until Friday afternoon, when I found it some additional data was needed. Extracting the data wasn't a problem (that was done in 5 minutes), but correlating it and getting the report format was. Should I add another hash? Redefine the hashes I'd written? Five o'clock on Friday (with restless hungry kids) is not a time for clarity of thought, but this morning I realized the Ruby I was written was as unreadable as the Perl I used to write back in the day.

Spending the afternoon driving out in the snow which turned to sleet which turned to rain finally beat some sense into me. I mapped out the data on paper (this time) and did right. Came up with 6 simple classes (2 base and 4 sub) to abstract away the hashes and ended up with less than 1/10th of the lines of code in the main loop and a 1/3 of the iterations. Nothing fancy, no Ruby foo, nothing that couldn't be done in Python. And the code is actually readable. The moral of the story? If you are using hashes 4-6 levels deep you have a problem. Stop, step away from the keyboard and come up with a cleaner design. Do it right the first time, you won't regret it. Because quick hacks have a funny way of running on systems for a long, long time.

Wednesday, November 28, 2007

File IO in Dynamic Languages or Fun Political You Tube?



Forget about Simple File IO in Dynamic Language when you've got a very pregnant wife with a bad YouTube habit, you are bound to find some good shit.

Tonight it is Clifton's Notes (and if you don't get his schtick you obviously don't know who Cornel West is) from David McMillan.

This guy is a genius.

Saturday, October 13, 2007

Can't Sleep? Read about Scaling Web Apps

If, you too, happened to have watched Knocked Up tonight and can't sleep (and no I haven't been reading the "baby books" but I probably should be), you might try reading this fairly vacuous article/discussion called Why most large-scale Web Sites are not written in Java but it lead me to High Scalability which actually pretty interesting and reminded me of the cool Joyeunt Prezo from RailsConf I ran into a while back.

Thursday, August 23, 2007

Non-Jython News and the irrelevance of Jython

So JDJ has an article on a major upgrade in Jython 2.2


Jython 2.2 has support for most of Python 2.2 and numerous features from Python 2.3. The new release - the first major overhaul in 4 years - includes many major changes:

* new-style classes
* Java Collections integration
* PEP 302 implementation
* iterators
* generators
* __future__ division
* support for running on JDK1.5 and 1.6
* new installer
* a significant number of features to bring Jython in line with CPython


Compatible with 2.2? Who cares? Given the incompatibilities between 2.3 and 2.4 (let alone 2.2 and 2.4) this makes Jython basically unusable.

JRuby has clearly won the war here in terms of major scripting languages to use with Java and that is too bad.