Good Bye BlogFranz and 20th Century

Hello franzgedank.

Finally decided to give WordPress and try and leave this klunky place behind.

The chroot Hack to making Debian kFreeBSD a Usable Firewall

A Rant Against Debian kFreeBSD
Don't get me wrong, I love the fact that there is a Debian distribution based on FreeBSD but if you are going to mention PF as a reason for using it, you should at least have a version of tcpdump that can read packet drops and build of pflogd so they can be logged. How useful is a firewall that you can't even tell when there are policy violations?

Not very. So I'd run across this blog and even sort of tried to apply the patches. Yes, that is the right answer (and I've been meaning to build my own Debian kFreeBSD package repository, but I'm lazy) but I today on the kFreeBSD FAQ that showed how you could run a native FreeBSD binaries in a chroot. Incidentally you can do the reverse as well, meaning run Debian *BSD chroots on top of the real FreeBSD.

Here is a quick and easy way to get tcpdump and pflogd working on a Debian kFreeBSD without having to compile anything.

Preparation
I won't repeat all the instructions from the FAQ because they just work but here are the high level steps:

1. Pull down the .iso for FreeBSD 8.2 (obviously pick the right architecture)
2. Mount the iso using mdconfig (this is basically like loopback filesystem)
3. Extract the sources from the install CD
4. Strip out the libraries and files that you need

I decided to create an /opt/native for my chroot jail

/opt/native/etc:

total 92

drwxr-xr-x 2 root root 512 Aug 14 13:26 .

drwxr-xr-x 10 root root 512 Aug 14 13:21 ..

-rw-r--r-- 1 root root 1667 Aug 14 13:25 aliases

-rw-r--r-- 1 root root 429 Aug 14 13:21 group

-rw------- 1 root root 1433 Aug 14 13:25 master.passwd

-rw-r--r-- 1 root root 1329 Aug 14 13:22 passwd

-rw-r--r-- 1 root root 40960 Aug 14 13:26 pwd.db

-rw------- 1 root root 40960 Aug 14 13:26 spwd.db

/opt/native/lib:

total 2848

drwxr-xr-x 2 root root 512 Aug 14 13:12 .

drwxr-xr-x 10 root root 512 Aug 14 13:21 ..

-r--r--r-- 1 root root 1432616 Aug 14 12:08 libcrypto.so.6

-r--r--r-- 1 root root 32104 Aug 14 12:08 libcrypt.so.5

-r--r--r-- 1 root root 1155172 Aug 14 12:09 libc.so.7

-r--r--r-- 1 root root 182668 Aug 14 12:08 libpcap.so.7

-r--r--r-- 1 root root 56832 Aug 14 13:12 libutil.so.8

/opt/native/libexec:

total 244

drwxr-xr-x 2 root root 512 Aug 14 12:15 .

drwxr-xr-x 10 root root 512 Aug 14 13:21 ..

-r-xr-xr-x 1 root root 220628 Aug 14 12:15 ld-elf.so.1

/opt/native/sbin:

total 24

drwxr-xr-x 2 root root 512 Aug 14 12:11 .

drwxr-xr-x 10 root root 512 Aug 14 13:21 ..

-r-xr-xr-x 1 root root 18924 Aug 14 12:11 pflogd

/opt/native/usr:

total 8

drwxr-xr-x 4 root root 512 Aug 14 12:06 .

drwxr-xr-x 10 root root 512 Aug 14 13:21 ..

drwxr-xr-x 2 root root 512 Aug 14 12:06 bin

drwxr-xr-x 2 root root 512 Aug 14 12:10 sbin

/opt/native/usr/bin:

total 4

drwxr-xr-x 2 root root 512 Aug 14 12:06 .

drwxr-xr-x 4 root root 512 Aug 14 12:06 ..

/opt/native/usr/sbin:

total 644

drwxr-xr-x 2 root root 512 Aug 14 12:10 .

drwxr-xr-x 4 root root 512 Aug 14 12:06 ..

-r-xr-xr-x 1 root root 627792 Aug 14 12:10 tcpdump

/opt/native/var:

total 10

drwxr-xr-x 5 root root 512 Aug 14 13:24 .

drwxr-xr-x 10 root root 512 Aug 14 13:21 ..

drwxr-xr-x 2 root root 512 Aug 14 13:23 empty

drwxr-xr-x 2 root root 512 Aug 14 13:26 log

drwxr-xr-x 2 root root 512 Aug 14 13:30 run

/opt/native/var/empty:

total 4

drwxr-xr-x 2 root root 512 Aug 14 13:23 .

drwxr-xr-x 5 root root 512 Aug 14 13:24 ..

/opt/native/var/log:

total 10

drwxr-xr-x 2 root root 512 Aug 14 13:26 .

drwxr-xr-x 5 root root 512 Aug 14 13:24 ..

-rw------- 1 root root 4290 Aug 14 14:12 pflog

/opt/native/var/run:

total 6

drwxr-xr-x 2 root root 512 Aug 14 13:30 .

drwxr-xr-x 5 root root 512 Aug 14 13:24 ..

-rw-r--r-- 1 root root 5 Aug 14 13:30 pflogd.pid

Create your chroot scripts on the host filesystem

They should look like this:

root@debian:/opt# ls -al /usr/local/bin/native*

-rwxr--r-- 1 root staff 74 Aug 14 13:30 /usr/local/bin/nativepflogd

-rwxr--r-- 1 root staff 47 Aug 14 12:12 /usr/local/bin/nativetcpdump

and inside

root@debian:/opt# cat /usr/local/bin/native*

exec chroot /opt/native/ /sbin/pflogd -s 250 -i pflog0 -f /var/log/pflog

exec chroot /opt/native /usr/sbin/tcpdump "$@"

Make sure the chroot jail can get to devices

# mount -t devfs devfs /opt/native/dev

This should show up as

# mount
devfs on /opt/native/dev (devfs, local, multilabel)

Confirm that it actually works

You can start pflogd by running /usr/local/bin/nativpflogd

root@debian:/opt# ps aux | grep pflog

root 9115 0.0 0.1 3240 1264 ? S+ 14:18 0:00 grep pflog

64 8965 0.0 0.1 9680 1508 ? S 13:30 0:00 pflogd: [running] -s 250 -i pflog0 -f /var/log/pflog

root 8964 0.0 0.1 9680 1448 ? Ss 13:30 0:00 pflogd: [priv]

root@debian:/opt#

# nativetcpdump -nr /var/log/pflog

reading from file /var/log/pflog, link-type PFLOG (OpenBSD pflog file)

17:30:47.166732 IP 192.168.56.1.64248 > 192.168.56.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST

17:30:47.437045 IP 192.168.56.1.64248 > 192.168.56.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST

17:30:47.707640 IP 192.168.56.1.64248 > 192.168.56.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST

The 5 Minute Guide to Running OpenVZ on CentOS 6.x

As much as I like Debian (and the fact that it has OpenVZ kernels in the repos, for now...) CentOS is really the best distribution for running this container-based virtualization environment. This assumes a minimal CentOS 6.0 install.

1) Add the OpenVZ repos

See the Quick Installation guide and ensure that:

[root@opti330 yum.repos.d]# cat /etc/yum.repos.d/openvz.repo
[openvz-utils]
name=OpenVZ utilities
#baseurl=http://download.openvz.org/current/
mirrorlist=http://download.openvz.org/mirrors-current
enabled=1
gpgcheck=1
gpgkey=http://download.openvz.org/RPM-GPG-Key-OpenVZ

[openvz-kernel-rhel6]
name=OpenVZ RHEL6-based kernel
#baseurl=http://download.openvz.org/kernel/branches/rhel6-2.6.32/current/
mirrorlist=http://download.openvz.org/kernel/mirrors-rhel6-2.6.32
enabled=1
gpgcheck=1
gpgkey=http://download.openvz.org/RPM-GPG-Key-OpenVZ

2) Install the packages (after updating of course)

# yum install openvz-kernel-rhel6 vzctl vzquota bridge-utils

3) Update /etc/sysctl.conf

net.ipv4.ip_forward = 1
net.ipv4.conf.all.proxy_arp = 1

This ARP proxying is really only needed if you are doing veth networking (the default we'll use below)

4) Reboot

You kernel should now be:

[root@opti330 yum.repos.d]# uname -a
Linux opti330 2.6.32-042stab024.1 #1 SMP Tue Jul 26 15:23:12 MSD 2011 x86_64 x86_64 x86_64 GNU/Linux

5) Update /etc/sysconfig/iptables to allow traffic to/from venet0 (only if you are using venet)

-A FORWARD -i venet0 -o eth0 -j ACCEPT

-A FORWARD -i eth0 -o venet0 -j ACCEPT

Or something close. You will also want to update your iptables policy or disable it or disable it.

6) Now you can create your VE's per the instructions on my OpenVZ Wiki Page.

Creating Debian Packages with FPM for the Impatient

So I've been aware of FPM for a while, but I actually hadn't got around to using it. This is an example on Debian but it should work on Ubuntu just fine.

1) Use the package maintainers version of Ruby and RubyGems.

root@debian64-10:~/tmp# apt-get install ruby ruby-dev rubygems
[snip]
Get:1 http://ftp.us.debian.org/debian/ squeeze/main rubygems1.8 all 1.3.7-3 [202 kB]
Get:2 http://ftp.us.debian.org/debian/ squeeze/main rubygems all 1.3.7-3 [66.7 kB]
Fetched 269 kB in 0s (518 kB/s)
Selecting previously deselected package rubygems1.8.
(Reading database ... 50363 files and directories currently installed.)
Unpacking rubygems1.8 (from .../rubygems1.8_1.3.7-3_all.deb) ...
Selecting previously deselected package rubygems.
Unpacking rubygems (from .../rubygems_1.3.7-3_all.deb) ...
Processing triggers for man-db ...
Setting up rubygems1.8 (1.3.7-3) ...
Setting up rubygems (1.3.7-3) ..

2) Install FPM

root@debian64-10:~/tmp# gem update; gem install fpm
Updating installed gems
Nothing to update
Building native extensions.  This could take a while...
Successfully installed json-1.5.3
Successfully installed fpm-0.3.7

3) This installed it but where does the script go? It isn't in /usr/local/bin. Nah that would be too obvious.

root@debian64-10:/# find / -name "fpm"
/var/lib/gems/1.8/bin/fpm
/var/lib/gems/1.8/doc/fpm-0.3.7/rdoc/files/lib/fpm
/var/lib/gems/1.8/gems/fpm-0.3.7/bin/fpm
/var/lib/gems/1.8/gems/fpm-0.3.7/lib/fpm

4) Update your path to included the gem wrappers and confirm that it runs

root@debian64-10:/# export PATH=$PATH:/var/lib/gems/1.8/bin
root@debian64-10:/# fpm
Missing package target type (no -t flag?)
Missing package source type (no -s flag?)
There were errors; see above.

Usage: fpm [options]
  -p, --package PACKAGEFILE        Th

To make it interesting, let's create a package that requires a dependency. The app I'm doing this for at work requires Java unfortunately, but I'll do Mono here for fun.

(I'm skipping the part here where I google for a lame Hello World C# app because I haven't touched the language since 2005 but the bottom line is I end up with .exe that will go in /usr/local/bin) so we have something like.

root@debian64-10:~/hellomono# pwd
/root/hellomono
root@debian64-10:~/hellomono# ls -al usr/local/bin/hellomono.exe 
-rwxr-xr-x 1 root root 4096 Jul 28 21:39 usr/local/bin/hellomono.exe

All we are doing here is specifying the source (a directory) and a destination target (.deb). I assume we could do the same thing for an RPM or even creepier build a Ruby gem that contained a Mono app.

root@debian64-10:~/hellomono# fpm -s dir -t deb -d mono-runtime .

This created a .deb based on your current directory

tar: ./build-deb-hellomono_1.0_amd64.deb/data.tar: file is the archive; not dumped
Created /root/hellomono/hellomono_1.0_amd64.deb
root@debian64-10:~/hellomono# ls
hellomono_1.0_amd64.deb  usr

And if we open it up

root@debian64-10:/tmp# ar -x hellomono_1.0_amd64.deb 
root@debian64-10:/tmp# ls
control.tar.gz data.tar.gz  debian-binary  hellomono_1.0_amd64.deb  hsperfdata_root
root@debian64-10:/tmp# tar xzvf control.tar.gz 
control
md5sums
root@debian64-10:/tmp# cat control
Package: hellomono
Version: 1.0
Architecture: amd64
Maintainer: 
Depends: mono-runtime
Standards-Version: 3.9.1
Section: default
Priority: extra
Homepage: http://nourlgiven.example.com/no/url/given
Description: no description given
root@debian64-10:/tmp# ls
control  control.tar.gz  data.tar.gz  debian-binary  hellomono_1.0_amd64.deb  hsperfdata_root  md5sums
root@debian64-10:/tmp# tar xzvf data.tar.gz 
./
./build-deb-hellomono_1.0_amd64.deb/
./usr/
./usr/local/
./usr/local/bin/
./usr/local/bin/hellomono.exe

Now I scp it over to an Ubuntu LTS box and install

virtual@ubuntu14:~$ sudo dpkg -i hellomono_1.0_amd64.deb 
[sudo] password for virtual: 
Selecting previously deselected package hellomono.
(Reading database ... 73505 files and directories currently installed.)
Unpacking hellomono (from hellomono_1.0_amd64.deb) ...
dpkg: dependency problems prevent configuration of hellomono:
 hellomono depends on mono-runtime; however:
  Package mono-runtime is not installed.
dpkg: error processing hellomono (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 hellomono

Fix the dependencies

virtual@ubuntu14:~$ sudo apt-get install -f
Reading package lists... Done
Building dependency tree       
[snip]
Setting up hellomono (1.0) ...
Setting up libmono-system2.0-cil (2.4.4~svn151842-1ubuntu4) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place

And it runs!

virtual@ubuntu14:~$ /usr/local/bin/hellomono.exe 
Hello World
virtual@ubuntu14:~$ dpkg -l | grep hello
ii  hellomono                       1.0                                             no description given

BCFG2: A non-religious Open Source Configuration Management Tool?

Yesterday I was reading a pissing match between Chef and Puppet and I'm starting to think that the only thing more religious than scripting languages (Ruby vs. Python) or web frameworks (Rails vs. Django) is configuration management tools. Just as the case with religious sects that spun off from each other due to theological (or personality) battles, dissatisfaction with cfengine begat Puppet which in turn begat Chef.

Enter BCFG2 which I discovered on Wikipedia's page on Open Source configuration management tools and this great page which compares and contrasts the 4 leading tools. Unlike these tools, this was an organic development project that was developed for internal use at Argonne National Labs and it is written in Python. Also it is non-commercial, unlike it's three other competitors. I don't think these factors are accidental.

Anyway, if you've never heard of it check out the BCF2 project page and some of the videos and see if you agree. I'm only a couple of hours in, but it definitely seems to have some interesting features that differentiate it from the alternatives, besides being a little less religious.

Meanwhile I'm testing it out on a few VMs. What I'm most interested in, is how it supports multiple Linux distributions simultaneous, in particular Debian, Ubuntu, and CentOS.

Python Boto + SSH SOCKS = Easy Cheap Coffeeshop VPN

A while back (well actually after reading Unencrypted Wifi Must Die) I started using SSH SOCKS tunnels to a small AMI I created with using the AWS Console. I end up paying less than a dollar a month for this. You'll need to install Python Boto for this, but on Ubuntu 10.04 and later this is as simple as

apt-get install python-boto

The code is pretty simple

First connect to EC2

e = boto.connect_ec2()

Create a new instance where AMI_TYPE is set to 't1.micro' (the smallest cheapest AMI)

r = e.run_instances(AMI,key_name=KEY_NAME,instance_type=AMI_TYPE)
i = r.instances[-1]

You basically have to create the reservation and then pull the instance from the list of reservations. Wait for the instance to come up so you can find out the dns name

print "Host name:",i.public_dns_name

--
ec2unnel.py is available on GitHub. You obviously need to change the name to your SSH key and set your AWS environment variables or hard code them in the script.

mfranz@mfranz-xp60sublts:~$ ./ec2unnel.py stop
Connecting to EC2...
Host name: ec2-75-101-195-59.compute-1.amazonaws.com
State: shutting-down
State: shutting-down
State: shutting-down
State: shutting-down
State: shutting-down
State: shutting-down
State: terminated
mfranz@mfranz-xp60sublts:~$ ./ec2unnel.py start
Connecting to EC2...
Creating instance in: us-east-1
Launch time: 2011-03-15T12:15:05.000Z
State: pending
State: pending
State: pending
State: pending
State: pending
State: running
Host name: ec2-50-17-19-144.compute-1.amazonaws.com

mfranz@mfranz-xp60sublts:~$ ssh -v -i duo.pem -D 1080 ec2-user@ec2-50-17-19-144.compute-1.amazonaws.com
OpenSSH_5.3p1 Debian-3ubuntu5, OpenSSL 0.9.8k 25 Mar 2009
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to ec2-50-17-19-144.compute-1.amazonaws.com [50.17.19.144] port 22.
debug1: Connection established.
debug1: identity file duo.pem type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3p1 Debian-3ubuntu5
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) lang =" en_US.utf8">

And you see it takes a while for the terminated instances to go away.

mfranz@mfranz-xp60sublts:~$ ./ec2unnel.py
Connecting to EC2...
2011-03-15T12:02:23.000Z terminated
2011-03-15T12:15:05.000Z terminated

Obviously you'll need to set your proxy

Large Scale Packet Dump Analysis with MongoDB

So when dealing with hundreds of MBs (or even several GBs) of packet captures spread across dozens of files, wireshark sort of breaks down, even with a fast CPU. In my case I have a laptop (with LUKS encrypted drive) so it is pretty slow. Yeah you can split them into smaller files but then you lose visibility into the complete picture when performing your queries. I think you could also write some .lua wireshark but you still have the bottleneck of tshark. So what to do? Let's back up a bit.

Over the years I've written a variety of Perl, Python, or Ruby scripts for processing .pcaps. Some that use C (or pure Python) .pcap parsers--or when I first started doing this over a decade ago just parsing the output of tcpdump and building hashes or dictionaries. Not only is this slow but you have to persist your hashes via pickling. And the challenge of the pcap libraries is they typically don't have any application layer decoding and they require a C version of the library which isn't very cross platform.

Enter pdml. I first described this in a Digital Bond blog post back in 2006. I can remember doing this on a lowly Powerbook G4 and it just worked. A year ago I was looking for a project to use MongoDB. So I wrote some code to automate the process of creating the .pdml files using wireshark and extracting the fields of interest and inserting them into a MongoDB database. I have a configuration file that specifies which PDML fields I want to extract.

[frame.len]
type = decimal

[ip.ttl]
size = 8
type = decimal

[ip.src]
size = 4
type = ipaddr

[ip.dst]
size = 4
type = ipaddr

[tcp.dstport]
size = 16
type = decimal

Because MongoDB doesn't support periods in key names (I learned this the hard way last year) I change the name of the field from ip.src to ip_src. Anything that wireshark knows about I can extract and it will become a key for that packet.

While I was still importing packets (I had close to 2 million packets in the database) I issue a query to see the unique source IPs. I can do this for any field that is supported in the PDML.

>> db.raw.distinct("ip_src")

Sun Feb 13 11:20:05 [conn17] query art0.$cmd ntoreturn:1 command: { distinct: "raw", key: "ip_src", query: {} } reslen:4457 1607ms

Or let's say I wanted to look at what are the unique TTLs.

>> db.raw.distinct("ip_ttl")

11:31:17 [conn17] query art0.$cmd ntoreturn:1 command: { distinct: "raw", key: "ip_ttl", query: {} } reslen:452 1779ms

Not bad for 2.7 million packets (and counting).

This example is pretty uninteresting because it is just standard TCP/IP headers and if you just wanted session data you could just use netflow but this far more flexible.

The downside is speed of import. Creation of the .pdml file by running tshark is very slow and the parsing of XML in Python is also not the speediest. I'm up to close to 3.1 million packets in about an hour that I've successfully imported into my database, but once they are in it is lightning fast and you are free. Where I'm (hopefully) headed today is some scripts that will create Graphviz representations of all the communications of interest perhaps like those available with Afterglow. Or I can use this to analyze and reconstruct streams from some of the proprietary protocols that I was most interested in. Or I can use this as an exercise to write a Node.js app to browse this data. The point is getting it into a useful database that allows flexible and fast queries and offloads a lot of manual tasks I would normally have to do.

Dark Horse (1999)

(Something I wrote last week thanks to whoever it was in Room 227 at the Holiday Inn Express I was staying at down in Virginia had their TV turned up way too loud, before I got the nerve to call the front desk and tell them to turn it down)

When you are twenty, you have all the time in the world, but nothing to say. When you are forty, you have something to say but no time (or space) to say it. Tonight, I ran across across an set of old CDs dating back to college. They were mixed in with CDRs (backups I will never ever recover from and long unsupported OS install disks) inside one of those thick translucent CDW bags from the their showroom in Vernon Hills, IL. I was surprised they even worked. Or at least some of them did. Despite four moves in five years, I have held on to them. Mostly for the bag and the memories of driving up Milwaukee Avenue at lunchtime when I worked at Hewitt.

I've always thought we left Chicagoland way too soon, even though I know why. For once, there is no imminent move to bring on the clutter-purge of contractor bags, the sound of Legos vacuumed off carpet, that smell of Murphy's Oil Soap on hardwoods. No more "staging" for the buyer's agents or cleaning the rental and holding your breath during the moveout inspection, wondering how much will get subtracted from your security deposit.

The beltway was eerie tonight in the fog. A brief warming in between twenty degree days.. It was strange to see the spires of Mormon Temple illuminated by floodlights, which normally means you have made it, that traffic will get better, regardless of your direction.

But tonight there was no such emotion. Traffic was moving and moving fast. Only the solace of motion with a musical assist. Not unlike my flights West from Austin to San Jose in the early 2000s, staring out the window watching over the West Texas plains, the Rockies, then the Sierras, then that perilous landing over downtown.

With three kids solitude only comes during the commute or travel. I will be in a hotel room tonight. I will write.

It only took forty five minutes to get Tyson's Corner. The Cure's Pornography played “Strange Days” and it has held up over time. Most of the rest have not. Brute force existentialism seems quaint. Now you have no time for the faux nihilism, critical theory, or other youthful excesses. The soundtrack from the Until the End of the World has held up through the years, however, and “Knocking at Death's Door” seemed strangely beautiful tonight. The movie, itself, not so much. However, the vision of us all glued to our small electronic screens reading our dreams was prophetic and perhaps the EMP blast silencing everything electronic.

I'm so glad I only discovered the Internet in my last year at A&M. I am thankful for all those hours in the computer lab, printing draft after draft of my poems and short stories. Some of these I have kept to. A Liberal Arts major out of place. Marking them up at wherever I went to review them in the pre-Starbucks age. Reading T.S. Eliot. Was it just the other day on the beltway en route to Tyson's I had the urge to read Proofrock?

And Wim Wenders. It was 1992, wasn't it? The Spring of poetry readings, after getting my first short story (and that one poem) published in the campus literary magazine. Readings at coffeehouses. Night stocking at the College Station Kroger on University Avenue. Oscar, the tyrannical foreman, kneepads around the shins of his stubby legs. “Faster, white boy. Faster.” Perhaps I've conflated him with a scene from My Own Private Idaho. Curses in Spanish. The semester my wife and I started dating. January, or was it February? March is the most beautiful month in Texas, weather-wise.

Later, I would learn that in between the 2nd and 3rd week of February is when Spring came to Austin. The twin Elms of our 3/1 barely 1100 square foot house in Brentwood, had blossomed. The night before, returning home from Russia, landing at the Bergstrom, a family, home, for the first time. I trudged to the cheap lot leaving Amelia with a screaming, traumatized toddler running away from her, screaming, while I looked for our Red Dodge Dakota and struggled with a car seat for the first time. There would be a lot of screaming, but by the time we turned onto Anderson Lane (or was it North Lamar) it had stopped. Kolya looked around the small house and knew he was home. "Kolya Dom," we repeated and he pointed at the ceiling fans, bookshelves, and the popcorn ceiling.

Only two days earlier, we had left Moscow. Four days before, had left the snowy Volga at night, our facilitator emptying a whole section of the Tupelov with only whisper, so the Americans (and a lone Scottish couple) could leave with their newly acquired children. Did she buy them off? Did she threaten them? In the airport lounge we learned how Russians deal with hysterical, stressed out women. Phenobarbital.

Just like yesterday, it was snowing the morning as we left the orphanage. Perhaps because we were the youngest couple (we had not suffered through years of failed fertility treatments before deciding to adopt,) standing outside in front of the Detsky Dom, we were interviewed by Russian television. Did that ever air? Paranoia causing us to lie and say we still lived in San Antonio because that was what was on our paperwork. It was the final days of Yeltsin. They knew Putin was coming, or so we were told over homemade fruit-flavored vodka.

"The dark horse," the only English-speaking Russian on our floor in the suburban Moscow flat, called him. Our final night in Moscow we stayed a block from Red Square but only ventured out one time to a supermarket for kefir and chocolate. It was too cold and we were too exhausted.