sector 42

A little late to the party perhaps, but the entry of Microsoft into this arena is going to prove interesting.

From the linked talk, it looks like they are taking a lot of ideas from existing frameworks, and not getting much wrong.

Microsoft are definitely making their stack much more appealing.

• IIS7 looks extremely pluggable (lifting pages from Apache), you can insert managed code into myriad extension points.
• VS2008 has LINQ (courtesy of .NET framework 3.5) and associated entity design and modeling tools for clean data access layers.
• Now the MVC framework, which looks like a very clean implementation, supporting both static and dynamic languages (courtesy of the DLR as far as I understand).

I know I’m going to get flak from some of my more opinionated friends for saying this, but look at the developments from a technical point of view, leaving aside the usual frothing about lock-in, and proprietary software, etc etc.

I can’t help but think this has the possibility to significantly decrease the amount of existing .NET developers switching to frameworks like Django or Rails – Microsoft has typically been very good with tooling, and this looks like it will be no exception.

Prove me wrong?

I am somewhat surprised that there is nothing that can be readily identified as “the” media tagging format for video content.

ID3 has spawned an industry of supporting software and pretty much everything capable of playing MP3 supports it as a tagging format. It has proved essential to organizing large libraries of music.
There are mature, stable libraries for reading and writing the format.

Where is the equivalent for video? Xvid is the de-facto video codec, but not a whiff of a similar format that I could find.

XMBC supports additional metadata about movies, but it stores in a seperate cache for its internal use, which is not a portable approach. I want my video to have metadata no matter where it is or what plays it. I also want metadata so a Tag & Rename style equivalent can normalize my media directory structure and file naming.

Update: I found this after finally nailing the right Google keywords. Happily, the developer of the software, is a developer’s developer, and put up links (and source) to how he implemented it. Now all it needs is some application with reach to adopt this instead of inventing their own seperate out-of-file format.

Seems there’s a bit of an opportunity here for someone to just go ahead and implement support for a few pluggable apps and get some traction.

It took a few months, but I have finally determined that the shout of “Kapla!” by the actors in Team America: World Police, just before they engage in battle, is Klingon.

The word is “Qapla”, and means “Success”.

I am not worthy to worship in the Church of Trek, clearly.

I had an interesting assembly usage requirement today.

(For my Java friends, assembly is analogous to JAR file).

We have an existing gateway application (could I be any more vague?) that receives relatively high amounts of message traffic. We’re in the progress up preparing to go live with a new version of the application for updated security requirements.

We also have a number of gateway users who are not quite ready to switch over to the updated version of the service (as usual), despite the long lead times they have been given to prepare.

Here’s the issue – The gateway application uses a particular version of an assembly (let’s call it A1) to connect to a backend system. This system needs to be connected to by customers who have not yet switched over. The new version of the gateway uses assembly version A2 to connect to the updated backend software.

It just so happens that both assembly A1 and A2 use the same base namespace, and by and large, the same class and type names, but with small incompatibilities.

It also happens that the loading of these assemblies will happen inside the same AppDomain. Hmm.

In this case, .NET reference aliases do what we want.

1. Add references to both A1 and A2 in your project. If they both have the same filename, either rename one, or put one in a different directory (I hope its obvious why).
2. On the reference properties pane, change the alias from global to something suitable, e.g. A1Ref and A2Ref.

In your code, declare the alias, and bring in the namespace by qualifying it:

extern alias A1Ref;
extern alias A2Ref;
using A1NS = A1Ref::Common.Namespace.Name;
using A2NS = A2Ref::Common.Namespace.Name;

Now you can use A1NS or A2NS like regular namespaces, depending on which assembly types you want to use.

You have to admit this is pretty cool. In Java I would probably have had to write a custom ClassLoader (correct me if this is no longer correct, it’s been a while).

Prompted by a post from John Gruber, I tackled my unwieldy GMail inbox, and cut approximately 300MB of fat (~10,000 messages), archiving the non-rubbish mail.

It’s scary how much of my online life traverses a Google property. I use Google infrastructure for all my personal domains (Apps for Domains), Google Reader has grown on me for feed reading, I use Google Analytics for every website I deploy, and I use Google Calendar to manage my schedule. I sometimes use Google Talk to chat with my brother & some mates.

And I use YouTube rather often too, of course.

Since I’ve migrated this site from running Mephisto on Rails to a custom project on Django, I thought I’d explain some of the rationale behind the switch.

Framework

My choice of Django was rather arbitrary, I could easily see myself use another full-stack Python framework like TurboGears, so don’t take it as a particular endorsement. The motivating factor was using a Python framework.

I wanted to move away from Rails, as for some time I have become disenchanted with the framework. DHH did a stellar job of marketing the framework through its birth phase, but in my opinion, the very success of Rails is what makes it less than what it could be.

Being so strongly opinionated, you run the risk of dismissing issues raised with your framework out of hand because you can’t see the forest for the trees.

And now that a particular opinion has been committed to it is somewhat harder to take a step back and maybe revisit some initial design decisions.

The primary issue I have with Rails is Active Record. It is, quite simply, a rather thin layer around the database driver with some syntactic sugar for declaring relations. Behind the scenes, its all string concatenation and direct queries to the database.

Since there is no real abstraction of an entity or its identity and lifecycle in an application, or of the concept of a session in which to do some work, the following code:

first = Person.find(1)
second = Person.find(1)

Will result in two different instances of an object in memory, referring to the same row. Changes in one will not be reflected in the other (in your application, where it counts). Since second.save() simply results in UPDATE being executed, changes can be overwritten.

Simple, you say, be careful when you load and don’t load the same object twice in the same context, when this identity is important. Except that when an object is being loaded for you by Active Record (such as, say, something declared by a belongs_to statement), a completely new instance is created and loaded if it was not loaded from the database yet, since there is no tracking of identity (and how can there be, without hacks, if there is no concept of a session?).

Eager loading does nothing to address this issue either, since if you eager load at the wrong time, or in two places, you still end up with the same problem.

This is when the need for something just a little bit more intelligent becomes apparent.

To be sure, its not an easy problem to solve. You have to live and breathe relational algebra to enjoy writing something like a full-fledged ORM that compiles language expressions into SQL. So I have my doubts about whether we’d ever see something like this arise in Rails (or Ruby, for that matter). It’s simply not sexy enough to the Rails mindset.

In Python, we have the excellent SQL Alchemy.

Language

My choice of language was largely based on similar reasons (and I didn’t really find any other Ruby Web framework that piqued my interest).

• Performance: Ruby is slow, and unfortunately, it doesn’t seem that 2.0 (with its associated VM) is going to be released any time soon. Given enough resources, anything can be made to scale (c.f. Twitter), but my resources on this Media Temple slice are quite limited, and I’m not about to shell out for an Engine Yard slice just to host my blog. Uhh, no.
• Metaprogramming Madness: A interesting characteristic of the Ruby community is to do a lot of invention of new Domain Specific Languages. Is too many DSLs an anti-pattern? Jury still out on this one, but I’ve gotten quite tired of tracing something to end up at a method_missing for the umpteenth time. I think I prefer the somewhat more explicit style of Python. “When you have a method_missing hammer, everything looks like a DSL nail?”

Python isn’t exactly the fastest language either, but its better than Ruby in that sense, and I’m not about to drop into anything other than a dynamic language for web programming, the productivity hit is just too big. I also happen to like Python’s spare syntax. I could have chosen an obscure or functional programming language as well, but I like to have good library support, and the Python community isn’t going away any time soon.

Web Server

I heard good things about nginx, so I decided to try it out and see how it goes. So far it’s doing a stellar job of acting as a reverse proxy and SSL frontend to my Django app running under mod_python, and to my Subversion repository under mod_dav_svn. I really like its configuration syntax and the fact that you can do much more natural conditional configuration that depends on some request state. My Apache setup is pretty conventional, if stripped down. It runs only the basic modules necessary at a minimum to support Django and Subversion, nothing else.

Database

I’ve always preferred PostgreSQL over MySQL, and this is no exception. I can’t take serious a database that treated transactions and referential integrity as second class citizens for so long, even if it does now support it properly in its latest incarnation. I’m not particularly fond of MySQL’s cavalier approach to SQL standards either, their implementation decisions are reminiscent of the approach that created the X.509 field mess.

Feel free to disagree :)

This blog is now powered by a custom Django application I cobbled together whenever I had a spare hour or two to work on it.

Some things are still a little rough around the edges, but I’m happy with how it’s shaping up. Since I didn’t get around to building an admin system yet, I loaded the content in a rather primitive manner:

./manage.py loaddata posts.xml 

But on the client side of things, articles, trackbacks and comments are working, everything that should have, has RSS feeds, and the layout is how I want it.

URLs from the previous system (Mephisto) should still be working, courtesy of some rewriting.

I’m hosting it at Media Temple on a nginx instance serving up static content, proxying to Apache and mod_python for dynamic Django content, using PostgreSQL as the database server.

I plan to use it as a bit of a test-bed for deployment and caching ideas I want to try out.

Installation of Linux has certainly come a long way – Getting Ubuntu onto my system was considerably less painful than even Vista’s more streamlined installation process.

The post-install experience is usually where things start to go a bit less smooth though, and this iteration was no exception.

I must state though, that once again, the amount of painful hacking and setup required this time around, was significantly less than the last time I went through this exercise.

I’ll give each of the things I had to do a completely unscientific difficulty rating, with higher being worse, and the person the difficulty applies to being someone rather unfamiliar with Linux.

I must warn you, if you’re a rabid free and only free software proponent, you may take issue with the software that I install onto my machine after a fresh installation. While I have tremendous respect for the people who insist on free software and only free software on their machines, I’m a bit more pragmatic personally, and having 3D acceleration and media support is kind of a big deal to me.

Graphics Driver

The first thing I do after installing a new Linux distribution is get my graphics in order. While the default NV driver that ships with X.Org is decent, I much prefer using NVIDIA‘s accelerated driver. I actually got a notification message that alternate drivers were available upon first login.

The installation process for this driver is fantastic this time around:

1. System -> Administration -> Restricted Drivers Manager
2. Select NVIDIA accelerated graphics driver, enable it, and wait for it to be downloaded and installed. Reboot, and X.Org is now using the NVIDIA driver.

Difficulty Rating 1/10, Brainless.

Font Rendering

Next up, my pet peeve with Linux is its absolutely atrocious default font rendering. I understand that their hands are kind of tied, given that there are some patents out on the algorithms for the much maligned “bytecode interpreter” patches to FreeType by David Turner.

But the reason that I am so picky about font appearance, is that I spend so much time looking at text, and imperfections in rendering really annoy me.

I also have purchased copies of the Sys (proportional) and Pragmata fonts. Without the proper FreeType patches applied, Linux doesn’t do such a great job of rendering these fonts by default, and I am used to seeing them render how I want.

There is a thread on the Ubuntu Forums that describes how to get the improved rendering packages installed, and it is simply a matter of adding the appropriate entries to sources.list, and installing the packages with the patches applied.

Difficulty Rating: 5/10 (Relatively easy, but you have to know what you’re looking for, and may involve command-line editing).

Wireless

Well, if this was a laptop, that is what I would be looking at. As it happens, the motherboard has built in onboard wireless via some RTL chipset. Unfortunately the NetworkManager applet reported that my hardware was not capable of WPA2 (which I use on my home WLAN, so I was not able to test the ease of use of switching between wired and wireless configurations). I imagine I might have been swearing a bit had I needed wireless to get network connectivity going.

Difficulty Rating: Possibly 10/10 had I tried to get it going.

Multimedia Codecs

Ah, yes. The Ubuntu documentation that showed up by default when opening Firefox actually contained some documentation that explained the reasoning behind excluding support for MP3 and other codecs, and offered some advice as to how to get playback going.

I have had less than good experiences with GStreamer for media playback. I don’t think it has ever not crashed, or had artifacting, or some other defect when playing back even formats like Xvid. If I was helping someone else to set this up, I’d go the same route that I did, which is to install the totem-xine package, and the libxine1-ffmpeg package (the latter which is needed for Xvid playback in the xine version of Totem, and this fact not documented that well either, annoyingly).

I also installed a number of DVD playback related packages from the Medibuntu project.

I spent more time than I expected on getting everything together and being able to play the media on my NAS and iPod (sans protected AAC support). A fair amount of forum trawling, and I would expect the average user to give up after a while, due to the amount of conflicting information out there.

Difficulty Rating: 7/10, this is still a weak spot.

The rest of my configuration process revolved around installing all the development toolchains I use (build-essential stuff, Rails, Django, PostgreSQL, etc), none of which was much more than an apt-get away.

Summary

Some good improvements in the overall setup experience, but hardware support (if drivers don’t happen to be bundled with the distribution) could be a bit of a tricky issue still, and I don’t see that being solved while the Linux kernel maintainers happily replace subsystems and break ABI in minor patches, as they seem to delight in doing. Less of an issue for servers, but more of an issue for desktops.

Multimedia experience is also centered around codec support issues, which will remain thorny while patent and DRM issues swirl around formats, something which proprietary operating systems in theory don’t struggle with (but Mac OSX isn’t exactly the model of being able to use all proprietary formats, to be honest – VLC is the only thing saving playback on that platform).

Overall, though, things are improving, and I have not noticed any regressions yet

I recently had reason to re-install Ubuntu Linux on my desktop PC, and thought I would write a little about my experiences, and how things have changed since the last time I had need to do this.

My Hardware:

• ASUS P5W DH Deluxe Motherboard (ICH7-R)
• 2x Corsair XMS PC5400 sticks
• Hitachi 160GB HDT722516DLA380 SATA HDD
• Western Digital WD3200KS 320GB SATA HDD
• ASUS NVIDIA 7900GT, 256MB PCI-e graphics card
• Few various other unimportant bits

As usual, the ISO image finished downloading in a few minutes, got burnt, and I booted into the LiveCD environment, and started the install process (It detected my iPod in the LiveCD environment, but kind of useless since the necessary plugins for MP3 support aren’t installed at that stage).

I had a pre-existing Windows Vista NTFS partition on the 160GB OS drive, I resized it to 70GB without issue (I’ll never get used to that, I remember the days when anything involving NTFS writes invariably meant data loss when used from Linux).

Likewise on the 320GB data drive, I split things 50/50 between Windows & Linux, the resize again transpiring without issue.

After that, it was simply a matter of playing Gnometris and surfing until the installation was complete (Again something other operating systems could learn from).

Reboot, the familiar GRUB menu, select Ubuntu, and wait for GDM to come up (I am not sure that I like “silent” bootup style, I’d prefer that the graphical progress screen shows the text messages of the init scripts when they run).

The easy part done, then.

(This was intended to be a follow up to Network Attached Storage for the Home)

After doing a fair amount of researching it seems that I was a bit optimistic to expect a relatively simple procedure.

Microsoft’s strategy looks to be that despite having a 360, a PC is still the digital hub of the home, since the 360 uses the WMC protocol (yawn) to talk to a Windows PC with the necessary stub software installed, this PC containing all the content. One would think that the 360 would be able to connect to the NAS directly, using SMB, but that is not an option unfortunately. And even if it was, the 360 supports very little in the way of codecs out of the box, necessitating some kind of transcoding proxy sitting between it and any Xvid encoded content.

There is an application called Connect 360 which lets you do some measure of sharing to the XBox, but its fatal flaw is that it requires an OSX system to run, and assumes all your media is in iPhoto/iTunes.

The whole point of my buying a NAS was so that I wouldn’t have to have a PC running to access my content in the first place, and could just stream my media to the XBox directly off it.

A German company called TwonkyVision has implemented a product with the same name, which sounds like it would do the necessary work. At first glance, it looked like the ideal solution, since they created a version that could be installed directly on my NAS. After installing it, I had some reason to hope, because the 360 was able to detect it, and I was also able to browse the directory structure on the NAS from the 360.

However, I ran into some issues:

• Only pictures and audio actually showed up as content on the 360, none of my Xvid encoded movies did (some MP4 content did, but unplayable, see below).
• I am not sure that transcoding actually works in TwonkyVision – I tried to play the MP4 video content that showed up, and I either got a strange codec-like error message, or I nothing happened when I pressed play.

Not very promising – I was hoping the 360 would be a nice option for playing some HD content, but it looks like I am stuck with the XBox first generation and XBMC for my home content needs, though the hardware is starting to feel a bit long in the tooth. If XBMC could run on the 360, and run well (without breaking Xbox Live), I wouldn’t even be posting this, XBMC‘s media navigation and metadata features blow the 360 out of the water, you simply can’t beat community for adding features people actually want, it seems.