Developing a cloud-based IoT service

In my previous post I describe my adventures in building an AWS IoT-enabled application for a proprietary embedded linux system and getting it to run. The next step in our journey is to create a service that communicates with our device and controls it in a useful way.

What can we do with a system running with the aws_iot library? We can use the MQTT message bus to subscribe to channels and publish messages, and we can diff the current device state against the desired device state shadow stored on the server. Now we need the service side of the puzzle.

My sample IoT application is to be able to view images on an IP camera from anywhere on the internet. I’m planning to incorporate live HD video streaming as well but that is a whole other can of worms we don’t need to open for this demonstration. My more modest goal for now will be to create a service where I can request a snapshot from the camera be uploaded to AWS’s Simple Storage Service (S3) which can store files and serve them up to authenticated users. In addition I will attempt to build the application server logic around AWS Lambda, a service for running code in response to events without actually having to deploy a server or run a daemon of any sort. If I can manage this then I will have a truly cloud-based service; one that does not consume any more resources than are required to perform its job and with no need to pre-allocate any servers or storage. It will be running entirely on Amazon’s infrastructure with only small bits of configuration, policy and code inserted in the right places to perform the relatively simple tasks required of my app. This is the Unemployed DevOps lifestyle, the dream of perfect lazy scalability and massive offloading of effort and operations to Amazon. There is of course a large downside to this setup, namely that I am at the mercy of Amazon. If they are missing a feature I need then I’m pretty much screwed and if their documentation is poor then I will suffer enormously. A partial description of my suffering and screwed state continues below.

I’ve been bitten before by my foolish impetuousness in attempting to use new AWS services that have clearly not been fully fleshed out. I was an early adopter of the CodeDeploy system, a super useful and nifty system for deploying changes to your application on EC2 instances from S3 or even straight from GitHub. Unfortunately it turned out to not really be finished or tested or documented and I ended up wasting a ton of time trying to make it work and deal with corner cases. It’s a dope service but it’s really painfully clear nobody at AWS has ever bothered to actually try using it for a real application, and all of my feature requests and bug reports and in-person sessions with AWS architects have all resulted in exactly zero improvements despite my hours of free QA I performed for them. As a result I am now more cautious when using new AWS services, such as IoT and Lambda.

In truth attempting to make use of the IoT services and client library has been one of the most frustrating and difficult uphill battles I’ve ever waged against a computer. The documentation is woefully incomplete, I’ve wasted tons of time guessing at what various parameters should be, most features don’t really behave as one would expect and the entire system is just super buggy and non-deterministic. Sometimes when I connect it just fails. Or when subscribing to MQTT topics.

Usually this doesn't happen. But sometimes it does!
Usually this doesn’t happen. But sometimes it does!

Why does it disconnect me every few seconds? I don’t know. I enabled autoReconnect (which is a function pointer on a struct unlike every other function) so it does reconnect at least, except when it just fails for no apparent reason.

setAutoReconnectStatus is only mentioned as being a typedef in the MQTT client documentation. One would assume you should call the function aws_iot_mqtt_autoreconnect_set_status(), but the sample code does indeed call the struct’s function pointer instead. No other part of the library uses this fakeo method call style.

On the boto3 (python AWS clienet library) side things are not really any better. The device shadow support (called IoT Dataplane) documentation is beyond unhelpful at least as of this writing. If you want to update a device state dictionary (its “shadow”) in python, say, in a lambda, you call the following method:

Usually when you want to specify a dictionary-type object as a param in python it’s customary to pass it around as a dict. It’s pretty unusual for an API that is expecting a dictionary data structure to expect you to already have encoded it as JSON, but whatever. What is really missing in this documentation is the precise structure of the update payload JSON string you’re supposed to pass in. You’re supposed to pass in the desired new state in the format {“state”: { “desired”: { … } } }:

My dumb lambda

If you hunt around from the documentation pages referenced by the update_thing_shadow() documentation you may uncover the correct incantation, though not on the page it links to. It would really save a lot of time if they just mentioned the desired format.

I really definitely have no reason why it wants a seekable object for the payload since it’s not like you can really send files around. I actually first attempted to send an image over the IoT message bus with no luck, until I realized that the biggest message that can ever be sent over it is 128k. This application would be infinitely simpler if I could transmit the image snapshot over my existing message bus but that would be too easy. I am fairly certain my embedded linux system can handle buffering many megabytes of data and my network is pretty solid, it’s really a shame that AWS is so resource-constrained!

The reason I am attempting to use the device shadow to communicate is that my current scheme for getting an image from the device into AWS in lieu of the message bus is:

  • The camera sends a MQTT message that indicates it is online
  • When the message is received, a DevicePolicy matches the MQTT topic and invokes a lambda
  • The lambda generates a presigned S3 request that will allow the client to upload a file to an S3 bucket
  • The lambda updates the device shadow with the request params
  • A device shadow delta callback on the camera is triggered (maybe once, maybe twice, maybe not at all, by my testing)
  • Callback receives the S3 request parameters and uploads the file via libcurl to S3
  • Can now display thumbnail to a web client from S3

I went to the AWS Loft to talk to an Amazon architect, a nice free service the company provides. He didn’t seem to know much about IoT, but he spoke with some other engineers there about my issues. He said there didn’t appear to be any way to tell what client sent a message, which kind of defeats the entire point of the extra security features, and he was going to file an internal ticket about that. As far as uploading a file greater than 128k, the above scheme was the best we could come up with.

Regarding the security, I still am completely at a loss as to how one is supposed to manage more than one device client at a time. You’re supposed to create a “device” or a “Thing”, which has a policy and unique certificate and keypair attached to it and its own device shadow state. I assume the keypair and device shadows are supposed to be associated with a single physical device, which means you will need to automate some sort of system that provisions all of this along with a unique ThingName and ClientID for each physical device and then include that in your configuration header and recompile your application. For each device, I guess? There is no mention of what exactly how provisioning is supposed to work when you have more than one device, and I kinda get the feeling nobody’s thought that far ahead. Further evidence in support of this theory is that SNS messages or lambdas that are invoked from device messages do not include any sort of authenticated ClientID or ThingName, so there’s no way to know where you are supposed to deliver your response. Right now I just have it hard-coded to my single Thing for testing. I give Amazon 10/10 for the strict certificate and keypair verification, but that’s only one part of a scheme that as far as I can tell has no mechanism for verifying the client’s identity when invoking server-side messages and code.

It wasn’t my intention to bag on AWS IoT, but after months of struggling to get essentially nowhere I am rather frustrated. I sincerely hope that it improves in usableness and stability because it does have a great deal of powerful functionality and I’d very much like to base my application on it. I’d be willing to help test and report issues as I have in the past, except that I can’t talk to support without going in to the loft in person or paying for a support plan, and the fact that all of my previous efforts at testing and bug reporting have added up to zero fixes or improvements doesn’t really motivate me either.

If I can get this device shadow delta callback to actually work like it’s supposed to I’ll post more updates as I progress. It may be slow going though. The code, such as it is, is here.


Diving into IoT development using AWS

I’m more allergic than most people to buzzwords. I cringe big time when companies suddenly start rebranding their products with the word “cloud” or tack on a “2.0”. That said, I realize that the cloud is not just computers in a datacenter and the Internet of Things isn’t all meaningless hype either. There exists a lot of cool new technology, miniaturization, super cheap hardware of all shapes and sizes and power requirements, ever more rapid prototyping and lot more that adds up to what looks like a new era in embedded system hardware.

People at the embedded linux conference can't wait to tell you about IoT stuff
People at the embedded linux conference can’t wait to tell you about IoT stuff

But what will drive this hardware? There is a lot of concern about the software that’s going to be running on these internet-connected gadgets because we all just know that the security on most of these things is going to be downright laughable, but now since they’re a part of your car, your baby monitor, your oven, your insulin pump and basically everything, this is gonna be a big problem.

So I’ve embarked on a project to try to build an IoT application properly and securely. I think it’ll be fun, a good learning experience, and even a useful product that I may be able to sell one day. At any rate it’s an interesting technical challenge.

My project is thus: to build a cloud-based IoT (ughhh sorry) IP camera for enterprise surveillance. It will be based on as much open source software as possible, ABRMS-licensed, mobile-first and capable of live streaming without any video transcoding.

I think I know how to do this, I’ve written a great deal of real-time streaming software in the past. I want to offload as much as the hard stuff as possible; let the hardware do all the h.264 encoding and let AWS manage all of the security, message queueing and device state tracking.

At the Dublin gstreamer conference I got to chat up an engineer from Axis, an awesome Swedish company that makes the finest IP cameras money can buy. He informed me that they have a new program called ACAP (Axis Camera Application Platform) which essentially lets you write what are essentially “apps” that are software packages that can be uploaded to their cameras. And they’re all running Linux! Sweet!

And recently I also learned of a new IoT service from Amazon AWS. I was dreading the humongo task of writing a whole new database-backed web application and APIs for tracking devices, API keys, device states, authentication, message queueing and all of that nonsense. Well it looks like the fine folks at Amazon already did all the hard work for me!

So I had my first development goal: create a simple AWS-IoT client and get it to run on an Axis camera.

Step one: get access to ACAP

Axis doesn’t really make it very easy to join their development program. None of their API documentation is public. I’m always very wary of companies that feel like they need to keep their interfaces a secret. What are you hiding? What are you afraid of? Seems like a really weird thing to be a control freak about. And it majorly discourages developers from playing around with your platform or knowing about what it can do.

But that is a small trifle compared to joining the program. I filled out a form requesting access to become a developer and was eventually rewarded with a salesbro emailing me that he was busy with meetings for the next week but could hop on a quick call with me to tell me about their program. I informed them that I already wanted to join the program and typed all the relevant words regarding my interest into their form and didn’t need to circle back with someone on a conference call in a few weeks’ time, but they were really insistent that they communicate words via telephone.

After Joe got to give me his spiel on the phone I got approved to join the Axis developer partner program. As far as ACAP they give you a SDK which you can also download as an Ubuntu VirtualBox image. Inside the SDK is a tutorial PDF, several cross-compiler toolchains, some shady Makefile includes, scripts for packaging your app up and some handy precompiled libraries for the various architectures.

Basically the deal is that they give you cross-compilers and an API for accessing bits of the camera’s functionality, things like image capture, event creation, super fancy storage API, built-in HTTP server CGI support, and even video capture (though support told me vidcap super jankity and I shouldn’t use it). The cross-compilers support Ambarella ARM, ARTPEC (a chip of Axis’s design) and some MIPS thing, these being the architectures used in various Axis products. They come with a few libraries all ready to link, including glib, RAPP (RAster Processing Primitives library) and fixmath. Lastly there’s a script that packages your app up, building a fat package for as many architectures as you want, making distribution super simple. Now all I had to do was figure out how to compile and make use of the IoT libraries with this build system.

Building mbedTLS and aws_iot

AWS has three SDKs for their IoT clients: Arduino Yún, node.js and embedded C linux platforms. The Arduino client does sound cool but that’s probably underpowered for doing realtime HD video, and I’m not really the biggest node.js fan. Linux embedded C development is where it is at, on the realz. This is the sort of thing I want to be doing with my life.

Hells yeah!

All that I needed to do was create a Makefile that builds the aws_iot client library and TLS support with the Axis toolchain bits. Piece of cake right? No, not really.

The IoT AWS service takes security very seriously, which is super awesome and they deserve props for forcing users to do things correctly: use TLS 1.2, include a server certificate and root CA cert with each device and give each device a private key. Wonderful! Maybe there is hope and the IoT future will not be a total ruinfest. The downside to this strict security of course is that it is an ultra pain in the ass to set up.

You are offered your choice of poison: OpenSSL or mbedTLS. I’d never heard of mbedTLS before but it looked like a nice little library that will get the job done that isn’t a giant bloated pain in the ass to build. OpenSSL has a lot of build issues I won’t go into here.

To set up your app you create a key and cert for a device and then load them up in your code:

 connectParams.pRootCALocation = rootCA;
 connectParams.pDeviceCertLocation = clientCRT;
 connectParams.pDevicePrivateKeyLocation = clientKey;

Simple enough. Only problem was that I was utterly confused by what these files were supposed to be. When you set up a certificate in the IoT web UI it gives you a public key, a private key and a certificate PEM. After a lot of dumbness and AWS support chatting we finally determined that rootCA referred to a secret CA file buried deep within the documentation and the public key was just a bonus file that you didn’t need to use. In case anyone else gets confused as fuck by this like I was you can grab the root CA file from here.

The AWS IoT C SDK (amazon web services internet of things C software development kit) comes with a few sample programs by way of documentation. They demonstrate connecting to the message queue and viewing and updating device shadows.

#define AWS_IOT_MQTT_HOST              "" ///< Customer specific MQTT HOST. The same will be used for Thing Shadow                                                                                                                       
#define AWS_IOT_MQTT_PORT              8883 ///< default port for MQTT/S                                                                  
#define AWS_IOT_MQTT_CLIENT_ID         "MischaTest" ///< MQTT client ID should be unique for every device                                 
#define AWS_IOT_MY_THING_NAME          "MischaTest" ///< Thing Name of the Shadow this device is associated with                          
#define AWS_IOT_ROOT_CA_FILENAME       "root-ca.pem" ///< Root CA file name                                                               
#define AWS_IOT_CERTIFICATE_FILENAME   "1cd9c753bf-certificate.pem.crt" ///< device signed certificate file name                          
#define AWS_IOT_PRIVATE_KEY_FILENAME   "1cd9c753bf-private.pem.key" ///< Device private key filename                                      

To get it running you edit the config header file, copy your certificates and run make. Then you can run the program and see it connect and do stuff like send messages.


Once you’ve got a connection set up from your application to the IoT API you’re good to go. Kind of. Now that I had a simple C application building with the Axis ACAP SDK and a sample AWS IoT application building on linux, the next step was to combine them into the ultimo baller cloud-based camera software. This was not so easy.

Most of my efforts towards this were spent tweaking the Makefile to pull in the mbedTLS code, aws_iot code and my application code in a setup that would allow cross-compiling and some semblance of incremental building. I had to up my Make game considerably but in the end I was victorious. You can see the full Makefile in all its glory here.

The gist of it is that it performs the following steps:

  • loads ACAP make definitions (include $(AXIS_TOP_DIR)/tools/build/rules/common.mak)
  • sets logging level (LOG_FLAGS)
  • grab all the source and include files/dirs for mbedTLS and aws_iot
  • define a static library target for all of the aws_iot/mbedTLS code – Screen Shot 2016-03-20 at 2.19.16 PM
  • produce executable:
    Screen Shot 2016-03-20 at 8.39.58 PM.png

The advantage of creating aws-iot.a is that I can quickly build changes to my application source without having to re-link dozens of files.

I combined the Axis logging macros and the aws_iot style logging into one syslog-based system so that I can see the full output when the app is running on the device.

Uploading to the Camera

Once I finally had an ACAP application building I was finally able to try deploying it to a real camera (via make target of course):

Screen Shot 2016-03-20 at 2.18.58 PM

Screen Shot 2016-03-20 at 2.14.21 PM

Getting the app running on the camera and outputting useful logging took quite a bit of effort. I really ran into a brick wall with certificate verification however. My first problem was getting the certs into the package, which was just a simple config change. But then it began failing. Eventually I realized it was because the clock on the camera was not set correctly. Realizing the importance of a proper config, including NTP, I wrote a script to configure a new camera via the REST API. I wanted it to be as simple as possible to run so I wrote it without requiring any third party libraries. It also shares the package uploader config for the camera IP and password so if you’ve already entered it you don’t need to again.

With NTP configured at least there are no more certificate expired errors. I’m able to connect just fine on normal x86 linux, but fails to verify the certs when running on the camera. After asking support, they suggest recompiling mbedTLS with -O0 (disable optimizations) when building on ARM. After doing so, it connects and works!

Screen Shot 2016-03-20 at 2.14.51 PM

🌭🍕🍔 !!!!! Success!

To summarize; at this point we now have an embedded ARM camera device that is able to connect and communicate with the AWS IoT API securely. We can send and receive messages and device shadow states.

So what’s next? Now we need a service for the camera to talk to.


A Night At The TechCrunch ‘Crunchies’

I’d received a free ticket to attend the 9th annual TechCrunch Crunchies award ceremony from one of my journalist friends, assuming they had something more important to report on, like some grass growing.

This celebratory gala event is put together by the fine folks at TechCrunch to honor the brave pioneers and visionaries in Silicon Valley and to give recognition to the humble and under-appreciated venture capitalists that have brought so much hope and prosperity to our peers.


The event, while not strictly formal, was treated by most as an occasion to put on their finery as though attending a real gala or ceremony thus greatly livening the atmosphere, at least as much as can be done at the San Francisco War Memorial Opera House. Slightly diminishing it perhaps were the advertisements for Toyota Priuses everywhere, accompanied by the cars themselves.

Photo by TechCrunch

This Year’s Fashions

All of the guests that went to the trouble to get their grown man or woman on were the very picture of distinguished radiance, play-acting in their nouveau riche dress-up party. The event was meticulously styled up to resemble the Academy Awards but with more sensible hybrid driving options and white people.

In the invitation letter I was tastefully informed,

Attire is up to you. At past Crunchies, attendees have dressed down and dressed up but the majority comes dressed to impress.

All eyes were on the outfits sported by those desiring to impress, and boy they outdid themselves this year!

Mrs. A. C. Came replete in a Vera Wang tout ensemble worn with tasteful aplomb. The hangers-on ordered her many drinks though possibly for the purposes of pouring onto her evening skirt topped with a Gap white button-down.

Screen Shot 2016-02-18 at 4.08.52 PMMr. B.-B. G. was strolling the promenade in his Men’s Wearhouse T-Bone Chuck Blazer with Carolina Herrara’s signature white dress shirt, attracting gazes from admirers wanting to take him for a spin in their new sedans.


Mr. M. A. as usual went in his signature Karl Lagerfeld chic-waiter look with belted waists and sans chapeau, letting his unusually pointy bald head complete the space alien wedding singer look. He was measurably engaged, posting a mere three dozen tweets during the entire span of the ceremony.

The Snapchat Ghost was in an impeccably tailored Tory Burch white inflatable ghost costume which drew hundreds of selfie-takers into his orbit of ghostiness and cash flow positive magnetism.


The Awards

The MC for the evening was some sort of actress who was actually on television sometimes, though I honestly have no idea. To her credit she was pretty mean to the audience and made fun of the crowd a lot, although nearly everyone I spoke to mentioned that last year the host was someone from HBO’s Silicon Valley and was really really drunk and went way over the line in ripping the attendees a new asshole for being terrible human beings. Needless to say they were glad he wasn’t invited back.

Really sad I missed it

But enough about the hosts, let’s talk about some of this year’s nominees and winners!

The TechCrunch Crunchies are all about Tech. As in technology. As in boilerplate database-driven software written in high-level boutique scripting languages employing proprietary walled-garden frameworks. Innovation and disruption are the name of the game, breaking down old staid ideas and replacing them with dynamic new market-driven efficiencies. So it should be no surprise that the Best Mobile App for 2016 went to Facebook’s Messenger app, a revolutionary new take on AOL Instant Messenger, like the ICQ of Western Europe, or kind of like WhatsApp but with ugly avatars (wait shouldn’t they be the same thing by now? Didn’t Facebook buy them?)

Props to Mark Zuckerberg of Facebook, winner of CEO Of The Year (his second award). Dude has hella cash so you pretty much gotta give it to him I guess.

Biggest Social Impact was won not by the people fighting child sexual exploitation as expected but, which teaches people to program.

Best overall startup was a tough race between Slack, Docker, Snapchat, Uber and Xiaomi. I had hopes for Docker but suspected I may have been the only person in the audience who’s actually used it. The award went to the service that everyone was seen using shortly after the event ended – Uber! I feel really happy for their CEO Travis Kalanick, we were all pulling for him! Maybe he can pull more chicks with his Uber hat at The Battery now.


Mega congratulations are in order for Bill Gurley, winner of VC of the Year. I don’t know who this man is but he sure is tall and could buy and sell my whole family on a whim! I enthusiastically clapped for his acceptance speech along with the rest of the crowd. He then went home to bathe in a tub of rare Wu-Tang albums to wash the hoi polloi germs off.

Scott and Cyan Banister, actual decent human beings, won Best Angel Investor(s). They have a sweet First Amendment Clinic in support of free speech legal amicus briefs which I’m a fan of.

Also Cyan was one of the few people who didn’t look like they were wearing a dress ironically


The Press

I sat in the press section by Owen Thomas, former Valleywag gossip columnist and noted writer of words. He recently quit his job at ReadWriteWeb and was attending the ceremony on behalf of his personal enterprise Ditherati. When questioned as to the nature of his new venture he replied that it was still being figured out.

A reporter seated next to me and I had some lively chats guessing who would win the awards. I did pretty good at predicting the winners and should have put down some money. Mostly I was a royal jerk, hollering and clapping loudly for Facebook and Apple, deriding the stupid nominations and generally trying to get thrown out. I am ashamed to report that I failed in my quest. Next year I’m bringing a bottle of something vicious to stoke the fires of righteous disgust. Or maybe the bubble will finally burst and impoverish the usual guests, as my Uber driver told me he prays for daily.

Congrats to all the winners!

Photo: the bold italic
Photo: the bold italic

Thanks to Toyota for their sponsorship – Discover the Redesigned Prius Now!

Poland and Ukraine in 2015

National Museum of the History of Ukraine in World War II
National Museum of the History of Ukraine in World War II

I just went on a vacation to Eastern Europe, one of my most favorite places on earth. Specifically Poland and Ukraine.

Wrocław, Poland is a fantastic city that almost nobody in the USA has ever heard of or even knows how to pronounce (“vraswav”). It is the fourth largest city in Poland and has a number of excellent universities. I’ve had the pleasure of working with a number of software engineers there and they are exceptionally sharp and reliable, and very friendly as well. Since the government gives everyone free schooling through five-year university the population in general is very educated and speaks better english than most outsourcing destinations.

Wrocław City Square
Wrocław City Square

As a result of recent history Wrocław is a pleasing mixture of east and west Europe, having been part of both the third reich and the USSR. Many of the shops and streets resemble Paris or Germany but there are also plenty of communist-style Soviet housing complexes and a definite slavonic ring to the language. As far as the economy goes, Poland is definitely one of the brightest success stories in Europe, being the only eurozone economy that did not fall into recession after 2008’s crash, having a highly educated population of english speakers and engineers, and a close trade relationship with Germany. And since they are not using the Euro it’s cheap for Americans to hire poles (and vacation there).

Beers on the water
Beers on the water
Vodka drinking at a BBQ with our good friends
Vodka drinking at a BBQ with our good friends
Found a WWII memorabilia shop
A WWII memorabilia shop

Warsaw is pretty chill too. Its buildings are much more modern as the entire city was flattened by Nazi bombs.

Royal gardens in Warsaw
Royal gardens in Warsaw

It’s a bit more touristy.

Warsaw city square
Warsaw city square
Building of culture - donated by Stalin
Building of culture – donated by Stalin

After a short tour though Warsaw it was time to head to Kiev, the capital of Ukraine. When we got there it was hot, muggy, draped in total fog and our taxi driver had a giant crack in the windshield of his Lada.

We weren’t really sure what to expect in Ukraine. According to the news in America and the advice of random uninformed friends it sounded like we might be headed into a war zone or collapsing post-Soviet radioactive wasteland. All utter nonsense.

First we checked in to our apartment.

Soviet elevators are not like western elevators. They are more like devices used to frighten claustrophobic people. Or sane and rational people.
Soviet elevators are not like western elevators. They are more like small rickety cages used to frighten claustrophobic people. Or sane and rational people.

And then off we went to „Club Decadence” disco.

The club was like many others we encountered on our trip later – face kontrol (to keep out ugly / poorly dressed people), expensive drinks to buy for women, ear-splittingly loud generic dance techno and hookahs. Interesting things about the menu were that it had cigarettes for about a quarter and also had “bodyguard” for about $30. We did not get the bodyguard but probably could have used one later in the night when my companion ended up arguing with five gypsies at 0400 on the street in downtown Kiev in a suit. After causing a massive scene in the grocery store attempting to purchase foodstuffs and booze over the course of about an hour with numerous gypsies hounding him we managed to escape back to our apartment, but only barely. Maybe they just didn’t know what to make of a six-and-a-half foot Norwegian man in a suit stumbling around in the middle of the night but I think they just couldn’t figure out how to successfully rob him. I think in most major cities in the world we would have ended up imprisoned or found ourselves awaking in a bathtub filled with ice missing kidneys. We escaped without incident though, hooray for Kiev. My companion passed out at around 7am, leaving us a small wooden club with a piece of paper wrapped around it instructing us to wake him with it if necessary.

Our apartment was in the most central location possible in all of Ukraine: across the street from the Maidan Nezalezhnosti (Independence Square) where basically all of the political protest shit went down recently.

In almost all of our travels there was little sign of any serious trouble in the country. The only thing we really saw that gave some indication that things were not status quo was the Maidan, where there were signs of military action, the occasional aggrieved person with a megaphone shouting stuff at a crowd of a half dozen curious onlookers, and a row of photographs and flowers for dead Ukrainian soldiers.

Perhaps a sign that things might be all well?
Perhaps a sign that things might not all be well?

Mostly it was happy tourists and costumed things trying to get money to be in photographs with people.

In front of the statue of the Four Badasses who founded Kiev

There was plenty to see in Kiev. Just taking the metro was an adventure. Like the Moscow metro the stations were hundreds of feet underground. You know, in case of a nuclear attack. The trains came every few minutes and were really quite fast and efficient. The system really puts any bay area public transit to shame, which is sort of depressing.

Deep, deep underground

I don’t come across too many mentions of world war II in my daily life here in San Francisco but in eastern Europe it is definitely a Thing that Happened Recently.

Rodina Mat, statue of the motherland with giant hammer and sickle on the shield
Rodina Mat’, enormous statue of the motherland with giant hammer and sickle on the shieldIMG_0443

Kiev is one of the “hero cities,” an honor that I think was bestowed on cities that did a good job of killing nazis. They all have monuments to the other Soviet hero cities.

Glory: Moscow, Leningrad
Volgograd, Kiev, etc

Here are some WWII armaments:

A nuke
A nuke

IMG_0467 (1)

Katyusha rocket truck, pride of Ukraine
Katyusha rocket truck, pride of Ukraine

The Katyusha rocket truck is a big deal, there is reverence for it including a famous song.

Also we saw some cool churches and stuff.



More disco
More disco
Amazing Georgian feast
Amazing Georgian feast
Ukrainian feasting
Ukrainian feasting. I don’t know what John is doing here
Awesome Kiev locals who showed us around
Last night in Kiev

After more disco and drinking and sight seeing it was time to pack our bags and say goodbye to Kiev, and get some obligatory tourist photos.




Time to go seaside!

Odessa is a nice resort sort of town. It has beaches and sort of a boardwalk and the weather is totally absolutely perfect. It’s on the Black Sea, which was surprisingly warm. Lots of people swam and jet skied in it.

Club „Ibiza”
Dan enjoying himself
Dan enjoying himself

Odessa was a bit of a strange place. The resort area where we were seemed like a fancy amusement park sort of place, with a water park, disco, lots of fancy theme restaurants, children’s rides, high-end clothing stores all in a big strip. It was sort of creepy though because it was almost entirely empty. This was a recurring theme: nice shops and restaurants, but nobody buying anything. I don’t know if this is the normal state of the Ukrainian economy but I doubt it.

Saw a lot of this.

Downtown Odessa had some nifty old buildings. There was a statue to Catherine The Great who officially created the city. She also gave my ancestors a land grant for being German to come farm winter wheat near Odessa. Many years later some goons of Stalin came and shot my great grandfather who tried to defend it. So these things go I guess.


I think this is the opera house
I think this is the opera house

Parts of Odessa were pretty grimy too. One of our party members said last visit he got chased by a pack of rabid dogs, which just kinda wander around. Fortunately we did not have any random encounters of that nature.


After a couple days of rest enjoying the sun and sea we were off to the final leg of our adventure, Lviv.

Lviv is where the real action began. This city is so densely packed with awesomesauce that I advise everyone to stay away to leave more awesomeness for us. Two adventurers in our party had spent time here before and knew where to go. Probably the best part of Lviv is the themed bars. There are numerous outrageous bars and restaurants that are just unlike anything I’ve ever been to in the U.S. or anywhere else for that matter.

There’s the House of Legends, a seven story bar staffed by midgets with a different theme on each level, like a lion tamer room (there was a group eating lunch inside a lion cage), or a cobblestone room with a seven-segment display showing the current number of rocks in the cobblestone roads and made up historical plaques next to rocks.


Current number of cobblestones in Lviv
Current number of cobblestones in Lviv

Another bar is a gas lamp bar with fake coal mine entrance. On the roof is a rickety Lada, cause why not.



At a famous bar (more on it in a bit) there’s a sign with masonic symbols pointing upstairs titled “The Most Expensive Galician Restaurant” but when you go upstairs there are just doors to flats. If you knock on one they let you into a small soviet-style apartment until you ask about the restaurant. Then they open a fake wall and let you in. All the prices on the menu are ten times the actual price but they charge you full price if you don’t act like you’re in the know.

We eat upon the Level and fart upon the Square

There’s another restaurant that we didn’t get a chance to go to that is Jewish-themed. Apparently there are no prices on the menu, you have to haggle. The absolute craziest place though was definitely Kryjivka. To get in you knock on a door, and a man in a soldier uniform opens the door holding a machine gun and demands the password. You must tell him “Слава Українi” and then he pours you a shot of honey vodka. You drink it and go down into a bunker.

Moskal detektor - (slimy) Russian Detector
Moskal detektor – (slimy) Russian Detector


The place is УПА-themed, which means it’s basically dedicated to the glory of killing nazis and soviets I think. It’s nuts. In the backyard is some sort of satanic transformer.


And on the roof is some artillery.




So yeah. Check the place out next time you’re in Lviv.

There was also a “brewery theater” which had some pretty special local varieties

Obama ttout
Obama stout
Putin sit on dick
Putin sit on dick
Yeah uh..
Yeah uh..

Anyway the restaurants are awesome. I think one guy owns them all. He has a handy map:


Our party eating yet more sausages, borscht, and pickled things

There was plenty more in Lviv besides the highly entertaining restaurants. We checked out the armory, containing various normal item drops from Diablo II.

Act II
(Act II)
Some software engineers at work
Some software engineers at work
Java street art
Java street art
Ukrainian book faire
Ukrainian book faire
Lots of Putin toilet paper for sale
Lots of Putin toilet paper for sale
A truly massive cemetery
A massive cemetery
A “Just Married” VW bus
I don't even know what's going on here
I don’t even know what’s going on here

Oh yeah, and our hotel was reallly nice. I heard it was a present for some oligarch’s girlfriend. It was really posh as far as Ukraine goes. The elevator did not seem like it was about to plunge into the abyss at any moment. And the place was pretty empty. I think it was maybe $30/night.



One thing I definitely noticed about Lviv is that there was a lot more visible Ukrainian national pride there. It’s pretty far west whereas all the recent troubles have been in the far east of the country. There were Ukrainian flags everywhere, tons of nationalist trinkets for sale everywhere and of course Kryjivka.

All in all, two weeks well-spent. I got to see all kinds of great places and historical sights, everyone we met was very friendly and helpful, we could basically do anything we wanted at any hour and I was even able to get by sort of decently with my very poor Russian skills.

Our Norwegian companion loves the place so much his dream is to move there and become a citizen. I think he had more national pride than a lot of the actual Ukrainians. There’s definitely something about the place that has a strong appeal if you’re sick of the politically correct, expensive western world and want to bust out of the liberal fantasy bubble that envelopes places like San Francisco and Oslo. For a little bit anyway. Maybe not for too long.

Programming With The Lowest Common Denominator

What’s the hippest programming language in 2015 that all of the fashionably dressed hungry young developers can’t get enough of? Something new? Something compiled? Something fast? Something well-structured? Something maintainable? Something someone spent more than two weeks designing? No! JavaScript!

I recall many years ago when node.js was new on the scene, a contractor who wrote one of the shoddiest hackjobs of C#.NET code I’ve ever seen, gave me some unsolicited advice: “hey you should rewrite your application in JavaScript.” I thought he was high on bath salts. “What, the backend??” I asked incredulously. He replied now it was now possible to do so. I called him a raving lunatic, thinking no one would ever voluntarily do such a thing.

Several years later JavaScript is everywhere. Frontends, backends, HTML5 mobile “apps”, cars, TVs, watches. As usual my prediction that the very suggestion of writing JavaScript by choice would be subject to ridicule was totally wrong. This has led me to reconsider my assumptions and opinions; if so many people are this into JS and promoting its usage, I must be missing something. So let’s go though some common objections.

JavaScript”. While being named “JavaScript” the language actually has very little to do with Java at all, although the syntax of both is closely derived from C. Netscape wanted a competitor to Java so some marketing genius decided to stick “Java” in the name. This caused a lot of confusion for a long time. The structure and primary features of the language are based more on Scheme or Lisp, most notably the presence of an “eval” directive and first-class functions which neither Java nor C have. There is a lot that has been said on the beauty of distilling the machinery of describing programs down to their essence of eval/apply that I won’t bother repeating here. All you have to do is watch this minute-long video:

JavaScript was a rather bare-bones language in terms of features and goodies to help structure large programs or catch common mistakes at compile-time, and there’s nothing wrong with that. In its original uses, simplicity was a great benefit since no programs were longer than ten lines and the fanciest thing you could do was change the status bar text.

If you wanted to mess about with “classes” and reusable components you were completely free to, but you had to build a system yourself. This gave programmers great flexibility to innovate and try out different models over time that evolved with the web as they weren’t locked into a rigid system of object-orientation or modularity. Many competing and incompatible frameworks like x.js, Moo, YUI, Prototype, jQuery and a zillion others popped into existence to fill in the gaps, with most large companies just making up their own. Eventually jQuery “won” the cross-browser compatibility layer competition and most everything in the browser uses that today.

There is much to be said for building the bare minimum and letting people design new layers to be put on top and having them all battle it out. I think JavaScript takes this a little too far though. Its complete disregard for typing sanity or really typing of any sort is hostile to writing performant code or code that you can sanity-check before running it. There’s a good little talk on this breakage.
Fucked up JavaScript examples

When writing a serious piece of software, such as one that a company or important system relies on, you want it to be maintainable, catch obvious bugs ahead of time, and make it easy for others to join your project and be able contribute easily. JavaScript encourages none of these things. JavaScript does not have any concept of modules, imports or dependency injection. Yes there are such capabilities in Node but I am speaking of the JavaScript one writes that is compatible everywhere. There is one and only one valid reason to write JavaScript as a server-side programming language and that is because you can share the same language and code on the frontend and backend. There are innumerable better-suited languages for pure server-side or application programming besides JavaScript so if you want to consider JavaScript that doesn’t run in a web browser then fine, but who cares? The only useful feature of JS is that it is executable by the lowest common denominator, namely web browsers.

Without a standard object model of inheritance or interfaces people generally roll their own, borrowing features from other languages that they miss from the prototyping system, such as the ability to call the superclass version of an overridden method, a nice feature available in obscure toy languages such as python, ruby, C#, java, perl, actionscript3, PHP and every other language except C++ (due to multiple inheritance). If you want this feature in JavaScript, just roll your own as the “Secrets of the JavaScript Ninjainstructs.

Typing. Ahh static typing. The way you save me from conversion errors, invalid function parameters and invalid method and property accesses. Too bad JavaScript provides none of those things. Good solution to the problem of BigInt and floating point conversion errors though: all numbers are a double. End of story. While you may think that JavaScript is not suitable for checking instance types due to its dynamic nature, you would be wrong! In fact another JavaScript-compatible language called ActionScript3 has the most pleasing type declaration system imaginable. In addition to actual semantic classes and interfaces (which sit alongside the prototype chain) it allows you to optionally specify static types for all objects and primitives. Rarely do you actually need to modify the methods on an object at run-time, but you can optionally declare that type as dynamic if you don’t want the compiler to yell at you. Or you can just eschew static types altogether if you aren’t writing anything serious. That language was a great example of how one could actually write very beautiful and well-structured code where many of your mistakes would be caught by the compiler, and it was a derivative of JavaScript! It is the one thing that I will truly miss from the demise of Flash.

Modularity. It’s great when you can encapsulate little bits of reusable functionality into packages that export only the public bits of your interface and declare their dependencies. Really makes it easy to develop serious software. Yep.

But really, what’s the point of complaining? No language is perfect. Lots of innovative things are happening in JavaScript, some day it may be possible to write ES6 and expect most peoples’ web browsers to understand it. There is undeniable progress here, and even a new push to replace it with something even more fundamental. I believe the strongest argument in favor of JavaScript is: it runs on servers and web browsers. No other language can make that claim. But just because you can does not necessarily mean that you should.

I’m not entirely sold on the benefits of the frontend and backend being so tightly coupled. I think of the JavaScript running in a browser as my application’s user interface, rather than thinking of the whole package as a “website.” Websites are a collection of HTML, CSS and JavaScript that are entirely handled client-side. A web application is an application that happens to have a user interface that is implemented in something a web browser or view can understand that talks HTTP. If there’s one rule every application developer should live by it is this: separate presentation from business logic. Meaning don’t couple your interface to your code that deals with the database. The flexibility and constraints this gives you are extremely valuable into not getting sloppy and putting functionality where it doesn’t belong. I’ve seen a number of frameworks now that make it way way too easy to query a database directly from JavaScript, hiding away all of the logic of building a query and validating it. This scares me and it should scare you too.

In all seriousness, all programming languages are roughly the same. A mistake many people make is falling into the trap of thinking that there are vast differences between languages and that they take years to master. Most common programming languages do pretty much the same things and are derived from C and have similar syntax and features, and they can be picked up in a week or two by someone who is an experienced developer. For some reason many people still ask “what language do you code in” or say “we need to hire some python people” or love to argue for hours about which programming language is best, which is an unsolvable question and just leads to religious wars.

That said, there ARE major differences between programming in different languages, but not because the languages themselves are hugely different. It’s much more about the problem you are trying to solve, the community you interact with and the quality of the libraries you are going to rely on to build something useful. These things can and do vary wildly. Python is a great general-purpose language with a friendly community, Ruby is probably a bit more web-focused but mostly the same. PHP is great for making web applications if you’re 12. C# is great for windows desktop applications. J is used by about 20 people in the whole world but is great for numeric applications. Java is not really good for writing software.

The JavaScript community though largely of well-meaning and enthusiastic programmers (not usually “software engineers”) who are fresh out of their Coder Bootcamp and ready to disrupt the stodgy old world of UNIX weenies who waste their time doing silly things like memory management and aren’t up on the latest frontend frameworks. With the notable exception of transpiling C to JavaScript, running JavaScript on an operating system instead of a web browser requires a whole lot of library support for things like networking, threading, files and more. So much to rewrite and make asynchronous!

My previous open source experience was with the Perl community. It had a large number of highly competent, serious-minded, occasionally smelly geeks who cared about designing and engineering high-quality software.

Definitely one of the most important aspects of any language is the community because they are the ones building the software you rely on, answer your questions when you need help, and are responsible for maintaining code quality. So when I first gave node.js a try I decided to write an IRC ASCII art flooder bot, always a great way to get acquainted with any new language. One of the requirements for a good IRC bot is that you need to be able to bind to various addresses on a socket to change your source IP up, and the node-irc module was lacking in this capability. No problem thought I, I’ll just submit a pull request with my addition to allow specifying the bind address.

I guess I was used to the community of CPAN maintainers for Perl, where someone would review my patch and merge it in, or tell me they weren’t maintaining that module anymore and give me maintainership, or occasionally tell me to add tests or go fuck myself. Here I got silence. And then more silence. Then, following up two years later, I got a request to rebase it, since every single file in the project had been deleted. I guess someone thought it was cool to come along and delete all files, destroying all history and any hope of merging in the 40 or so outstanding pull requests, and then started a rewrite of the code, which they kinda gave up on halfway through. For some reason this had been accepted.

Instead I had to do some gnarly git incantations to copy a branch of the proper mainline back into master and remove the history of some random asshole shitting all over the module. Then we got things back on track. The guy who helped me with fixing everything was someone I recognized from CPAN, interestingly enough.

While I can’t say for certain that this one incident is typical of most of the people contributing to JavaScript and npm modules, more than a few anecdotes and questionable practices are found here. More than most. Some gems include multiple people asking why their node application crashes on invalid input when eval()-ing JSON from a client to parse it, callback functions a dozen levels deep, and attempting to rewrite build systems in lots of ways that suck. Why bother with make when it’s not even written in JavaScript, right?

I don’t want to hate on JavaScript too much. Everything sucks if you dig deeply enough into it. And JavaScript is the only option for a lot of applications, specifically their user interfaces. When you do have the choice though, for server-side software, I still think there are better options.

For more, please see this talk from the year 2035 on the birth and death of JavaScript.

Flash Sucks HTML5 Sucks Everything Sucks

A lot of people are making a lot of noise about expunging Flash from the web, mostly for security reasons as more and more vulnerabilities are discovered in it. One of the first people to make a big deal about it was Steve Jobs when he introduced iOS, claiming it was slow and battery-draining and “proprietary,” as in not open-source.

I’m not the biggest fan of closed web technologies even though I’ve developed with Director, Shockwave, Flash and Flex. It’s really time that Flash died. The problem is, there still aren’t good ways to do video on the web without it.

Video formats on the web are still an unsolved problem, and they’re mostly an unsolved problem because of Apple and Microsoft’s fault. HTML5 introduced the <video> element for playing back (pre-recorded) video, but it didn’t define a codec to be the standard, essentially leaving it up to the popular browser makers to support whatever they feel like. Microsoft and Apple could have used this opportunity to embrace an open and relatively patent unencumbered video format along with Mozilla’s Firefox and Google’s Chrome. Then everyone could happily use HTML5 video.

Instead of choosing a video format that everyone could be free to use for video on the web, Apple and Microsoft have steadfastly refused to offer native support. They only support the format h.264, which is heavily protected by patents and must be licensed. While it is one of the best quality standards its legal status and closed nature preclude it from being supported by other browsers.

Why would Apple and Microsoft force everyone to use this one format that requires licensees to cough up cash? It could possibly maybe have something to do with the fact that they are both members of the H.264 licensor pool of the MPEG Licensing Association so they benefit financially from ruining any hope of open formats for video on the web. Pretty shitty, but patent holders gotta be dicks, right?

No they don’t as a matter of fact. Compare this to Google’s efforts on the HTML5 video front. Google purchased the rights to a competing codec called VP8 (which they renamed to webm) and gave away the rights to use the codec to everyone for free in perpetuity. They basically spent a bunch of time and money and effort to promote a free and unencumbered option that was at least better than the only major open option at the time (Ogg Theora). Turned out that VP8 probably infringed on a number of H.264 patents but at least they tried to play nice and not hold the entire world of web users hostage at the point of a Flash gun and demanding they fork over money to create video content for the web.

So far I’ve only been referring to playing back video files on the web. The situation is even more depressing with streaming live video. Apple decided to only support HTTP Live Streaming, a new format they devised which is made up of chunked MPEG transport stream segments in little 10-second files and a modified mp3 playlist file. It’s nice because they can be handled by normal web servers and CDNs but it’s really pretty shitty in terms of encoding headache and latency imposed by the segmentation.

Of course the best thing about companies that don’t give a flying fuck about promoting open standards is that you end up with lots to choose from! Similar to Apple’s HLS is an actual attempt at a real standard called MPEG DASH, which is great except that it’s not actually supported by anything except some fancy TVs and Chromecast.

Never one to miss a chance to make up their own incompatible standard, Microsoft also has a very similar option called Microsoft Smooth Streaming and Silverlight, which is a lot like Flash except it isn’t supported by anything except IE and is made by Microsoft instead of Adobe. So in Microsoft’s view it’s great, I guess.

In summary, Flash really ought to die. It would just be great if we had a free standard to replace it instead of Apple and Microsoft trying to force everyone to use H.264 which they happen to be in the business of licensing.