A Night At The TechCrunch ‘Crunchies’

I’d received a free ticket to attend the 9th annual TechCrunch Crunchies award ceremony from one of my journalist friends, assuming they had something more important to report on, like some grass growing.

This celebratory gala event is put together by the fine folks at TechCrunch to honor the brave pioneers and visionaries in Silicon Valley and to give recognition to the humble and under-appreciated venture capitalists that have brought so much hope and prosperity to our peers.

IMG_0644

The event, while not strictly formal, was treated by most as an occasion to put on their finery as though attending a real gala or ceremony thus greatly livening the atmosphere, at least as much as can be done at the San Francisco War Memorial Opera House. Slightly diminishing it perhaps were the advertisements for Toyota Priuses everywhere, accompanied by the cars themselves.

screen-shot-2016-02-09-at-1-43-00-pm.png
Photo by TechCrunch

This Year’s Fashions

All of the guests that went to the trouble to get their grown man or woman on were the very picture of distinguished radiance, play-acting in their nouveau riche dress-up party. The event was meticulously styled up to resemble the Academy Awards but with more sensible hybrid driving options and white people.

In the invitation letter I was tastefully informed,

Attire is up to you. At past Crunchies, attendees have dressed down and dressed up but the majority comes dressed to impress.

All eyes were on the outfits sported by those desiring to impress, and boy they outdid themselves this year!

Mrs. A. C. Came replete in a Vera Wang tout ensemble worn with tasteful aplomb. The hangers-on ordered her many drinks though possibly for the purposes of pouring onto her evening skirt topped with a Gap white button-down.

Screen Shot 2016-02-18 at 4.08.52 PMMr. B.-B. G. was strolling the promenade in his Men’s Wearhouse T-Bone Chuck Blazer with Carolina Herrara’s signature white dress shirt, attracting gazes from admirers wanting to take him for a spin in their new sedans.

 

Mr. M. A. as usual went in his signature Karl Lagerfeld chic-waiter look with belted waists and sans chapeau, letting his unusually pointy bald head complete the space alien wedding singer look. He was measurably engaged, posting a mere three dozen tweets during the entire span of the ceremony.

The Snapchat Ghost was in an impeccably tailored Tory Burch white inflatable ghost costume which drew hundreds of selfie-takers into his orbit of ghostiness and cash flow positive magnetism.

Cavi-VEVIAAJ3VO.jpg

The Awards

The MC for the evening was some sort of actress who was actually on television sometimes, though I honestly have no idea. To her credit she was pretty mean to the audience and made fun of the crowd a lot, although nearly everyone I spoke to mentioned that last year the host was someone from HBO’s Silicon Valley and was really really drunk and went way over the line in ripping the attendees a new asshole for being terrible human beings. Needless to say they were glad he wasn’t invited back.

tj-miller-techcrunch-crunchies
Really sad I missed it

But enough about the hosts, let’s talk about some of this year’s nominees and winners!

The TechCrunch Crunchies are all about Tech. As in technology. As in boilerplate database-driven software written in high-level boutique scripting languages employing proprietary walled-garden frameworks. Innovation and disruption are the name of the game, breaking down old staid ideas and replacing them with dynamic new market-driven efficiencies. So it should be no surprise that the Best Mobile App for 2016 went to Facebook’s Messenger app, a revolutionary new take on AOL Instant Messenger, like the ICQ of Western Europe, or kind of like WhatsApp but with ugly avatars (wait shouldn’t they be the same thing by now? Didn’t Facebook buy them?)

Props to Mark Zuckerberg of Facebook, winner of CEO Of The Year (his second award). Dude has hella cash so you pretty much gotta give it to him I guess.

Biggest Social Impact was won not by the people fighting child sexual exploitation as expected but code.org, which teaches people to program.

Best overall startup was a tough race between Slack, Docker, Snapchat, Uber and Xiaomi. I had hopes for Docker but suspected I may have been the only person in the audience who’s actually used it. The award went to the service that everyone was seen using shortly after the event ended – Uber! I feel really happy for their CEO Travis Kalanick, we were all pulling for him! Maybe he can pull more chicks with his Uber hat at The Battery now.

IMG_0649

Mega congratulations are in order for Bill Gurley, winner of VC of the Year. I don’t know who this man is but he sure is tall and could buy and sell my whole family on a whim! I enthusiastically clapped for his acceptance speech along with the rest of the crowd. He then went home to bathe in a tub of rare Wu-Tang albums to wash the hoi polloi germs off.

Scott and Cyan Banister, actual decent human beings, won Best Angel Investor(s). They have a sweet First Amendment Clinic in support of free speech legal amicus briefs which I’m a fan of.

IMG_0647
Also Cyan was one of the few people who didn’t look like they were wearing a dress ironically

 

The Press

I sat in the press section by Owen Thomas, former Valleywag gossip columnist and noted writer of words. He recently quit his job at ReadWriteWeb and was attending the ceremony on behalf of his personal enterprise Ditherati. When questioned as to the nature of his new venture he replied that it was still being figured out.

A reporter seated next to me and I had some lively chats guessing who would win the awards. I did pretty good at predicting the winners and should have put down some money. Mostly I was a royal jerk, hollering and clapping loudly for Facebook and Apple, deriding the stupid nominations and generally trying to get thrown out. I am ashamed to report that I failed in my quest. Next year I’m bringing a bottle of something vicious to stoke the fires of righteous disgust. Or maybe the bubble will finally burst and impoverish the usual guests, as my Uber driver told me he prays for daily.

Congrats to all the winners!

Photo: the bold italic
Photo: the bold italic

Thanks to Toyota for their sponsorship – Discover the Redesigned Prius Now!

Poland and Ukraine in 2015

National Museum of the History of Ukraine in World War II
National Museum of the History of Ukraine in World War II

I just went on a vacation to Eastern Europe, one of my most favorite places on earth. Specifically Poland and Ukraine.

Wrocław, Poland is a fantastic city that almost nobody in the USA has ever heard of or even knows how to pronounce (“vraswav”). It is the fourth largest city in Poland and has a number of excellent universities. I’ve had the pleasure of working with a number of software engineers there and they are exceptionally sharp and reliable, and very friendly as well. Since the government gives everyone free schooling through five-year university the population in general is very educated and speaks better english than most outsourcing destinations.

Wrocław City Square
Wrocław City Square

As a result of recent history Wrocław is a pleasing mixture of east and west Europe, having been part of both the third reich and the USSR. Many of the shops and streets resemble Paris or Germany but there are also plenty of communist-style Soviet housing complexes and a definite slavonic ring to the language. As far as the economy goes, Poland is definitely one of the brightest success stories in Europe, being the only eurozone economy that did not fall into recession after 2008’s crash, having a highly educated population of english speakers and engineers, and a close trade relationship with Germany. And since they are not using the Euro it’s cheap for Americans to hire poles (and vacation there).

Beers on the water
Beers on the water
Vodka drinking at a BBQ with our good friends
Vodka drinking at a BBQ with our good friends
Found a WWII memorabilia shop
A WWII memorabilia shop

Warsaw is pretty chill too. Its buildings are much more modern as the entire city was flattened by Nazi bombs.

Royal gardens in Warsaw
Royal gardens in Warsaw

It’s a bit more touristy.

Warsaw city square
Warsaw city square
Building of culture - donated by Stalin
Building of culture – donated by Stalin

After a short tour though Warsaw it was time to head to Kiev, the capital of Ukraine. When we got there it was hot, muggy, draped in total fog and our taxi driver had a giant crack in the windshield of his Lada.

We weren’t really sure what to expect in Ukraine. According to the news in America and the advice of random uninformed friends it sounded like we might be headed into a war zone or collapsing post-Soviet radioactive wasteland. All utter nonsense.

First we checked in to our apartment.

Soviet elevators are not like western elevators. They are more like devices used to frighten claustrophobic people. Or sane and rational people.
Soviet elevators are not like western elevators. They are more like small rickety cages used to frighten claustrophobic people. Or sane and rational people.

And then off we went to „Club Decadence” disco.

The club was like many others we encountered on our trip later – face kontrol (to keep out ugly / poorly dressed people), expensive drinks to buy for women, ear-splittingly loud generic dance techno and hookahs. Interesting things about the menu were that it had cigarettes for about a quarter and also had “bodyguard” for about $30. We did not get the bodyguard but probably could have used one later in the night when my companion ended up arguing with five gypsies at 0400 on the street in downtown Kiev in a suit. After causing a massive scene in the grocery store attempting to purchase foodstuffs and booze over the course of about an hour with numerous gypsies hounding him we managed to escape back to our apartment, but only barely. Maybe they just didn’t know what to make of a six-and-a-half foot Norwegian man in a suit stumbling around in the middle of the night but I think they just couldn’t figure out how to successfully rob him. I think in most major cities in the world we would have ended up imprisoned or found ourselves awaking in a bathtub filled with ice missing kidneys. We escaped without incident though, hooray for Kiev. My companion passed out at around 7am, leaving us a small wooden club with a piece of paper wrapped around it instructing us to wake him with it if necessary.

Our apartment was in the most central location possible in all of Ukraine: across the street from the Maidan Nezalezhnosti (Independence Square) where basically all of the political protest shit went down recently.

In almost all of our travels there was little sign of any serious trouble in the country. The only thing we really saw that gave some indication that things were not status quo was the Maidan, where there were signs of military action, the occasional aggrieved person with a megaphone shouting stuff at a crowd of a half dozen curious onlookers, and a row of photographs and flowers for dead Ukrainian soldiers.

Perhaps a sign that things might be all well?
Perhaps a sign that things might not all be well?

Mostly it was happy tourists and costumed things trying to get money to be in photographs with people.

In front of the statue of the Four Badasses who founded Kiev

There was plenty to see in Kiev. Just taking the metro was an adventure. Like the Moscow metro the stations were hundreds of feet underground. You know, in case of a nuclear attack. The trains came every few minutes and were really quite fast and efficient. The system really puts any bay area public transit to shame, which is sort of depressing.

Deep, deep underground

I don’t come across too many mentions of world war II in my daily life here in San Francisco but in eastern Europe it is definitely a Thing that Happened Recently.

Rodina Mat, statue of the motherland with giant hammer and sickle on the shield
Rodina Mat’, enormous statue of the motherland with giant hammer and sickle on the shieldIMG_0443

Kiev is one of the “hero cities,” an honor that I think was bestowed on cities that did a good job of killing nazis. They all have monuments to the other Soviet hero cities.

IMG_0475
Glory: Moscow, Leningrad
IMG_0476
Volgograd, Kiev, etc

Here are some WWII armaments:

A nuke
A nuke

IMG_0467 (1)

Katyusha rocket truck, pride of Ukraine
Katyusha rocket truck, pride of Ukraine

The Katyusha rocket truck is a big deal, there is reverence for it including a famous song.

Also we saw some cool churches and stuff.

IMG_0511

IMG_0514

More disco
More disco
Amazing Georgian feast
Amazing Georgian feast
Ukrainian feasting
Ukrainian feasting. I don’t know what John is doing here
Awesome Kiev locals who showed us around
Last night in Kiev

After more disco and drinking and sight seeing it was time to pack our bags and say goodbye to Kiev, and get some obligatory tourist photos.

IMG_0531

IMG_0540

IMG_0522

Time to go seaside!

Odessa is a nice resort sort of town. It has beaches and sort of a boardwalk and the weather is totally absolutely perfect. It’s on the Black Sea, which was surprisingly warm. Lots of people swam and jet skied in it.

IMG_0581
Club „Ibiza”
Dan enjoying himself
Dan enjoying himself

Odessa was a bit of a strange place. The resort area where we were seemed like a fancy amusement park sort of place, with a water park, disco, lots of fancy theme restaurants, children’s rides, high-end clothing stores all in a big strip. It was sort of creepy though because it was almost entirely empty. This was a recurring theme: nice shops and restaurants, but nobody buying anything. I don’t know if this is the normal state of the Ukrainian economy but I doubt it.

Saw a lot of this.

Downtown Odessa had some nifty old buildings. There was a statue to Catherine The Great who officially created the city. She also gave my ancestors a land grant for being German to come farm winter wheat near Odessa. Many years later some goons of Stalin came and shot my great grandfather who tried to defend it. So these things go I guess.

IMG_0577

I think this is the opera house
I think this is the opera house

Parts of Odessa were pretty grimy too. One of our party members said last visit he got chased by a pack of rabid dogs, which just kinda wander around. Fortunately we did not have any random encounters of that nature.

IMG_0612

After a couple days of rest enjoying the sun and sea we were off to the final leg of our adventure, Lviv.

Lviv is where the real action began. This city is so densely packed with awesomesauce that I advise everyone to stay away to leave more awesomeness for us. Two adventurers in our party had spent time here before and knew where to go. Probably the best part of Lviv is the themed bars. There are numerous outrageous bars and restaurants that are just unlike anything I’ve ever been to in the U.S. or anywhere else for that matter.

There’s the House of Legends, a seven story bar staffed by midgets with a different theme on each level, like a lion tamer room (there was a group eating lunch inside a lion cage), or a cobblestone room with a seven-segment display showing the current number of rocks in the cobblestone roads and made up historical plaques next to rocks.

IMG_0749

Current number of cobblestones in Lviv
Current number of cobblestones in Lviv

Another bar is a gas lamp bar with fake coal mine entrance. On the roof is a rickety Lada, cause why not.

IMG_0747

IMG_0744

At a famous bar (more on it in a bit) there’s a sign with masonic symbols pointing upstairs titled “The Most Expensive Galician Restaurant” but when you go upstairs there are just doors to flats. If you knock on one they let you into a small soviet-style apartment until you ask about the restaurant. Then they open a fake wall and let you in. All the prices on the menu are ten times the actual price but they charge you full price if you don’t act like you’re in the know.

11053157_10207444848549649_2139937664523907512_n
We eat upon the Level and fart upon the Square

There’s another restaurant that we didn’t get a chance to go to that is Jewish-themed. Apparently there are no prices on the menu, you have to haggle. The absolute craziest place though was definitely Kryjivka. To get in you knock on a door, and a man in a soldier uniform opens the door holding a machine gun and demands the password. You must tell him “Слава Українi” and then he pours you a shot of honey vodka. You drink it and go down into a bunker.

Moskal detektor - (slimy) Russian Detector
Moskal detektor – (slimy) Russian Detector

11952695_10207421359642441_5067349547427786172_o

The place is УПА-themed, which means it’s basically dedicated to the glory of killing nazis and soviets I think. It’s nuts. In the backyard is some sort of satanic transformer.

IMG_0765

And on the roof is some artillery.

IMG_0769

IMG_0772

IMG_0771

So yeah. Check the place out next time you’re in Lviv.

There was also a “brewery theater” which had some pretty special local varieties

Obama ttout
Obama stout
Putin sit on dick
Putin sit on dick
Yeah uh..
Yeah uh..

Anyway the restaurants are awesome. I think one guy owns them all. He has a handy map:

IMG_0734

IMG_0754
Our party eating yet more sausages, borscht, and pickled things

There was plenty more in Lviv besides the highly entertaining restaurants. We checked out the armory, containing various normal item drops from Diablo II.

Act II
(Act II)
Some software engineers at work
Some software engineers at work
Java street art
Java street art
Ukrainian book faire
Ukrainian book faire
Lots of Putin toilet paper for sale
Lots of Putin toilet paper for sale
A truly massive cemetery
A massive cemetery
A
A “Just Married” VW bus
I don't even know what's going on here
I don’t even know what’s going on here

Oh yeah, and our hotel was reallly nice. I heard it was a present for some oligarch’s girlfriend. It was really posh as far as Ukraine goes. The elevator did not seem like it was about to plunge into the abyss at any moment. And the place was pretty empty. I think it was maybe $30/night.

IMG_0780

Chillin
Chillin

One thing I definitely noticed about Lviv is that there was a lot more visible Ukrainian national pride there. It’s pretty far west whereas all the recent troubles have been in the far east of the country. There were Ukrainian flags everywhere, tons of nationalist trinkets for sale everywhere and of course Kryjivka.

All in all, two weeks well-spent. I got to see all kinds of great places and historical sights, everyone we met was very friendly and helpful, we could basically do anything we wanted at any hour and I was even able to get by sort of decently with my very poor Russian skills.

Our Norwegian companion loves the place so much his dream is to move there and become a citizen. I think he had more national pride than a lot of the actual Ukrainians. There’s definitely something about the place that has a strong appeal if you’re sick of the politically correct, expensive western world and want to bust out of the liberal fantasy bubble that envelopes places like San Francisco and Oslo. For a little bit anyway. Maybe not for too long.

AWS Orchestration / Unemployed DevOps Professionals

Now that we live in the age of ~the cloud~ it strikes me that for many software projects the traditional roles of system administrator and their more recent rebranded “DevOps” are not strictly required.

I can’t speak much to other cloud hosting platforms but Amazon Web Services really is the flyest shit. All of Amazon’s internal infrastructure has been built with APIs to control everything for many years by decree of Jeff Bezos, the CEO. This was a brilliant requirement that he mandated because it allowed Amazon to become one of the first companies able to re-sell its spare server capacity and make their automated platform services available to everyday regular developers such as myself. They’ve been in the business of providing service-oriented infrastructure to anyone with a credit card longer than most everything, and their platform is basically unmatched. Whereas before setting up a fancy HTTPS load balancer or highly available database cluster with automated backups was a time-consuming process, now a few clicks or API calls will set you up with nearly any infrastructure your application requires, with as much reliability and horsepower as you’re willing to pay for.

I’ve heard some people try to argue that AWS is expensive. This is true if you use it like a traditional datacenter, which it’s not. If you try running all your own services on EC2 and pay an army of expensive “DevOps” workers to waste time playing with Puppet or Chef or some other nonsense then perhaps it’s a bit costly. Though compared with power, bandwidth, datacenter, sysadmin and hardware costs and maintenance overheard of running on your own metal I still doubt AWS is going to run you any more. In all likelihood your application really isn’t all that special. You probably have a webserver, a database, store some files somewhere, maybe a little memcached cluster and load balancer or two. All this can be had for cheap in AWS and any developer could set up a highly available production-ready cluster in a few hours.

Us software engineers, we write applications. These days a lot of them run on the internet and you wanna put them somewhere on some computers connected to the internet. Back in “the day" you might have put them on some servers in a datacenter (or your parents’ basement). Things are a little different today.

Some time ago I moved my hosting from a traditional datacenter to AWS. I really didn’t know a lot about it so I asked the advice of some smart and very experienced nerds. I thought it would be pretty much the same as what I was doing but using elastic compute instances instead of bare metal. They all told me “AWS is NOT a datacenter in the cloud. Stop thinking like that.”

For example, you could spin up some database server instances to run MySQL or PostgreSQL OR you could just have AWS do it for you. You could set up HAproxy and get really expensive load balancers, or simply use an elastic load balancer. Could run a mail server if you’re into that, but I prefer SES. Memcached? Provided by ElastiCache. Thinking of setting up nagios and munin? CloudWatch is already integrated into everything.

Point being: all the infrastructure you need is provided by Amazon and you don’t need to pay DevOps jokers to set it up for you. AWS engineers have already done all the work. Don’t let smooth-talking Cloud Consultants talk you into any sort of configuration management time-wasters like Puppet. Those tools impose extra overhead to make your systems declaratively configured rather than imperatively because they are designed for people who maintain systems. In EC2-land you can and should be able to kill off any instance at any time and a new one will pop up in its place, assuming you’re using autoscaling groups. You are using ASgroups, right? You will be soon!

When you can re-provision any instance at will, there is no longer any need to maintain and upgrade configuration. Just make a new instance and terminate the old ones. Provision your systems using bash. Or RPMs if you want to get really fancy. You really don’t need anything else.

I’m a fan of Amazon Linux, which is basically just CentOS. I use a nifty yum plugin that lets me store RPMs in the Simple Storage Service (S3) and have instances authenticate via IAM instance roles. This is a supremely delightful way of managing dependencies and provisioning instances.

The last piece of the puzzle is orchestration; once you have all of your infrastructure in place you still need to perform tasks to maintain it. Updating your launch configurations and autoscaling groups, deploying code to your servers, terminating and rebuilding clusters, declaring packages to be installed on EC2 with cloud-init and so on. You could do all of this by hand maybe or script it, except that you don’t have to because I already did it for you!

To be totally honest, my AWS setup is pretty freaking sweet. The reason it is freaking sweet is because I listened to grumpy old AWS wizards and took notes and built their recommendations into a piece of software I call Udo – short for Unemployed DevOps.

Udo is a pretty straightforward application. It essentially provides a configuration-driven command-line interface to Boto, which is the python library for interfacing with the AWS APIs. It is mostly centered around autoscaling groups, which are a very powerful tool not only for performing scaling tasks but also for logically grouping your instances. In your configuration file you can define multiple clusters to group your ASgroups, and then define “roles” within your clusters. I use this system to create clusters for development, QA, staging and production, and then in each cluster I have “webapp” roles and “worker” roles, to designate instances which should handle web requests vs. asynchronous job queue workers. You can of course structure your setup however you want though.

Using Udo is as simple as it gets. It’s a python module you can install like any other (sudo easy_install udo). Once it’s installed you create a configuration file for your application’s setup and a Boto credentials file if you don’t already have one. Then you can take it for a spin.

The cluster/role management feature is central to the design. It makes it so you never have to keep track of individual instances or keep track of IP addresses or run any sort of agents on your instances. Finding all of your stage webapp server IPs for example is as easy as looking up the instances in the stage-webapp autoscaling group. You can easily write tools to automate tasks with this information. We have a script that allows you to send commands to an autoscaling group via SSH, which works by reading the external IPs of the instances in the group. This is so useful we plan on adding it to Udo sometime in the near future, but it’s an example of the sort of automation that would normally require fancy tools and daemons or keeping track of IPs in some database somewhere, but is totally simplified by making use of the tools which Amazon already provides you.

Udo has a few nifty features on offer. One handy command is “updatelc” – update launchconfiguration. Normally you cannot modify a launch configuration attached to an autoscaling group, so Udo will instead create a copy of your existing launchconfig and then replace the existing launchconfig on your asgroup, allowing you to apply udo.yml configuration changes without terminating your asgroup. Very handy for not having to bring down production to make changes.

Another powerful feature is tight integration with CodeDeploy, a recent addition to the AWS ops tools suite. As far as I’m aware Udo is the first and only application to support CodeDeploy at this time and I actually have an epic support ticket open with a sizable pile of feature requests and bug reports. Despite its rather alpha level of quality it is extremely handy and we are already using it in production. It allows AWS to deploy a revision straight from GitHub or S3 to all instances in an autoscaling group or with a particular tag, all without any intervention on your part other than issuing an API call to create a deployment. You can add some hooks to be run at various stages of the deployment for tasks like making sure all your dependencies are installed or restarting your app. I’d honestly say it’s probably the final nail in the coffin for the DevOps industry.

Fixing a typo the hard way – UNIX standards style (Part 2)

I use UNIX-based systems quite a lot in my daily life. Linux servers, Mac OS X for my desktop, Android for my phone, and so on. The UNIX specification powers most modern software and technology so I’ve found it fun to read up a bit on the history of it. One highly entertaining historical document I found long ago was the UNIX Haters Handbook. While the technical details described therein are extremely dated today one wonders if the general thrust of the book is still relevant. Specifically that “UNIX weenies” (to use the handbook’s phraseology) are rigid, stuck in the past, resistant to progress and proud of comically ugly hackjobs.

While this may seem harsh considering the virulent success of the UNIX model, I wondered if there might be some truth to all of this entertainingly-packaged vitriol in the pages of the haters from a time before mine. I set out to see if there was some merit to their claims.

There is a very famous quote from one of the original designers and implementors of UNIX, a highly accomplished man named Ken Thompson whom I admire. When asked what he would change if he could do UNIX all over again, he replied “I’d spell creat with an e”.  This is a reference to the flag O_CREAT in the open() function in the C standard library and the associated system call. If you look at the reference documentation for the flags to open() you can see that there is a certain “style” to the naming of the flags. Let’s call it lazy typist style. O_RDONLY, O_RDRW, O_EXCL, etc…

I can only assume that Mr. Thompson got a little too carried away with this style and left off the ‘e’ on O_CREAT, and now regrets this mistake. When given the chance to correct it on the operating system interface in Go he famously made the first public commit (with an assist by Rob Pike) on the project to correct the spelling with what I can only imagine was great aplomb and satisfaction.

One must appreciate this great man’s legacy and his determination to correct such a trivial error that has persisted for so many years. It inspired me to see if it was possible to correct this mistake in the UNIX standard, adhered to by every operating system from Apple’s OSX to Linux to BSD and even Windows. This standard is known as “POSIX”.

To quote the POSIX FAQ:

POSIX is an acronym for Portable Operating System Interface.

The name POSIX was suggested by Richard Stallman. It is expected to be pronounced pahz-icks, as in positive, not poh-six, or other variations. The pronunciation has been published in an attempt to promulgate a standardized way of referring to a standard operating system interface.

Big ups to my main man Richard Stallman on that one.

So I decided to try to correct this simple spelling mistake left over from the 1960’s. After all, how hard could it be?

I decided to start small, simply adding an alias to the GNU standard C library implementation. This is essentially the “GNU” in “GNU/Linux” that Mr. Stallman is always harping about. Now, please note that I did not attempt to rename O_CREAT to O_CREATE, I merely added a definition for O_CREATE so that people writing code in this millennium could benefit from this correction going forward. Actually changing the definition of O_CREAT would break basically everything everywhere. So I proposed my change in the #glibc IRC channel with underwhelming results.

IRC

Long story short, glibc did not seem receptive to my change, which is somewhat reasonable. glibc is an implementation of the POSIX standard, and the ideal way to make this change would be to get the proper spelling included in the standard itself. Then no doubt the glibc project would be delighted to accept my one-line addition.

So let’s try making a change to POSIX. I wonder what is involved?

POSIX is a standard (ratified by the IEEE), though the actual standard itself is decided upon by a nefarious group known as the Austin Common Standards Revision Group. I invite you to take a look at their sweet homepage. It’s really an amazing site.

Bracing myself, I dug deeper. Where do I submit a proposal to correct a typo? Maybe I will try the mailing list, since many old-school open source projects still use those to submit patches.

(I love the sweet HTML 1.0 <TABLE BORDER=2> style btw)

Also, I was informed that I must use “proper netiquette” on this list. What nostalgia.

 

Okay, so to subscribe I need an account with the Open Group. Sounds promising. Who doesn’t love open standards??

Should be no problem. Just a small perfunctory form to fill out…

A half hour later, I’m in!

Uh… cool I guess? This is a really sweet PHP app by the way. If you click too hard it sorta falls over.

image

Moving on. After some searching I found a page that mentions leading the strategy for the UNIX standard. Is this what I want? Unsure. I wonder if I need to be a “platinum member” to work on it?

image

????????

Getting nowhere, I decide to try and find the last POSIX revision publication they put out. It’s Open and stuff, right? I find a sweet download link. Ah, but I must verify my email address before downloading the PDF describing the revisions. Can’t be too careful these days.

image

After some fun times resetting my password because it appeared to forget it and their captcha system didn’t actually work, I was fortunate enough to get a nice download link.

I find these jerks:

image

Well… I mean I’m sure some of those people are good guys. I didn’t mean to imply that ALL of them are jerks, though some of them must be responsible for this sad state of affairs. I did recognize at least one name on that list… oh yeah it’s this guy who accused me of “crapping in the implementation namespace.”

Ah, just write a cross-compiler and get a licensed vendor involved! Of course! Or just make the Open Group a giant pile of money to get my change approved. Good to know.

Obviously that won’t be happening any time soon, at least if I can help it. Let’s try another approach. After some more surfing the world wide web circa 1996 I come across a bug tracker for the Austin Group. Maybe I can report a bug! Just need to register, once more.

Well… okay. Nice mail system (godaddy) there. I think the chapter on sendmail in the UNIX Haters Handbook is my favorite, just by the way.

Fixing the spelling of O_CREAT

In which I attempt to correct a mistake made by Ken Thompson, one the of the original designers of Unix:

diff --git a/bits/fcntl.h b/bits/fcntl.h
 index f273f84..43e8e6b 100644
 --- a/bits/fcntl.h
 +++ b/bits/fcntl.h
 @@ -29,6 +29,7 @@

/* Bits OR'd into the second argument to open. */
 #define O_CREAT 0x0200 /* Create file if it doesn't exist. */
 +#define O_CREATE 0x0200 /* Create file if it doesn't exist. */
 #define O_EXCL 0x0800 /* Fail if file already exists. */
 #define O_TRUNC 0x0400 /* Truncate file to zero length. */
 #define O_NOCTTY 0x8000 /* Don't assign a controlling terminal. */

16:08 < wooster> what do you guys think of my patch? http://pastebin.com/raw.php?i=B8gfZG1m
17:26 < azanella_> wooster, http://pubs.opengroup.org/onlinepubs/9699919799/
17:26 < wooster> my patch is still POSIX-compliant
17:26 < wooster> i did not rename it
17:27 < azanella_> wooster, so which specification defines O_CREATE?
17:28 < wooster> i am merely acceding to Ken Thompson‘s wishes
17:28 < wooster> http://unix.stackexchange.com/questions/10893/what-did-ken-thompson-mean-when-he-said-id-spell-create-with-an-e
17:28 < wooster> O_CREATE is not defined in a specification, but it is not prohibited either by any specification
17:29 < azanella_> wooster, which is the gain of adding a non standard type on glibc? what happen if a program uses it and try to compile on a system that does not define?
17:29 < wooster> is every single thing in glibc only implemented if it is defined in a standard? everything?
17:30 < azanella_> wooster, no, but the extension either make sense to fix an not specified trait or necessity
17:31 < azanella_> this does not add anything, in fact only add confusion because it is a field to duplicate an already define field
17:31 < wooster> as i said, i am attempting to correct an error that ken thompson regrets making
17:31 < wooster> better late than never
17:31 < wooster> and i am attempting to do it in a backwards-compatible fashion
17:32 < azanella_> to late, this change is useless
17:32 < azanella_> imho
17:32 < wooster> i’
ll take ken thompson‘s judgement over yours
17:32 < wooster> no offense
17:35 < azanella_> sure, fell free to send a patch to libc-alpha
17:35 < wooster> thank you
17:35 < wooster> i shall
17:35 < azanella_> I am just foreseeing the possible answer to your patches
17:36 < azanella_> people try to avoid such useless changes that do not follow any specification
17:36 < wooster> understandibly
17:37 < wooster> however, you must admit that the design of unix was not 100% perfect the first time around, and it is useful to have the ability to correct mistakes, especially those mentioned rather famously by one of the original authors
17:41 < azanella_> wooster, in this specific case this change does not make sense, the compiler will get any typo error, there is no gain in expressiveness and you add a non-specification change, that will require additional prepcoressor defines
17:45 < wooster> i accuse you of standing in the way of progress
17:45 < azanella_> right…
17:45 < wooster> i will post to libc-alpha
17:47 < wooster> in the meantime may i suggest some light reading on the subject: http://web.mit.edu/~simsong/www/ugh.pdf
17:47 < azanella_> alright ….

Unix cannot move forward if everyone is resistant to fixing defects in the original design. This mentality of preserving the original misspellings is as much a blight on the POSIX standard as the spelling of “Referer” in the HTTP header specification. We can do better. Adding a vowel should not be obstructed with charges of “that will require additional prepcoressor defines”.
I noticed in a thread elsewhere on the internet that this mistake was corrected in the Go language Apr 04 23:42:14 2011 with Ken’s blessing.

c.f. The first commit after Go was released publicly, by Ken himself: https://github.com/golang/go/commit/c90d392ce3d3203e0c32b3f98d1e68c4c2b4c49b

MySQL vs PostgreSQL – Why You Care

Like many others before and since, my introduction to the world of databases was via MySQL. It was a good enough database that did what I wanted, so I was content to use it without realizing what I was missing out on.

Comparing MySQL to PostgreSQL is a bit like comparing notepad.exe to emacs. While they both let you type in some text and save it to a file, that doesn’t quite tell the whole story. They are both RDBMSes that allow for the execution of queries via a client connection, but the philosophies, capabilities and sophistication of each are hardly even on the same plane of existence.

The way I think about it is that MySQL is a simple database program. Technically it does have pluggable engines; you can choose between a terrifically useless engine and one that at least knows what a transaction is. I consider Pg more of a “database framework”: an incredibly sophisticated and flexible program that provides a standard frontend interface, much as emacs ties everything together through “modes” and elisp, Pg has a C client library and speaks SQL.
To give you a more concrete example of what I mean, in Pg you can create “foreign data wrappers”, where the actual implementation of a query can be defined by a plugin. One example of a foreign wrapper is “MySQL” – you connect to Pg like normal through your client library and run a SQL query, but it’s actually executed on a remote MySQL server and presented as a table view in Pg. Or perhaps your query is auto-parallelized and distributed among many mongoDB nodes instead. No big deal.
Pg also has an astonishingly powerful set of extensions. My absolute favorite is PostGIS, a geospatial extension that adds geographic and geometric types, operators, indexed storage and much much more. Seeing a demo of it at the SFPUG meetups blew my mind, and I’ve been an avid user of it ever since.
Did I mention it also has a full-text search capability every bit as useful as Solr or ElasticSearch, with stemming and your choice of GIN or GiST indexes? My life is so much better now that I can simply use a trigger to keep my tsearch columns updated instead of application-level logic to keep Solr indexes in sync.
Pg is chock-full of awesomesauce that I’ve used to replace many moving parts of my application. I chucked out Solr and replaced it with PostGIS and tsearch. Also I ditched ZeroMQ (nothing against ZMQ – it’s a great project) and just use Pg’s built-in async message queue instead. Oh, you didn’t know it had a message queueing system? As a matter of fact it does, and I gave a talk on it for SFPUG at UC Berkeley, coincidentally a few feet from where the precursor to Postgres was first written. In my talk I showed how to construct a location-based storage engine using PostGIS and a single trigger, which would fire off an async notif of JSON-encoded lat/lng updates whenever a row was inserted or updated. Add in a WebSocket<->Pg client/server (such as the esteemed WSNotify) and you have a real-time event push system that notifies a JavaScript browser client whenever a location field on a row is changed, all without a single line of application code (other than the SQL trigger). Let’s see MySQL do that. (Slides and example code are here: https://github.com/revmischa/pgnotify-demos)

We all love fancy features and reducing the number of moving parts in our infrastructure, but I actually think the most compelling argument for Pg is that it is an community-driven project and is not owned by Oracle. MySQL is not an open project, and it is owned by Oracle. One does not need to look far for bone-chilling examples of Oracle’s management of open-source database projects. Let’s talk about BDB.
BDB, the Berkeley DataBase, was a nice little embedded database engine used by many pieces of software dating all the way back to 1994. Any of the maintainers of software using BDB might be in for a bit of a nasty shock should they decide to upgrade to the latest version, now that Oracle’s acquired the software and changed the terms of the license. Now they have two options which are helpfully explained on Oracle’s website.
You can choose from option A – the “open source” version, or you can choose option B – the “pay Oracle money” version. That webpage does seem to leave out one minor little detail though, the “open source” version is actually the Affero GPL. The AGPL sounds a lot like the GPL but with an important difference – not only does it require any programs you distribute to provide the source code, but also covers server software as well (N.B. I am totally not a lawyer and probably don’t know what I’m talking about).
Many projects sort of skirt around the requirements of sharing their code even though they use GPL software because they don’t actually distribute binaries, instead they just run a server and let it communicate to clients. The AGPL was designed to close that rather sizable loophole. What this means in practice is that the thousands of existing commercial products that use BDB, or use code that uses BDB, are all going to be prevented from upgrading to the latest BDB unless they make their product open source… or choose option B (cough up the dough).
You gotta give Larry props for that one. I respect him and am very pleased with the fact that at least someone in this world is willing to go full James Bond villain, complete with tropical island fortress. However I’ll stick with the community-run project for my database system.
Last point to make here: there is clearly a hard limit on how good MySQL is going to be able to get. If it was truly awesome and powerful, why would anyone need to buy Oracle DB?

Well at this point you might be saying “gee, that PostgreSQL option does sound pretty nifty, but how on Earth am I going to switch my existing application to Postgres?” Worry not, friend. There is a very handy DB dump conversion tool (mysql2pgsql) which does the job for you. I used it myself and had a bit of trouble with converting BLOBs and some ordering of foreign keys, but I was able to patch those problems up quite easily and get my changes upstream, so no big deal. I switched in 2012 or so and haven’t looked back since.

Finally, the PostgreSQL community is wonderful. I have always had a great time at the SF postgres user group and seen some amazing stuff people do with it that you would never imagine could be done with your database server if you’re stuck in the tiny MySQL world. Go check it out.

I watch this video every day: