Is Software Contracting For You?

It was in the last great recession that I started doing contract software development, about 2008-2010. The bubble didn’t burst with as much force and shrapnel as in 2000, but it had a distinct dampening of the animal spirits of SOMA, San Francisco where all the startups lived.

It was somewhat by design. The previous job I had, writing Java and ActionScript for a marketing research company was chill enough but felt aimless. I had so little motivation I ended up coasting for a few months and playing a lot of pingpong. I wanted some new challenges, so decided to go freelance.

The first gig I got was off of Craigslist, although I imagine these days Upwork would be a better place to look for work. The project was very limited in scope; mostly adding a Google Maps visualization on top of some grant data for a nonprofit. It was a couple weeks of work, a couple grand, and time to move on.

Some time later a former colleague of mine hired me for some development work at a SF startup doing IP telephony combined with podcasting. This was 2009 so nobody had ever heard of podcasts and smartphones were still a fresh new technology people weren’t entirely sure what to do with. The gig was a fun challenge – enabling people to listen to and produce podcasts using only a telephone, no apps involved. I ended up developing some pretty innovative software that resembled a web framework but for touch tone and interactive voice response (IVR) phone applications. We successfully moved the application from an expensive managed solution to our own in-house platform that I designed, significantly cutting down operating costs.

After some time working on that project, I met a couple of guys who wanted to build a SEO-optimized directory of medical professionals, starting with Spanish-speaking plastic surgeons. I said I could cobble together a search engine in my spare time and was engaged. Some days I would walk the two blocks over from the small telephony company office to the small office housing the nascent doctor directory business and show my progress.

It seemed clear at the time that the podcasting telephony company while highly experimental, was professionally run. It had not one but two Stanford business school co-CEOs running it, with what amounted to a successful track record in the form of an early dot-com electronic greeting card company. Remember those? Weird shit. There were respectable investors, a small team of smart and highly competent professionals, and a beautiful office on Howard St. I recall situated on the floor below our office sat a little room consisting of no furniture save a well-stocked bar with never any people in sight, and a sign on the door reading “GitHub.” I felt like great things could happen.

In contrast, my side gig seemed like some small-time SEO hustling along with a smaller team and paycheck and no Stanford business vibes or major VC funding. I didn’t see anything wrong with that, and still tried to do a professional job, but it seemed like more of a dead end compared to the “real” engineering I was doing, fighting battles with touchy open-source PBX software and voice recognition grammars.

Then things turned out completely differently from what I expected. The doctor directory project kept growing, expanding, and taking on a life of its own. We got a proper office at 1st and Mission, hired an engineer, a designer, a salesperson. Plastic surgeons were mostly ditched, and now we were primarily helping American dentists establish a presence on this new “world wide web” technology they couldn’t quite wrap their heads around. This contacting gig that I imagined would consist of a couple months of basic work kept growing and there was always more work to do. Without any planning or expectations it turned into a real company, with eventually a staff of 25 talented and terrific people and an extremely respectable office in the Financial District. We built software to help all kinds of small medical practices in the US manage their patient communication, from appointment reminders to e-visits to actually useful online medical Q&A. Six years later we sold the company to a large practice management software firm in Irvine, CA.

Following that experience my business partner John and I started a new company together again, this time on purpose. We started hiring and training some of the best young engineers, taking on projects and filling outstaffing needs for our clients, staffing a couple offices in Eastern Europe until covid forced us to go purely remote. This has in effect scaled up from my original single-person consulting operation into a powerhouse team of crack young engineers ready to take on complex software projects.

The classic Silicon Valley VC-backed, Stanford-connected, hip startup went nowhere, and closed its doors. I got permission to open-source the IVR framework we built but little else came of it. As it happened, consulting across different clients helped me to gain a broader picture of what was actually possible and break my preconceived notions founded on image instead of substance. I accidentally ended up starting a company which went on to help make American health care just a little tiny bit less terrible, created a couple dozen jobs, and had the profound and unique experience of building up the software for a company starting completely from nothing up through due diligence and acquisition. Along the way I learned some lessons about software contracting I want to share with others who may be considering going rōnin and setting out on their own as freelancers.

Tradeoffs and Considerations

It’s my nature to think of everything in terms of trade-offs. Maybe because I have engineer-brain, or because I’m a libra, who knows. There are real benefits to consulting as opposed to being a full-time employee, but also some downsides.

Legal Concerns

In America at least, contracting means forming your own business, doing 1099 tax forms and racking up deductions, and drafting and reviewing contracts. It’s more effort and responsibility than being a full-time employee somewhere, as you’re now responsible for taxes and legal matters. Even if you’re not in America, being able to work across borders means creating a legal business entity.

I started by initially getting a DBA, or “Doing Business As” name, under which I could legally create contracts and other paperwork using an official-looking business name. Later on I “upgraded” to a California S-Corporation, which gives favorable tax treatment once your income reaches a certain threshold along with some legal liability protection. If your corporation is sued, all that can be collected usually is what the corporation has, shielding you personally to some degree. A Cali S-Corp will run you $800 a year not counting the time spent on paperwork and taxes you or your accountant/tax attorney will be doing.

Even getting a DBA or Fictitious Business Name is far from simple. In Contra Costa County for example:

Within 30 days after a fictitious business name statement has been filed, the registrant shall cause it to be published in a newspaper of general circulation in the county where the fictitious business name statement was filed or, if there is no such newspaper in that county, in a newspaper of general circulation in an adjoining county. If the registrant does not have a place of business in this state, the notice shall be published in a newspaper of general circulation in Sacramento County. The publication must be once a week for four successive weeks and an affidavit of publication must be filed with the county clerk where the fictitious business name statement was filed within 30 days after the completion of the publication.
https://www.ccclerkrec.us/clerk/clerk/fictitious-business-name/

You also need to take out an ad in a local paper announcing the new business and inform your county.

You don’t need to do this yourself necessarily, services like BusinessRocket can take care of business formation and taxes for a small fee.

One of the first things you’ll need to do is ask your friends for legal services recommendations. If you know anyone who is a contractor or a small business owner they probably have lawyers they work with and can recommend. Or you can hit me up. I enlisted the services of a business attorney to help me draft and review contracts, and a tax attorney to take care of the corporation taxes and paperwork. Obviously lawyers are not cheap and in theory you can do all of this yourself, but they can also save you a great time of time and money as well by warning you about common pitfalls and fuckups, and suggest ways to better protect yourself or take advantage of favorable tax laws.

When you are going to agree to do work for a client they will want to know your hourly or project rate and a contract to sign. Understand that around a third to a half of your hourly rate is going to go to taxes, so adjust accordingly. A contract will need to have some important pieces of information. I highly suggest not listening to me and listening to an Actual Lawyer in your country or state about drafting a contract, but as far as software development you will typically need to include a “schedule of work.” This is the scope of what you will be expected to deliver in order to get paid.

Come at the schedule of work with a PM mindset – you have to figure out what the client actually wants and what actually needs to get built. If you end up needing to do more work outside of this scope then it can be a headache, and having a contract which spells out what you’ve been asked to do can make an effective backstop against scope creep. As a related matter, I highly suggest doing per-hour billing rather than doing projects for a fixed price whenever possible, because as we all know the best laid o’ work statement often go awry.

Technology

Sometimes you have a particular skill or framework or vertical or other some such specialization, and you can look for relevant gigs. When I started out long ago mine was Perl, and I didn’t have a hard time at all finding work. The first nonprofit project I worked on was already using my favorite Perl web framework. Some jobs will leave it entirely up to you to create something from nothing, and you can have your choice of technology for solving the problem. Sometimes you will go work for a company with their own existing codebase that wants to expand it, fix existing problems, or throw it away and rewrite from scratch.

For me this is one of the most thrilling parts of contracting; getting to see how different companies operate. I’ve gotten an opportunity to look at a good amount of different codebases and operational setups. It gives you a broader view of the landscape, allowing you to borrow best practices others have landed on, and learn from the mistakes of others. I’ve gotten to encounter a lot of technologies I would not have otherwise run into, like seeing different implementations of microservices, getting really familiar with SNMP, and deconstructing a J2EE application. When you work for one company or for yourself for a long time, it can be hard to stay current or get experience with other technologies. When working with different companies, you can rapidly take in and observe various stacks that organizations have coalesced around, usually ending up with good practices. There’s an infinite combination of frameworks, languages, architectures, libraries, development environments, and security practices and the state of the art is always in flux. Having exposure to new assemblies of technology keeps you curious and informed and better able to make decisions for your clients and projects.

Most of our business consists of either building new projects for people, taking over existing projects, or joining existing teams. We get to experience not just code of course, but see how different organizations are run, different business models, all kinds of personalities and team dynamics. It can potentially open you up to a richer tapestry of experiences and cultures than working on the same product and team for years and years will.

The technology you encounter will vary wildly, from hipster web microframeworks to ancient enterprise Java. Being flexible and able to rapidly adapt and figure out the basics of a lot of different technologies is a very valuable skill, as is learning how to start up every kind of bespoke development environment out there. I sort of have a weird perverted dream of going around rewriting ancient COBOL applications for desperate businesses to run on modern serverless cloud-first architecture.

You never know where freelancing will take you.

Clients

The coolest thing about being a contractor is that you can be your own boss. You set your own schedule, work from where you want, and don’t technically have to wear pants a lot of the time. Maybe this is less of a big deal than it used to be thanks to the ‘rona but flexibility is definitely something I value a lot.

Of course in the end, everyone has a boss, you can’t escape from it. The CEO has to answer to the board and investors, the investors have to answer to their partners, the partners have to answer to funds, and so on. Your boss is now the client, since they’re the one cutting your checks now.

In my experience this has been a good thing. I can report honestly that I’ve enjoyed working with 100% of my past and present clients and things have on the whole gone very smoothly. Much of it comes down to choosing your clients. You will turn some people down because they are looking for someone with different skills, don’t pay enough, have a Million Dollar App Idea I Just Need Someone To Build It, are unprofessional, or just not cut out for the whole business thing. Just don’t work for these people. It’s okay to turn down work. Always act professionally though, no matter what situation you find yourself in. Your reputation absolutely follows you around, and taking pride in your work and professionalism is a requirement for being a freelancer.

Another skill that you must be consciously aware of and always seeking to improve is communication. Being direct, open, and transparent with your clients can often mean the difference between a successful project and one that ends up in a mess of assumptions and bad feelings. Underpromise and overdeliver is the mantra. Over-communicate, bring any concerns about the project to the fore, have regularly scheduled progress meetings when applicable, and do demos for your client. Focus on delivering something visible, something the client can look at and play with, so you can get feedback early. There will almost always be some gray area between what your client has in their mind and what you envision in your own, not just in designs but in all the details in the details.

Sometimes a client may come to you with detailed designs and specifications, but I’ve never been in a situation where all the information needed to deliver a project was hashed out up front. Most of the time very little of it is. You need to establish a good two-way street of communication and always be asking for feedback and clarification of ambiguities. Get a MVP in their hands as early as possible and iterate on it.

Other Skills

Consider also what skills you can bring besides just writing code. Familiarity and expertise in UX and design is very valuable and basically a requirement for most software jobs these days. We’ve had clients come to us to help them perform due diligence on codebases of companies that they are considering acquiring. We’ve performed security audits on codebases, sometimes unbidden. And thanks to our extensive commercial experience building successful companies we also can provide valuable consulting services on marketing, sales and raising capital.

Whatever extra you can bring to the table for your clients be sure to market it and build up your experience and knowledge in that area. It can make the difference between being just another coder and a valuable partner to your client.

Working on three laptops at once can be a valuable skill.

Getting Started

For many contractor-curious folks getting started may be scary or daunting.

If you’re a full-time employee, becoming a contractor means giving up some job security. You may not have a guaranteed paycheck for a while, if ever. Being a contractor, especially if you’re starting out and doing it alone, brings many uncertainties. However working as a full-time employee carries its own sort of risks. Job security isn’t what it used to be, spending time on bureaucracy and pleasing your superiors may not make you fulfilled, and there’s a real limit to how much you can accomplish for yourself as part of a larger organization.

There is comfort in working at a company; all you have to do is show up and be told what to do, you don’t have to think much about taxes or contracts, someone else can make a lot of the decisions about how to run the company. But if you want to take on more responsibility, have the opportunity to grow and learn to run your own business, and do things your way then consider becoming a contractor. Almost everyone I know who works for a company of any size will be happy to tell you all about the mistakes and boneheaded ideas of their superiors. Everyone has ideas of how the company they work for could be run better. I say if you really believe this then work for yourself.

So how to begin? You have two options: line up to your first gig, or join a company doing freelance work. You can put the word out to your network that you are available for work, or you can go look for jobs posted on sites like Upwork. There are also companies that specialize in doing contract work and are often looking for contractors to augment their pool of developers. If you go this route know that such companies may not always have work immediately lined up for you, but they may be happy to interview you and keep you on file in case some work comes up that matches your qualifications.

My suggestion is to do both; look at what’s out there, get a feel for what people are looking for and how much they’re offering and also check out companies that are doing the kind of contracting work you’re interested in and offer your services to them. There certainly is no shortage of work and job opportunities out there for contractors, it’s more a matter of finding a good fit for you that will be engaging work and well-compensated. Even if you start with some small simple jobs, they can definitely lead to greater opportunities as you gain more confidence, experience, references, and a better understanding of the market.

I would of course be remiss as a small business owner if I did not mention that our consulting company JetBridge is always looking for smart and talented engineers. If you’re thinking of becoming a contractor, feel free to drop me a line, and I might be able to help you get started or refer you to other work out there. I know it can be an intimidating career jump, but it can be extremely rewarding and full of new opportunities as well. And if the current tech bubble happens to pop again someday, it just might be a great time to try something new.

Communication Tips for Engineers

Communication is a fundamental skill for engineers. No one builds anything on their own. Whether participating in an open-source project or being employed to crank out code, you need to work with others. The value of effective communication skills cannot be overstated.

Even Linus Torvalds, a curmudgeon who will never lack work and whose word is law in the Linux community, has acknowledged the need to be more measured in his criticisms and more generous with empathy.

Soft skills, in the parlance of our times, are in vogue. Employers and potential collaborators will judge you based on your ability to lucidly communicate your thoughts in an agreeable and succinct fashion.

There are many great guides and books written about how to communicate effectively with other human beings. A lot centers around having empathy. Having an understanding of where someone is coming from, considering the information that they have that you don’t or vice-versa, and being respectful are basic tenets. I offer a few suggestions here:

Use interrogatives instead of declarations

Even when you are pretty sure of a fact you want to communicate to someone else, it is often better phrased as a question rather than a statement of fact.

“Why did you write it this way?” is infinitely preferable to “this code is wrong.”

If discussing a solution or implementation with someone, ask them what they think first, regardless if you already have a plan in your head. They may suggest what you are already thinking, may have thought about things you haven’t, and may bring up other good ideas. Especially for junior people; it helps them engage their critical thinking skills instead of learning to rely on you to provide the answer. It gives people more ownership and desire to defend their idea because it’s theirs.

Don’t argue forever over things

This is one I am especially guilty of, and it has to do with knowing when to concede a technical argument. Very often there are no clear right answers for how to proceed with a feature or fix, and a healthy discussion of the tradeoffs can be illuminating and help arrive at a reasonable solution.

However since many tradeoff estimations involve a lot of guesswork and feelings and intuition, the “best” answer may never be agreed on. At some point you have to agree and move forward, and different people may have ideas of when that time has passed. I know I’ve driven at least one person crazy by continuing past the point they considered to be entering the domain of diminishing returns.

Often the proper solution isn’t clear. Agreeing on a proposal, prototyping, and gathering more data is more fruitful than making people exasperated or arguing for hours. Once you sit down to actually try it, it may quickly resolve the debate.

Try to not talk too much shit on other people’s code

“Oho!” said the pot to the kettle; “You are dirty and ugly and black! Sure no one would think you were metal, Except when you’re given a crack.”

“Not so! not so!” kettle said to the pot; “‘Tis your own dirty image you see; For I am so clean – without blemish or blot – That your blackness is mirrored in me.”

Source

If there’s one thing engineers love to do it’s complain about languages, libraries, tools, operating systems, service providers, interfaces, APIs, containerization systems, people on mailing lists, etc… When your tools don’t work right it can cause you hours of frustration and confusion, which nobody enjoys.

A moment’s contemplation will recollect the vast amount of sub-standard, buggy, hacked together balls of mud that the experienced engineer has thrown together in the past. If you write perfect defect-free code then you should continue with your expressions of distaste for the inferior engineers out there making fools of themselves. If on the other hand, you realize you have made plenty of blunders of your own, consider going easy on the target of your ire.

I’ve seen many online communities devolve into an ever-smaller group of grumpy guys, mostly chatting about how much everyone else is stupid and sucks. These communities sap your soul. Inject some positivity into your world when you can.

Communicate proactively

People aren’t mind readers and they generally are focused on their own work and problems. If you have information that may be useful to others, don’t wait for them to come to you and ask for it. If someone hasn’t asked you for an update or provided you with what you need, reach out rather than suffering in silence.

Over-communicating in a team is almost always better than under-communicating. By letting people know what you’re doing you can help others prioritize, not duplicate work, know whom to ask questions, inform you if they’re making changes that could affect your work, and reduce the need for people to make assumptions.

Don’t assume

Assumptions are frequently foolish and rarely right. Often a brief message can save you time, making sure you don’t start off down a fruitless path. And by asking, you are communicating information about what you are working on and the fact that the knowledge you seek is not as widely shared or accessible as the owner of it may assume.

Comment and document what you’re doing and why

Nothing makes working with code others have written go smoothly like having comments liberally sprinkled throughout the source code. Whether it’s a new hire or yourself in two years when you’ve forgotten why you need to check that condition in that insane query, a brief summary of your logic and thinking at the time can help impart understanding and reduce the need for assumptions.

Thanks for reading!

Org2Blog

This is my first post published using the emacs-to-wordpress tool Org2Blog. It’s pretty nifty.

Decentralizing Social Media

No, it has nothing to do with blockchain.

What kind of language should Facebook forbid? What kind of regulations should the U.S. government promogulate regarding whom Twitter can ban?

☞ Who cares?? ☜
Not me.

A depressing amount of energy and ink is wasted on these questions which shouldn’t even be issues in the first place. We don’t have to base our public discourse on platforms that corporations or even governments control.

The great news is that there does exist an alternative to the model of having all social media content go through a couple of companies. There is certainly no technical reason it should work that way, and there is a solution to the problem that has a foundation in technology, though there is naturally a social component as well.

What is this problem that needs a solution? I think it’s fantastically illustrated by all of these articles and experts and laws being passed to try to nudge Facebook, Youtube, Twitter to control what people are allowed to say and post. Busybodies, Concerned Citizens, corrupt politicians, think tanks, your parents, all want to petition these platforms to decide what you should be able to read or write. I view this as a problem, because I don’t think anyone should decide for me what information I should be able to share or consume. Not Mark Zuckerberg, not Donald Trump, not Jack Dorsey, not my congressional representatives, not the People’s Republic of China.

Government and private corporations in control of censorship are not the only problem here. As everyone knows these services are free, and as everyone also knows if the product is free then you are the product. Facebook and Google make almost all of their money from extracting and mining as much personal data about you as possible to sell to advertisers, PR agencies, and politicians. There is a better way.

Federated Social Media

The answer is federation. Decentralization. Distributed systems. You’re already familiar with the concept, just think of email. You don’t have an email username, you have an address. Your email address is a username on a host – mspiegelmock@gmail.com specifies the user mspiegelmock on the system gmail.com. I can write an email to someone else like rms@gnu.org, even if they don’t use gmail. I ask my provider gmail.com to send a message to the gnu.org host which is responsible for delivering it to the user rms.

No company “runs” email, yet all email servers know how to pass messages to each other. There is a vast array of different email hosts, providers, server applications and client apps. You can choose to sign up with a free provider like Google or Microsoft, your employer may provide you an account, or you can run your own server. You can use any sort of app you like with email, such as gmail.com, Apple Mail, Superhuman, Outlook, mutt, or emacs. In the earlier days of web-based mail there were a few options to choose from, like Yahoo and Hotmail, and eventually the company which provided the best user experience ended up grabbing a significant slice of email users, thanks to the wonders of competition. There were and are some issues with spam and malicious content to be sure, though a great amount of progress has been made on systems to combat it (spam and virus filters, real-time blacklists, DKIM/SPF). This is what federation looks like.

To grasp the concept behind federated social media, think of email. You sign up for an account with an instance (“host”) that you feel comfortable with, or run your own if you’re so inclined.

To follow someone you need their address, like @wooster@social.coop.

Think Twitter, but as an email address. The address denotes the username (@wooster) and the host the user is registered on (social.coop).

You have two timelines, in addition the people you follow. One timeline is the “local” timeline, which is everyone else on your instance. If you join an instance of people that share a particular hobby, language, interest, region or philosophy you get to start out with a feed that may have posts that may be relevant for you. Your instance can link, or “federate”, with other similar instances, connecting users on your instance with users on the other instances.

Moderation

Why not run a whitehouse.gov instance instead?

Just because there is no central authority for content moderation doesn’t mean that the system is full of abuse and Those Sorts Of People you would like to avoid. These things exist to be sure, as they do on any platform, but they are confined to their own instances. Moderation does exist, but unlike Facebook or Twitter you can choose your moderators. Most instances have policies about what external content they block, what types of instances they want to federate with, and what kinds of content they permit. If you disagree with their policies, you are free to join an instance that fits with your preferences, or start your own.

There are plenty of people I don’t want to hear from, there are plenty of posts our there that would decrease my quality of life, and I’m fine with outsourcing some moderation. I just don’t want this guy to be the final arbiter of all information.

Propaganda, trolls, abuse, and misinformation exist on every platform. You can find it on YouTube, LiveJournal, TikTok, Twitter, and no doubt on federated social media. Media literacy is an important skill that should be taught to help media consumers understand biases and distortions inherent in all media. A platform that helpfully provides fact-checking would be desirable to many users. But the fact remains that you cannot outsource critical thinking. There’s no getting around this.

The problem with top-down centralized structures that take it upon themselves to decide what information can or can not be spread should be plain. Corporations and politicians have bad incentives and the temptation to misuse such power to cover up misdeeds is too powerful for most to resist. We all know now what happened with Chernobyl and the misery caused by the suppression of information. Maybe if we had some supremely enlightened and benevolent information despot it would be okay to put them in charge, but I can’t really think of anyone I want to grant that authority over me.

The instance I belong to is social.coop, a social media cooperative. It’s a group of people who donate a small amount of money to pay for a server to host a Mastodon instance and volunteers who help maintain and administer it. There is an online forum for discussions and consensus-based decision-making, and a lot of smart people on it. This is just one example of the kind of self-organization that is possible in the fediverse.

Mastodon

Today the most popular software for plugging into the federated social media network is Mastodon. It’s free (AGPL) and open source naturally, and there are a number of apps you can use with it, including some slick paid apps. The Mastodon web interface looks something like this:

And the “Toot” app looks like this:

Many communities run Mastodon instances, some are public, some are cooperatives, some are private. You don’t have to use Mastodon to talk to people using Mastodon, if you use software that speaks ActivityPub then you can follow, share, post, like, comment, and communicate with anyone else in the fediverse. Just like if you use any email software, you can email anyone else using email software, which I think is pretty neat.

Mastodon is federated social media, federated social media is not Mastodon.

Technical Details (ActivityPub)

In the early days there were a number of attempts at creating social networking protocols with really obnoxious names like “pubsubhubbub.” After a bit of experimentation an official standard was published by the World Wide Web Consortium (W3C) in 2018, going by the name ActivityPub. You may know the W3C from their earlier hits like HTML, CSS, XML, and SVG.

As it’s an extremely recent standard, we’re still in the very early days of implementations. I expect there will be a number of libraries and clients popping up, along with not just standalone servers but server capabilities integrated into existing platforms and sites. Any site that lets you log in and post content could be modified to plug into the fediverse by implementing the relatively simple, JSON-based protocol. Your existing accounts could turn into ActivityPub Actor objects. New social networks can in effect bootstrap themselves by leveraging existing users, software, and federation networks, and we’ll see companies and open source projects compete to offer the best user experience.

Honestly the best way to learn about ActivityPub is to go read the standard doc. It’s in plain English with friendly cartoons and very straightforward.

Actor with messages flowing from rest of world to inbox and from outbox to rest of world

The very brief gist of it is that there is a client ↔︎ server protocol and a server ↔︎ sever protocol, much like IRC. You can read notifications that arrive in your inbox and you can publish messages to the world via an outbox. All content and objects on the platform are simple JSON documents that live at a URL (an IRI to be precise). Technically you could make a compliant ActivityPub server with a static webserver (I think? Tell me if I’m wrong).

Social media isn’t just text and image posts. The site PeerTube is a decentralized version of YouTube:

PeerTube is a free and open-source, decentralized, federated video platform powered by ActivityPub and WebTorrent, that uses peer-to-peer technology to reduce load on individual servers when viewing videos.

User Base

The network effect is what makes a social network attractive. The more people on it, the more useful it is. Having celebrities is a big draw for many people. It’s the biggest challenge that any challenger to the status quo faces.

I believe there is nothing permanent about Facebook or Twitter. I remember when it was hard to imagine anything replacing MySpace, or when everyone on the Russian-speaking internet had a LiveJournal. Fads change and great masses of people move smoothly to new platforms with ever increasing rapidity.

https://the-federation.info/ – Federation Stats

The thing that is so powerful about a social media protocol is that it seems, at least to me, like the logical conclusion of social media. Once a growing critical mass of people move to it, either because they are sick of Facebook’s shit or some cool new company’s platform happens to be powered by ActivityPub, most of the innovation around communities, software, moderation, new forms of media, organization and technology can happen within the fediverse. Because of the open and extensible nature of the underlying foundation anyone can plug in a conformant piece of software and shape their piece of it how they want while still interoperating with the wider world. It’s a perfect vehicle in which one can imagine fictional visions of the future internet taking shape, like the Metaverse from Snowcrash or the VR Net from Otherland.

https://pawoo.net – Japanese Mastodon with 625,000 users and 45 million posts

The internet is a decentralized collection of autonomous systems held together by communities defining standards and protocols. Distributed social media maps very nicely onto the architecture of the internet and promotes freedom of expression and experimentation with new forms of media, social organization, and technology. Doubtlessly it will create new problems as well, like intensifying the internet hypernormalization effect. It will likely be some time before norms and robust community structures are figured out at scale. The ability for anyone to participate in the wider social community within bounds and parameters they set for themselves will be messier but ultimately more powerful and driven by more positive incentives than the current social media monopolists.

Links

If you want to give Mastodon a spin you can check out joinmastodon.org.

To learn more about ActivityPub read the spec (really, it’s not dry). More info about the types of Actors, Activities and Objects on ActivityPub can be found here.

PeerTube homepage explaining federation of video content.

Why does decentralization matter? Mastodon op-ed.

The-federation.info – statistics and open source fediverse projects.

Excellent ActivityPub Conf Talk on this topic, from September 2020

Will America Ever Be A Socialist Country?

“America will never be a socialist country!” some exclaim these days. For the first time in a hundred years a viable presidential candidate proudly boasts of their democratic socialistic ideals and is welcomed by many. In fact there is already a great deal of wealth redistribution and government aid programs in America, though much of it is directed to the most well-off and well-connected.

Where does socialism fit in America?

In the mainstream the fact that America is capitalist is axiomatic. America is by definition capitalist and therefore anything done by its government or people is capitalistic. This is a useful piece of propaganda both for its boosters and haters in exactly the same way calling the USSR “socialist” was.

The USSR, or Union of Soviet Socialist Republics (worker’s council) was axiomatically socialist, indicative by the name, as is Democratic Republic of North Korea. The ideal of socialism, the scientifically determined natural end state of economic development, was always just around the corner for the Soviets. Domestically it was a positive term and something the government worked hard to convince its citizens it was the system they were living under, or about to be living under. For America it was a great way to tarnish the name of socialism by pointing out how cruel and impoverished their conditions were. Calling the USSR “socialist” was instrumental in propaganda both internally and externally and so it was encouraged to become the unchallenged categorization of the economic system in the Union of Soviets, even though by any objective measure it was in no way socialist. It was a centrally controlled oligarchy with almost no discretion or devolution of government given to local workers and those in the Party or at the center got to accumulate wealth and power. Common workers had little voice in how the means of production or the fruits of their labors should be democratically distributed, and those that had any such ideas in public could face grave consequences.

America is not the USSR by any stretch, however this propaganda exercise of always explicitly describing it as capitalistic and definitely not socialistic is one aspect that it shares in common with the Soviets. It is a market economy, sort of, sometimes. It does certainly seem to have Capitalists, the sort who in earlier eras would probably sport monocles and top hats instead of Teslas and favorable capital gains tax rates. It’s my opinion however that the US federal government does engage in tremendous wealth redistribution efforts and socialized risk and capital disbursements that would have made a Soviet or Maoist central planner feel uneasy.

Image result for america will never be a socialist

When American politicians or their media mouthpieces decry socialism, it’s a worthwhile exercise to ask for whom they mean. In the American context, the technical term “socialism” means “government giving money to the underserving,” i.e. the poors. More than almost any other country, the question of whether someone deserves government handouts is a central concern when considering what our hard-earned tax dollars should be spent on. In many parts of the world, the poor, disabled, young, elderly, disadvantaged and so on are considered deserving by default and the obscenely wealthy and powerful maybe not as much. Somehow, and I don’t claim to know exactly how (although I have some ideas), these determinants have been flipped in Capitalist America.

Again speaking in the media argot, “Socialism” in America is considered Bad because the money goes to the undeserving, like poor children. In our “market economy,” unimaginable quantities of money from taxpayers and future generations is hoovered up by the government and spat out at the inconceivably wealthy and well-connected. Even a moment’s reflection can easily illuminate this but you won’t find any mention of it as socialist in the press.

To give one current example of many, there is a bill far along in the U.S. Senate to modernize America’s nuclear arsenal. This is estimated to cost around $1 trillion over ten years, which likely means $3-4 trillion given the usual Pentagon overruns. Nowhere in the media will you find a discussion of the costs vs. benefits of the proposed spending. Citizens will not find themselves pondering the “how will we pay for it?” question, never mind asking whether America really needs all 4,000 nuclear warheads in the first place. This money will of course go directly into the pockets of the wealthiest corporations and individuals to be spent on gold-plated hardware that hopefully will never be used. When a politician makes the suggestion that we should have some money spent on food stamps or health care suddenly everyone’s breaking out the eye shades and Excel to carefully examine these socialist policies.

“Where will the money come from for the F-35 program?” has never been posed in a debate. Rarely is this wealth redistribution upwards considered socialist, much less challenged for its basic assumptions about needs. But when someone needs to eat food, they get a vastly more stringent auditing of their finances than the president does.

At the same time a bill is passed to dole out $867 billion to (mostly) giant farm conglomerates to farm, or not, the Trump administration takes action to increase work requirements for food stamps because we don’t want socialism. The people who want to eat might not be deserving, so why give them handouts? However while making food stamps less funded and harder to access, no such limitations or work requirements were imposed on the colossal agribusinesses receiving more free government money.

The now $1 trillion budget deficit under Trump is somehow growing, all while programs for basic needs are being cut. A decent new part of the deficit is the giant tax breaks given to the wealthiest Americans and even greater military spending. The tax cuts prompted some of the largest stock buybacks in history, helpfully boosting stock prices but likely damaging everyone else in the process.

However none of these issues are new or unique to the Trump administration. Federal intervention in the marketplace and socialization of risk was of comparable scope under Obama. Thanks to state aid, pressure, and financing, entities such as AIG, Caterpillar, foreign-owned banks, and General Motors were explicitly and directly supported by the state, not to mention the $800 billion bailout of some of his biggest campaign funders, namely Wall Street.

Whether or not this was good policy is subjective, but we can’t just call it pure capitalism when the Federal government owns shares in carmakers. But when China engages in state aid of its industrial champions (like nearly every modern rich economy did at some point) we threaten a trade war and incapacitate the WTO. Again to be clear, China’s interventions are on a much grander and explicit scale and the U.S. economy isn’t equivalent to China’s. Part of what I’m saying is maybe we shouldn’t think in such black and white terms of either/or Socialist or Capitalist, and most people dunking on China for unfair trade practices conveniently forget about similar U.S. practices. Except the Libertarians who are consistent at least.

Policies of China enacted on a grand scale are instructive by nature. Under Mao’s idiotic backyard blast furnace program and state-directed agriculture tens of millions of people died unnecessarily of hunger in one of the worst atrocities in history. After Deng Xiaoping introduced some market-based reforms, more people than ever before in history were lifted out of poverty in an incredibly short time. Hopefully we can learn something from this experience. Markets should be the default solution for economic organization because the vast complexity of interactions and incentives of billions of people is impossible for humans to comprehend or organize centrally, but this principle can be abused for propaganda purposes and intrude into realms where it is inappropriate. Most importantly, markets are not the solution to everything. Some problems do not have a market-based solution. Questions of basic education, law and justice, health care, and public infrastructure do not have market solutions.

In the most recent issue of the Economist, Bernie Sanders is called out for having a foolish plan to erase all student debt and the more “reasonable” plans of candidates Joe Biden and Mayo Pete are lauded for their more measured approach. Again, how could it possibly be paid for? Where would the money come from? Education isn’t free. The U.S. federal government has backed loans from private educational institutions that fail to teach while incurring sometimes hundreds of thousands of dollars in debt with atypically high drop-out rates. Once again, socialism is directed towards for-profit universities, fully subsidized and encouraged by the government. Again socialism for the rich, but the students get loaded with stupefying amounts of debt before they even begin working, putting a massive damper on earnings potential.

“This country has socialism for the rich, and rugged individualism for the poor.”
Martin Luther King, 1968

It doesn’t have to be like this. In Ukraine or Poland, a student can attend a five-year university for free. Those universities may not top world rankings but imagine what they could do if they had an economy the size of America’s. If people can get a master’s degree in a technical program for free in these countries, what’s stopping it from happening in America? As another comparison, when I had temporary residence in the Czech Republic, another poor post-soviet country, I was able to get fully covered health care for $100 per month. That covered absolutely any health-related procedure or visit that would ever need to be performed, and probably only cost that much because I was a foreigner. The quality of care may not be as good as in America but it’s a vastly poorer country. Why can’t we have these things in America? In any other country we would apply the term “corruption”, but in America we say it’s a “lack of political will.”

Sure America spends vast sums of money on health insurance and education, but the money doesn’t actually reach patients and students. It goes to vast corrupt institutions of well-connected plutocrats and middlemen who spend the money on bloated administrations and unnecessary layers of bureaucracy. Libertarian institutes like Cato argue that poor people have never had it better than today, because if you actually add in transfers, we spent tons of money on them! Well that is not factually incorrect as we do spend vast amounts of money on health insurance and educational loans and SNAP and all the rest, but the money isn’t benefiting the supposed recipients. A lot of it is eaten up by middlemen like health insurance companies and for-profit university shareholders. Again the money is funnelled to the donor classes in what we would call corruption in any other country, and the people on the bottom have to fight against constant introductions of means-testing, cuts and more and more soul-crushing bureaucracies.

Consider how much money is wasted on health insurance. Not health care, but the insurance part of it. Even small practices have specialist employees who deal with billing insurance companies, faxing paperwork and arguing over reimbursements. Students can now major in billing health insurance companies (it’s complicated), a sign that maybe resources aren’t being spent in the best way here. The Mercatus Center, a Koch-backed libertarian outfit that is absolutely not supportive of Medicare For All released a report fretting about the price tag for the program. According to the report, government expenditures would be increased to pay all health-care expenses increasing the share of spending by the federal government, but overall expenditures for human beings would decrease, by about $2 trillion over ten years. Again the scale of graft and waste in the existing system is staggering, and I think this savings estimate is on the low side, considering the source of the research and support. This is of course just focusing on the economics and ignoring the large social costs and moral questions.

This isn’t to say that fiscal responsibility should be abandoned. We should have conversations about how to fund programs we as a society deem important. We should be cautious about borrowing against future generations to pay for our expenses today and we ought to be sure that our government is spending wisely with the money it does have. I believe that there are purely economic arguments to be made in favor of single-payer health care and universal free education, not to mention moral arguments, but that isn’t really the point I’m trying to drive home here. The point is that the media and politicians in this country have an endless supply of concern for fiscal rectitude when it comes to supplying Americans with basic amenities like food, shelter, and medical care but this intense scrutiny suddenly evaporates when we’re looking at splashing out trillions of dollars for tax cuts for wealthy individuals and corporations, or military hardware, or any other of the innumerable transfers of money from the taxpayers and future generations to elites.

Absolute cretins would have you believe that market forces would be a more efficient way of distributing health care, contrary to the experience in most parts of the developed world. The concept that people are informed consumers when it comes to health care is laughable. You don’t comparison shop MRIs, you aren’t going to check the reviews of hospitals while you’re having a heart attack, nobody makes a choice to get old or be born with health issues. Most people aren’t qualified to make rational choices about comparative cancer treatment costs and don’t want to spend time figuring out how to work in their overpriced insulin into their budgets. Only someone with a chronic case of Nudge-brain could possibly think healthcare should be treated as a marketplace. Even one of the most laissez-faire capitalistic governments on the planet, Hong Kong, has free at point of service universal hospital care and one of the highest life expectancy rates at 84 years versus the USA’s 78 years and dropping. And expanding insurance to be universal doesn’t make any sort of logical sense. As I wrote before, what’s the point of insurance if it’s subsidized by the government, you aren’t allowed to price risk (i.e. pre-existing conditions), and it’s universal? Why have the insurance layer there in the first place then? Corruption, or socialism for health insurance and pharmaceutical companies.

America is certainly fairly capitalist, but also fairly socialist in a big way already. Vast, unimaginable quantities of cash are borrowed and taxed and funnelled to the ultra-wealthy donor and plutocrat classes. If we’re going to ask hard questions about budgets and costs, maybe we should start with looking at the welfare programs for the rich instead of basic health care or education.

Python 2020: Modern Best Practices

Python and related tooling continues to progress and evolve. I’d like to share some of the tools and practices we’re using at JetBridge to develop python web applications.

This is by no means an exhaustive account or a definite list of all best practices, and I hope readers will share what’s working well for them so I can learn and incorporate that knowledge. I don’t know about everything out there but I can at least present a survey of what we’ve been using on multiple projects with success.

Python

Let’s start with… python. As of January 1st, 2020 python 2 support was officially discontinued. If you are still maintaining any python 2 code you are using the language equivalent of Windows XP. Not only is python 2 no longer receiving security updates but now all python module authors will feel comfortable dropping any support for python 2 in any future versions of their modules, which means your dependencies are unlikely to receive security updates as well. Using python 2 is now a legitimate security risk.

Python 3.8 is out. What’s new in it?

The “walrus operator” := allows you to initialize a variable as part of any expression and save a line or two of code. The battle in PEP 572 over getting this operator included in the language was so unpleasant that it caused Guido van Rossum to ragequit his Benevolent Dictator For Life of Python role.

__pycache__ directories are now managed out-of-tree so they stop polluting your deployments and source control.

New additions to python’s type system – TypedDict lets you define the shape of a dictionary type, Literal lets you easily construct literal value constraints such as for enumerated value options, and at long last we have built-in support for structural subtyping, also known as Protocols.

F-string debug syntax – now instead of writing:

print(f"blorp={blorp}")

You can write:

print(f"{blorp=}")

Which is terrific news for those of us who will continue using print statements to debug until the day we die.

Python 3.9 is expected out in October 2020.

Linting and Formatting

Keeping your code neat and formatted can really help with readability and enforcing a consistent style. The tooling can also help catch potential bugs or mistakes. Here’s what we’re using:

Flake8 – Classic Linting Tool

Run as a pre-commit hook or in your CI flow. We suggest installing and enabling the plugins:

tox.ini configuration:

[flake8]
ignore = E305,E402,E501,I101,I100,I201
max-line-length = 160
exclude = .git,__pycache__,build,dist,.serverless,node_modules,migrations,.venv,.bento
enable-extensions = pep8-naming,flake8-debugger,flake8-docstrings

Mypy – Type Checking

Mypy performs the useful function of type-checking, to the extent one can in python. It does on some occasions catch useful errors for you and is improving as time goes on. Still, the usefulness of python’s bolted-on type system afterthought is limited compared to say, any other typed language.

If you are adding it to an existing project with many dependencies you may need to add ignore_missing_imports = True to your mypy.ini configuration file until you can resolve all of the warnings you’re going to get.

Bento – Static Analysis

Bento is a very new tool that attempts to be sort of a meta-linter, combining a number of different checker tools into one, most notably Bandit, a “Security oriented static analyser for python code.” It’s designed to integrate into git hooks and CI workflows relatively easily. It’s still quite new and not super mature yet but this is definitely a tool to keep your eye on. The analysis engines are open source and provided for free, though the company behind it is working to offer paid features for larger teams.

Black – Formatting

Black is a brutal and fantastic code formatter, much like prettier for python. It can be run as a pre-commit hook to make sure your code is formatted correctly, or you can have your editor run it automatically on save (my preference). It is technically possible to modify the formatting rules but there is no reason you should ever do that. Just enable it, always run it on every changed file, and never worry about 97% of code formatting issues ever again.

Workflow Integration

People are of differing opinions on whether you should add these tools into your editor, git hooks, or CI pipeline. Personally I have all of these tools hooked into my editor (mostly spacemacs but giving PyCharm a try) and love having my code formatted upon saving and seeing type errors inline in my code. This is definitely the best way to develop but it doesn’t enforce any standards in your team. Maybe you can always expect the people working on your project to have their editors configured correctly but this is mostly unrealistic for most teams.

You can add it as a pre-commit (or pre-push) git hook, which ensures everything is run before it goes to CI. The downside is this can add extra setup steps for the project or greatly increased execution time for common git commands.

Another option is to run all of your checks in CI and let developers be responsible for committing code that is correct or suffer failed tests. I have CircleCI configured to install dependencies and then run the checks as separate jobs in parallel.

And these options are not mutually exclusive. You can totally do all three together.

Testing

Switching away from unittest.TestCase and lots of custom helper functions to create objects in favor of pytest fixtures and factoryboy made testing vastly more pleasant, especially when writing tests that talk to the database.

Our setup for writing tests that interact with Flask and SQLAlchemy is to set up fixtures with factoryboy which helps you declaratively write fixture factories for all your database models and pytest-factoryboy which lets you register your factories as pytest fixtures. The plugin pytest-postgresql allows easy creation of a PostgreSQL database for running tests and pytest-flask-sqlalchemy patches in a mocked database session (or sessionmaker or engine if you need them) during tests that ensures each test runs in a subtransaction. Subtransactions (aka SAVEPOINT) allow you to run each test isolated in its own transaction and all changes are rolled back at the end of the test. This allows each test to be invisible to any other test or transaction and also to have all database changes cleaned up automatically. This is the most efficient way to run database tests with a high degree of reproducibility to how your application will be running for real.

There are a lot of pieces here but they fit together beautifully in the end. Your test setup may look something like this:

myapp/db/fixture.py – where we like to define database factories. These can be used for populating development environments and tests with sample DB rows.

from faker import Factory as FakerFactory
import factory
from jetkit.db import Session  # see https://github.com/jetbridge/jetkit-flask/blob/e3fc3448933ffbfb573cc1dfc873364cd17d4aca/jetkit/db/__init__.py#L10

faker: FakerFactory = FakerFactory.create()

class SQLAFactory(factory.alchemy.SQLAlchemyModelFactory):
    """Use a scoped session when creating factory models."""

    class Meta:
        abstract = True
        # by providing access to our current sqlalchemy session the factory can automatically 
        # add newly-created objects to the session (i.e. insert into the DB)
        sqlalchemy_session = Session


class UserFactoryFactory(SQLAFactory):
    """Base class for user factories with common fields."""
    class Meta:
        abstract = True

    dob = factory.LazyAttribute(lambda x: faker.simple_profile()["birthdate"])
    name = factory.LazyAttribute(lambda x: faker.name())
    password = 'my-default-pw!'
    avatar_url = factory.LazyAttribute(
        lambda x: f"https://placem.at/people?w=200&txt=0&random={random.randint(1, 100000)}"
    )


class NormalUserFactory(UserFactoryFactory):
    """Create a user with type=Normal."""
    class Meta:
        model = NormalUser

    email = factory.Sequence(lambda n: f"normaluser.{n}@example.com")

This sets us up with a factory that can produce NormalUser objects. In our setup we use SQLAlchemy polymorphism to distinguish between different user types with different model classes and the UserFactoryFactory (how very enterprise) gives us a base class to quickly define factories for each type of user model.

myapp/test/conftest.py – place to add fixtures made available to your tests. Documentation on these fixtures is provided here.

from myapp.db.fixtures import NormalUserFactory
from pytest_factoryboy import register

register(NormalUserFactory)

This register helper function takes our factory and creates two pytest fixtures out of it. One fixture will be called normal_user which will always return a user object in our DB session, created on demand once per test. The other fixture will be normal_user_factory which will accept arguments to override the factory defaults.

Next we set up fixtures for database, app, and our DB session:

@pytest.fixture(scope="session")
def database(request):
    """Create a Postgres database for the tests, and drop it when the tests are done."""
    with DatabaseJanitor(DB_USER, DB_HOST, DB_PORT, DB_NAME, DB_VERSION):
        yield

This provides a new database for the entire test session – it’s only created once and dropped when everything is finished.

@pytest.fixture(scope="session")
def app(database):
    """Create a Flask app context for tests."""
    # here we pass in config overrides to our create_app
    app = create_app(config=dict(SQLALCHEMY_DATABASE_URI=DB_CONN, TESTING=True))

    with app.app_context():
        yield app

The above code provides us with a Flask app and context for the duration of the entire test session. You can push a new context for each test if you like (remove the scope fixture argument) but I’ve never needed to do this.

@pytest.fixture(scope="session")
def _db(app):
    """Provide the transactional fixtures with access to the database via a Flask-SQLAlchemy database connection."""
    from myapp.db import db
    db.create_all()
    return db

This is the magic hook to provide our database session to pytest-flask-sqlalchemy. We need to provide the package of our SQLAlchemy instance to our pytest configuration in tox.ini:

[pytest]
# mock sqlalchemy database session during testing
mocked-sessions = myapp.db.db.session

Now we can define a fixture for a HTTP client to talk to our app:

@pytest.fixture
def client(app, normal_user):
    # get flask test client
    client = app.test_client()

    access_token = create_access_token(identity=normal_user)

    # set environ http header to authenticate user
    client.environ_base["HTTP_AUTHORIZATION"] = f"Bearer {access_token}"

    return client

This fixture has a dependency on two other fixtures; app and normal_user. We defined the app fixture just above, and the normal_user fixture is automatically added for us by the pytest_factoryboy register helper.

So now that we have a client fixture and a normal_user fixture, we can write very straightforward tests for API calls. Suppose we want to test a user API:

 def test_user_api(client, normal_user):
    response = client.get("/api/user/0")
    assert response.status_code == 404

    user_response = client.get(f"/api/user/{normal_user.id}")
    assert user_response.status_code == 200
    assert user_response.json.get("id") == normal_user.id

The simplicity and compactness of this test is striking. We don’t have any test cases, we define our dependencies in the function arguments, we use straightforward assert statements to check our responses. The test runs in an isolated subtransaction, dependency injection is performed to load the complete dependencies for this particular test, and it couldn’t possibly be any cleaner.

If you’re curious why we’re doing a simple assert here and not something like self.assertEqual() the answer is that pytest overrides the built in assert function with a more test-friendly and powerful version. You will still receive output exactly as you would expect from any test framework if the assertion fails. See the pytest documentation for more details.

Virtual Environments ﹠ Dependencies

The most modern tool for managing dependencies and virtual environments is Pipenv. It’s a bit more npm-style than venv or virtualenvwrapper, with a lockfile, split dev dependencies, and environment management via command line instead of sourcing anything in your shell. It saves the virtual environment files away out of tree.

The downsides for Pipenv are that it is frankly super slow and there hasn’t been an official release in over a year despite very active development. I hope that a faster new release will come out sometime soon.

Pipfiles are the future, no reason to be using requirements.txt anymore.

One more feature that may be of interest to some is the ability to define multiple sources in a Pipfile. If you have certain dependencies that need to be pulled from an internal package index server for example, you can define that source for only those dependencies instead of having to globally change your pypi mirror.

Web Framework

Some of the popular modern web frameworks are Django, Flask, and Falcon.

Django

Django is a pretty heavy solution but has the benefit of everything being set up for you. It’s not a tool I reach for because I normally only try to create lightweight API servers, with little to no server-side rendering of HTML, and I don’t find Django as suited to a serverless architecture as something more lightweight.

Flask

Flask has been our go-to tool for years. It gives you a basic core into which you can plug in components and features as needed. The setup involved in creating the perfect enterprise-ready Flask app from scratch is considerable and takes some experience to get right on your own. The flexibility and ability to craft an application perfectly suited to your needs is invaluable for serious projects, and the simplicity and whipupitude makes it perfect for dead-simple services too.

I’ve written at length about writing serverless web applications with Flask:

Falcon

If Flask is too heavy for you, there’s the Falcon microframework. If you’re writing a web service for a system with 64k of RAM and it’s not talking to any database or external services and the CPU overhead of handling HTTP requests and responses is the main bottleneck, Falcon may be a good choice. Their documentation really emphasizes how fast it is. I don’t think your web framework is usually the primary concern when it comes to speed but doubtless there are situations where this is needed.

Digression: Request Globals

<Digression>

There is one funky aspect of how Flask provides access to the current “app” context and the current request context that bothers or confuses some people. There exists an instance of your web application that contains configuration, routes, error handlers, and extensions that comprise your app. When your app is started up a new “app context” is pushed onto the app context stack to keep track of what app is currently active:

from myapp import app
with app.app_context():
    do_stuff_with_my_app()

In any code running inside of this context, you can access the current application.

from flask import current_app
def do_stuff_with_my_app():
    print(current_app.config['SOME_KEY'])

What’s important here is that current_app is a context variable proxy, which you can treat like a global variable but actually belongs to a context stack and is thread safe. Typically you only need to deal directly with pushing an app context if you’re writing scripts or wrappers that utilize your Flask app instance.

A similar approach is used for the current request context. When your Flask app is running (inside an app context) and a new request comes in, a new request context is pushed onto the request context stack to keep track of the request and request-local variables.

So whereas in many web frameworks like node’s Express you get passed in request and response objects as part of your handler:

app.post('/', function(request, response) {
  console.log(request.body);
  response.send(request.body);
})

Or in python’s Falcon:

import json
import falcon
class Resource(object):
    def on_post(self, req, resp):
        body = json.load(req.stream)
        print(body)
        resp.body = body
        resp.status = falcon.HTTP_200

In Flask one might write:

from flask import request
@app.route("/", methods=["POST"])
def app_index():
    body = request.get_json()
    print(body)
    return body

Again, request looks somewhat like a global variable but in reality it is a proxy object to a thread-local object on a context stack. The request is pushed automatically for you by Flask when the request comes in, so you mostly don’t have to know or care about manipulating this stack, unless you are writing some of the more exotic kinds of test cases.

This global-seeming access to context may feel dirty to some, likely conditioned by a healthy aversion to global variables or “god-objects” because of thread safety issues, poor code organization, and the inability to grapple with multiple instances of such objects simultaneously in the same program. These are valid concerns that the LocalProxy objects and context stacks effectively mitigate, while still providing a simple and convenient method to access the instances as needed from anywhere in your codebase, with the only caveat that you are responsible for pushing an app context if you are doing something outside the normal request flow.

I confess that the appeal of this approach was not obvious to me until I tried building a Flask app that talked to a database without using the Flask-SQLAlchemy extension. This extension integrates SQLAlchemy (an ORM) sessions with the Flask contexts so you can always easily access a database session that is local to the current request and transaction, or linked to your app context if not inside a request.

The real value of these context variables comes when you try to modularize your code and database routines. One problem that this solves is when you have a database transaction started inside a request, and then you call into some other code which may call other code which performs queries that should be inside the same transaction, as in a typical atomic operation that a RESTful endpoint might do. Somewhere you must retain a database handle to this operation, and expecting it to be passed through every function that might conceivably call another function that might perform a query is not feasible or clean. Being able to simply import a database session object that is automatically scoped to the finest level of application work you are performing (i.e. to the current request, or not) and assume it belongs to the current database transaction is a truly simple and elegant solution.

This approach has been recognized as a useful tool and in fact in python 3.7 gained first-class support in the form of contextvars from PEP 567. Opinions certainly may differ on the purity and magical-ness of this mechanism but I consider the simplicity and accessibility it affords to be the stronger argument. And given that it is now enshrined in python core means it is unlikely to go away anytime soon.

</Digression>

Putting Into Practice

If some of these ideas sound just splendid to you and you want to try them out, by all means give them a spin. If you’re looking to incrementally adopt new tools and features to your codebase implementing each of these suggestions independently should be manageable. However if you’re starting a new project or want to maximally embrace JetBridge style, it’s a daunting task to configure and wire up all of these practices into a well-organized and clean template. Honestly, setting up the database tests and Flask extensions is tedious. I’m lazy and don’t feel like doing it for new projects. That’s why we’ve created an open-source app starter kit and utility library for rapidly building modern, enterprise-ready python web applications with all of these practices and many more baked in and ready to go. Sort of a Create-React-App (we have one of those too) for our very opinionated python web service setup where we can put these recommendations into practice and save ourselves time setting up each new service.

sls-flask

Our starter kit is called sls-flask. It generates a Flask app skeleton with pytest fixtures, RESTful APIs and serialization, database factories, linting, authentication and more in a serverless-first package. It utilizes our handy JetKit-Flask python library that provides common database utilities (soft delete, upsert, UUIDs), S3 asset support, starting points for authentication and user access and other bits of functionality we’ve found useful in many projects.

Updates:

It’s February and already this article is out of date!

I tried setting up Github Actions for CI and boy is it a lot easier to configure than CircleCI. Highly recommend giving it a shot.

Poetry and pipx have been suggested as alternatives to pipenv for package management and running python applications (think npx) and I definitely plan to give these tools a closer look for my next project.

Also async python programming is becoming much more popular now. Some favorite servers and libraries mentioned were: starlette, fastapi, aiohttp, httpx, databases, Gino, asyncpg. At the application server level I’m not sure there is a big benefit to using asynchronous invocation handling if you’re already using functions as a service, but there are absolutely benefits to be had for making asynchronous calls to external services and database queries within a function invocation to parallelize requests.

Studying Russian as a Foreign Language

Click here to read the Russian version of this article.

I started studying the Russian language in university for a short time, then picked up formally studying it again since living and spending a lot of time in Eastern Europe. I love languages and find them super fascinating, especially syntax and orthography. For whatever reason these things are greatly interesting to me and fun to learn about. Like all nice theoretical studies, actual practical applications and reality dampen the fun somewhat.

To study practical everyday Russian is to study exceptions and unnecessary complexity. The language it most resembles for me is Latin both grammatically and in terms of the extra work you have to do that just seems somehow extra or not adding much.

When a student begins to study the Russian language, the first word they encounter as in every other language study is “hello!” In Russian this word is “Здравствуйте!” (Zdrastvuyte!) Which is five or six syllables depending how you count that preliminary imposing consonant cluster with a rolled “R”. The student immediately gets the sense that this isn’t going to be so easy. A first impression is that some key vowels seem to be missing from most words. How does one say “zdr”? From here it goes downhill.

When you want to say you are going somewhere in Russian it’s not enough to express the concept of travel, but it’s almost required to be accompanied by information about the mode of travel. There are words for specifying walking, going by means of wheeled conveyance, sailing, flying. Unlike many other languages, not only should the mode be specified but each verb consists of a unidirectional/multidirectional pair that must be discriminated (“I went to the store on foot end of story” vs “I went on foot to the store and then somewhere else”) as well as an imperfective/perfective pair that forces the speaker to consider if the action was completed or ongoing (“I walked” vs “I was walking”). On top of having to pick the right verb for the transport modality/perfective aspect/directionality combination one also should often prefix the verb to indicate if the motion is into, around, out of, on top of, under, through, out from, up to but not inside, and so on.

Along with learning the verbs of motion another fundamental aspect of the language is the noun inflection system. There are three genders, two numbers, and six main cases. Being an Indo-European language this system roughly similar to Latin, German, Old English and lots of other languages in this family.

While Russian’s case system is not particularly unique there is the case of the particularly unpleasant Russian genitive. There are a couple of historical reasons for this; the Proto-Indo-European ablative case was folded into the modern genitive case, and the genitive is used for expressions of quantity which still retain an archaic dual number form used for numbers of things between two and four. The abessive construction (“there is no book”) also uses the genitive form. The formation of the genitive plural is famously complicated and difficult and as a student of Russian you devote considerable time and practice to this very common and challenging puzzle.

As if that wasn’t enough to keep straight, the genitive form (but not construction) comes into play when a direct object has a soul, though only if it is a masculine noun. When declining a masculine animate direct object one uses the genitive form not the accusative.

The declension of numerals in Russian is its own special challenge which is hard to convey in English but involves a lot of genitive cases. You can look at the Wikipedia section to get a little taste though.

Most numbers ending with “1” (in any gender: оди́н, одна́, одно́) require Nominative singular for a noun: два́дцать одна́ маши́на (21 cars), сто пятьдеся́т оди́н челове́к (151 people). Most numbers ending with “2”, “3”, “4” (два/две, три, четы́ре) require Genitive singular: три соба́ки (3 dogs), со́рок два окна́ (42 windows). All other numbers (including 0 and those ending with it) require Genitive plural: пять я́блок (5 apples), де́сять рубле́й (10 rubles). Genitive plural is also used for numbers ending with 11 to 14 and with inexact numerals: сто оди́ннадцать ме́тров (111 meters); мно́го домо́в (many houses). Nominative plural is used only without numerals: э́ти дома́ (these houses); cf. три до́ма (3 houses; G. sg.). These rules apply only for integer numbers. For rational numbers see below.
https://en.wikipedia.org/wiki/Russian_declension#Numerals

The Russian writing and sound system isn’t so hard in my opinion for native English speakers, once you get used to the Cyrillic alphabet. There are some things that stand out as distinct compared to English however.

The first is the letter <Ы> which one pronounces a bit like the “i” in “bill”, but pronounced much lower in your throat. There is a distinction made between this sound and “И” (sounds like “ee”) although when the two sounds are spoken it’s damn hard to hear the difference normally. Also important distinctions are made between voiced and unvoiced “sh” and palletized consonants, which again I think are extremely difficult for a native English speaker to pick up on or enunciate without a great deal of practice. There is a letter “Ё” (“yo”) which is often just written as “E” (“ye”) which a Russian student just has to accept and assume they are being tricked sometimes. Finally there are the unspoken letters “Ь” and “Ъ” which are called soft sign and hard sign. The soft sign palletizes the letter in front of it and the hard sign mostly doesn’t do anything (occasionally depalletizing or demarcating a stop), and used to be extremely common in writing until the Bolsheviks ruthlessly purged it almost entirely from the language.

The Cyrillic alphabet is named after St. Cyrill who did not invent it but did invent an alphabet called Glagolitic (“speaking”) of which a few letters were taken for the Cyrillic alphabet, though aside from those characters Cyrillic is mostly cribbed from the Greek alphabet.

One of the easiest things to learn in Russian is the conjugation of verbs. In fact there is only really one set of personal endings you have to learn (well, two conjugations but they’re the same pattern). On top of this there are only three tenses to contend with (past, present, and future). The past is formed by adding gender markers (a standard masculine/feminine/neuter or one shared plural suffix) to the root, which is refreshingly simple although it seems strange that the gender is relevant for the conjugation of verbs and only in the past. Participles do exist but as far as I can tell they are basically never encountered and only as sort of a final footnote in most textbooks.

The only real complication with verbs is the previously mentioned perfective/imperfective aspect and prefixes. Verbs can have things stuck on the front to indicate if it is perfect or imperfect, and a future tense is simply formed by using a perfective form in what would normally look like the present, but since you can’t have a perfect action in the present, it means it’s the future instead. A bit odd but does make a certain kind of sense when you think about it.

Finally one last notable difference between English and Russian is swearing. The level of expressivity and creativity that appears to go into swearing in Russian is on another level, and probably beyond any sort of deep comprehension to a non-native speaker. Swear words also have vastly more force in Russian, and would almost never be uttered in any sort of semi-polite company and some probably would get you ejected from respectable society.

I am unable to really grasp the subtleties and complexity of the different prefixed and suffixed versions of “fuck” in Russian, and Google Translate doesn’t even try.

Studying Russian is a challenge, but getting all of the grammar right is really not the most important thing for daily usage. Even if you screw up most of the inflections people can still understand your meaning, and even native speakers get things wrong frequently. For me the hardest part is not speaking Russian but understanding what people say, mostly due to my unimpressive vocabulary and the speed in which people speak, but that can be true in any language and isn’t specific at all to Russian.

Русский – Изучать Как Иностранный Язык

Click here to read the English version of this article.

Изучать русский язык – изучать исключения и ненужную сложность.

Когда студент начинает учить язык, первое слово – «Здравствуйте!»

И сразу же он знает, что не будет легко. Чтобы просто сказать “hello” нужно пять слогов, и постоянно хочется найти недостающие гласные, как будто их не хватает. Как произносить “здр-”? Это очень пугающее начало, и дальше только труднее.

Например, когда человек хочет сказать о идти куда-то, ему нужно учитывать или “идти” это одно место или много мест – ”идти” или ходить – пешком или на колесном транспорте (ехать/ездить), “в” или “под” или “у” или “из” (уйти, выйти, прийти, …), и если совершенный или несовершенный вид (“пойти”), или на самолёте или на корабле.

После того как студент выучил “идти”, он может выучить как использовать существительные “на каждый день”. В отличие от английского, у русских существительных есть падежи. Есть простые правила, как например винительный падеж – ”я читаю книгу.” Окей, есть три рода, и это не так трудно, например окончания слов в мужском и среднем родах похожи, но когда студент изучает дальше, у него возникают сложности побольше .

Родительный падеж (кого?/чего?) – проклятие каждого изучающего русский язык. Родительный падеж был аблатив падежом давным давно, но сейчас они оба – только родительный. Поэтому, он используется во многих случаях. Когда чего-то нет, родительный падеж – ”нет книги.” Когда считаете, родительный падеж – ”12 стульев.” Образовывать множественные существительные в родительном падеже – возможно самое сложное в русском языке. Студенты русского практикуют это часто.

Форма родительного падежа тоже появляется когда используется одушевлённый объект в винительном падеже (только если мужской) – ”Я вижу коня”, а когда неодушевлённый объект – “Я вижу дом”, и тоже самое с числами два-четыре – ”Я вижу три вилки.”

Использовать числа тоже очень сложно. Они склоняются конечно, а также сохраняют свои архаичные двойсвестние формы из старославянского языка для чисел 2-4. Когда несчастный студент хочет сказать о количествах, ему обязательно нужно думать или число заканчивается на один, два, три, четыре, или больше.

Один стул – (форма именительного падежа)

Два/три/четыре стула – (форма родительного падежа – двойсвестный)

Пять и т.д. стульев – (форм множественного родительного)

Дальше, «двенадцать стульев» но «двадцать один стул». Ой вей.

И если бы хотел говорить о собирательных числительных, также используют множественный родительный даже для чисел 2-4: «двое друзей». Смотрите ещё больше сложностей здесь.

Русская орфография и фонология менее трудна, немного исключений для носителей английского языка. Буквы <Ы> (делается из «ъ + і») и её звука /ɨ/ нет в английском языке. Трудно произносить, и слышать разницу между <И> и <Ы>, так же как и между <Ш> и <Щ>. Мягкий знак <Ь> и твёрдый знак <Ъ> просто сбивают с толку. А буква <Ё> часто пишется как <Е>, интересно, что людей которые пишут её правильно называют ёфикаторами.

Кириллица – алфавит назван в честь Кирилла и Мефодия главным образом состоит из Греческого алфавита, с немногими символами из Глаголицы «Аз Буки Веди» представляющими звуки, которых нет в Греческом алфавите (как <Ш>). Носители русского языка не верят мне, когда я говорю русский алфавит – это почти всё из Греческого алфавита. Ещё я покажу эту таблицу.

Некоторые вещи легче в русском языке чем у большинства индоевропейских языков. Например глаголы. Спряжение глагола обычно регулярное, и только одна основная форма, это легко запомнить.

Так же глаголов только три времени (прошлое, настоящее, будущее), реже используемые деепричастия, а наиболее совершенные глаголы формируются префиксами (писать ⇾ написать).

А много слов таких же как и в романском, немецком, латинском и английском языках. Немного примеров:

Есть – как “est” на латином (to be)

Ярмарка – как “jahr” на немецком

Проект – как “projekt” на немецком

Бровь – “brow”

Бить – “beat”

Быть – “be”

Ложь – “lie”

Носок – “sock”

Щека – “cheek”

Пламя, плыть, плот, полёт – (flame, fleet, float, flight) – общий индоевропейский сдвиг из /ф/ от /п/ (c.f. Lat. “pater” ⇾ “father”)

О мате, я знаю мало. Не понимаю русские ругательства – это так сложно. В английском языке ругательства не очень сильные или оскорбительные, но люди советуют мне не говорить эти слова на русском. На самом деле в английской Википедии – ”In modern Russia, the use of mat is censored in the media and the use of mat in public constitutes petty hooliganism, a form of disorderly conduct, punishable under article 20.1.1 of the Offences Code of Russia.” Мало информации в английской Википедии о мате, но в русской Википедии – длинная статья, которую я не могу понять и Гугл Перевод не помогает.

Изучать русский язык – это приключение и довольно не быстро. С практикой, возможно обращаться к людям, но сложнее понимать когда они отвечают. Если я читаю со словарём или кто-то говорит медленно, я могу понять много. Говори со мной и помогай мне учить!

Serverless WebSockets

WebSockets, the standard for doing real-time bidirectional communication typically between a browser and a server, is a fair attempt to create a standard to supplant the previously employed hacky solutions and continues to evolve in terms of implementation.

The basic idea has primarily been to establish some sort of channel in which a server can “push” events to a client, rather than the client “polling” every so often to see if there is new information. This was until fairly recently a relatively obscure concept, but now any smartphone owner is extremely well-acquainted with push notifications. This real-time channel has been used for not just notifications but also services like VOIP and gaming.

In the days before the WebSocket standard various semi-clever attempts to implement push notifications were devised. The first was using <iframe>s to load an HTML document using chunked encoding, where the server would write a script tag with some new data in the form of JavaScript commands when the data became available. When the browser encountered a closing script tag it would execute the JS immediately even though the document was still streaming.

The next scheme was using XML HTTP Request (aka XHR [aka AJAX]) to do something similar but without needing an <iframe>. This was known as “long-polling”, or “comet.” This was still mostly a unidirectional channel and suffered from timeouts and reconnection issues with potential race conditions.

Now with WebSockets we have a much improved system and wide browser support. But what about the backend? What happens when a browser or other client connects to a WebSocket server?

Previously we’ve developed and hosted WebSocket servers written in Perl, Go, and Python, using PostgreSQL asynchronous events as the message passing system. Deploying WebSocket servers is not as straightforward as HTTP servers because of the long-lived connections and having to perform TCP load balancing. Depending on your hosting setup you may have to deal with internal timeouts or getting events from your message bus to the right backend via some subscription mechanism.

Architecture

Since I love not running servers I’ve been excited about the chance to use serverless WebSockets via AWS API Gateway. In this new scheme you define Lambda functions that react to events such as authentication, connect, disconnect, and user-defined events that can be read from JSON message bodies.

Infrastructure-wise the setup is extremely basic. All of the real work to handle authorization and events and done in code, which we will look at shortly. Let’s use a concrete example of a typical WebSocket use case – sending notifications from the server to the client to inform it of some data change in order for the client to update some information in real time or notify the user.

For my application I created an authorizer function that validates a JWT encoded in the WebSocket URL query parameters (there is no good way in a browser to set headers when opening a WebSocket connection). This function denies or grants access to proceed and saves the authenticated user ID in the principalId response field, which is passed along to subsequent event handlers.

Once the authorization check is successful the special $connect route is called if there is a handler defined. In this handler we have the user ID in the invocation event passed along from the authorizer response and we have a connectionId. We save this user ID and connection ID pair in our database so that we can know who is connected and have the ability to send them a notification later on using their connectionId.

The API Gateway makes a best-effort attempt to detect disconnections and invokes the special $disconnect route whereupon our handler removes the connection record from the database.

Putting all of these pieces together with actual working code required me gathering a fair bit of information from different sources and working out the proper request fields and response formats but it all worked out wonderfully in the end. I’d like to share the working code examples for the handlers and some sample client code as well.

The Code

To define your handlers and when they get invoked you need to configure API Gateway to register your authorizer handler and the assorted route handlers. Using the Serverless toolkit this is straightforward and nicely documented. My configuration looks something like:

functions:
  # websocket authorizer
  wsAuth:
    handler: notifier.ws.handler.authorizer

  # websocket $connect
  wsConnect:
    handler: notifier.ws.handler.connect
    events:
      - websocket:
          route: $connect
          authorizer:
            name: wsAuth
            identitySource:
              - route.request.querystring.token  # token query param

  # websocket $disconnect
  wsDisconnect:
    handler: notifier.ws.handler.disconnect
    events:
      - websocket:
          route: $disconnect

And the authorizer:

def authorizer(event, context):
    method_arn = event.get("methodArn")
    def deny(msg):
        return {"message": msg,
                "policyDocument": gen_policy(method_arn=method_arn, allow=False)
        }

    # get access token from query string
    query_params = event.get("queryStringParameters")
    if not query_params:
        return deny("missing queryStringParameters")
    if "token" not in query_params:
        return deny("missing token in query string")
    token = query_params["token"]
    if not token:
        return deny("empty token")

    # decode and verify JWT token
    decoded = None
    try:
        decoded = decode_token(token)
    except ExpiredSignatureError:
        return deny("Expired token")

    identity = decoded.get("identity")
    if not identity:
        raise Exception("invalid JWT; missing identity")

    # allow access
    policy = gen_policy(method_arn=method_arn, allow=True)
    context = {}  # can add more auth context info here if desired
    res = {
        "principalId": identity,
        "policyDocument": policy,
        "context": context
    }
    return res

def gen_policy(method_arn: str, allow: bool):
    effect = "Allow" if allow else "Deny"
    return {
        "Version": "2012-10-17",
        "Statement": [{
            "Action": "execute-api:Invoke",
            "Effect": effect,
            "Resource": method_arn
        }],
    }

This looks for a JWT in the query string and attempts to parse and validate it. If successful then an IAM policy is returned along with the decoded identity ID. The details of the event and policy can be found in the Lambda REQUEST WebSocket authorizer documentation.

If the client is granted Invoke access to the execute-api service then API Gateway will call our $connect route next:

def connect(event, context):
    ctx = event.get("requestContext", {})
    # get user and connection id
    conn_id = ctx.get("connectionId")
    auth = ctx.get("authorizer", {})
    user_id = auth.get("principalId")

    if not user_id:
        return make_response(401, "Not authorized")

    if not conn_id:
        raise Exception("missing connectionId")

    # save the connection id/user id pair in DB
    WebsocketClient.save_connection(
        user_id=user_id,
        connection_id=conn_id,
        domain_name=ctx["domainName"],
        stage=ctx["stage"],
    )
    db.session.commit()

    return make_response(200, "ok")

def make_response(status_code, body):
    if not isinstance(body, str):
        body = json.dumps(body)
    return {"statusCode": status_code, "body": body}

The purpose of this route is to store the user ID and connection ID in the database along with the connection’s domain and stage. We will use this to send our notification to the client.

def send_ws(user_id, message):
    """Push a notification to the user if they have an active websocket connection."""
    connections = WebsocketClient \
        .query \
        .filter_by(user_id=user_id) \
        .all()

    for conn in connections:
        conn.send(message)

And conn.send():

import boto3
import json
from notifier.db import db, Model
from botocore.exceptions import ClientError

class WebsocketClient(Model):

    ...

    def send(self, message):
        """Send a message to an active connection.

        :param message: can be anything that is JSON-serializable."""
        # get APIGW management client
        apigw_mgmt_client = boto3.client(
            "apigatewaymanagementapi",
            endpoint_url=f"https://{self.domain_name}/{self.stage}",
        )
        try:
            # send message
            apigw_mgmt_client.post_to_connection(
                Data=json.dumps(message).encode("utf-8"),
                ConnectionId=self.connection_id,
            )
        except ClientError as err:
            # gracefully handle case where client is no longer connected
            code = int(err.response["Error"]["Code"])
            if code == 410:
                # client gone, cleanup
                db.session.delete(self)
                db.session.commit()
                return
            raise

This is the where the real action happens. When we want to send a message from the server to the client we do it with the PostToConnection call. We need to provide the API Gateway domain and stage for it to construct the URL needed for the API call. Boto is simply doing HTTP requests to interact with the WebSocket connection as documented here. And you can use an HTTP client directly if you like to get connection info, send a message, and close the connection.

For completeness let’s look at handling the $disconnect route:

def disconnect(event, context):
    # get connection ID
    ctx = event.get("requestContext", {})
    conn_id = ctx.get("connectionId")
    if not conn_id:
        raise Exception("no connection id found")

    # delete the connection record from our DB
    WebsocketClient.delete_connection(connection_id=conn_id)
    db.session.commit()
    return make_response(200, "ok")

Client ➞ Server Messages

But wait, there’s more!

Our application is now ready to send notifications to our client, but if we want to be able to receive messages from the client we can support this case as well. We can define custom routes that are matched based on a route key as documented here and here. In practice this means that if API Gateway receives a JSON message it looks for the route name by default in a field called "action" and decides which Lambda to call based on that value. You can also create a $default route to catch any unhandled message if you prefer to do things that way as well.

Client Code

I implemented a basic WebSocket client in TypeScript using the standard WebSocket API. The only special thing it does is append your access token (managed with axios-jwt) to the WebSocket connection URL.

import { refreshTokenIfNeeded } from 'axios-jwt'

export const WEBSOCKET_EVENT = 'onwebsocketmessage'

export class WSEvent extends Event {
  message: object

  constructor(msg: object) {
    super(WEBSOCKET_EVENT)
    this.message = msg
  }
}

export type WSEventHandler = (ev: WSEvent) => void

export default class WSClient extends EventTarget {
  ws: WebSocket | undefined
  public isConnected: boolean = false
  reconnectTime: number = 1 // time in seconds before reconnect

  // connect
  public open = async () => {
    if (this.ws) {
      if (this.ws.readyState === WebSocket.CONNECTING || this.ws.readyState === WebSocket.OPEN)
        // already open/opening
        return

      this.ws.close() // do reconnect
    }

    // config from create-react-app+dotenv
    if (!process.env.REACT_APP_WS_URL) throw new Error('REACT_APP_WS_URL missing')
    const host = new URL(process.env.REACT_APP_WS_URL)

    // make sure auth token is fresh
    // requestRefresh defined elsewhere - see axios-jwt documentation
    const accessToken = await refreshTokenIfNeeded(requestRefresh)

    // add auth token to URL
    if (accessToken) host.searchParams.set('token', accessToken)

    // create new websocket client
    if (!this.ws) {
      this.ws = new WebSocket(String(host))
      this.ws.onopen = this.handleOpen
      this.ws.onclose = this.handleClose
      this.ws.onmessage = this.handleMessage
    }
  }

  // disconnect
  public close = () => {
    if (this.ws) this.ws.close()
  }

  public reconnect() {
    if (this.ws) this.ws.close()
    this.open()
  }

  // CALLBACKS

  protected handleOpen = (ev: Event) => {
    this.isConnected = true
    this.reconnectTime = 1 // reset reconnect timer

    const ws = this.ws
    if (!ws) return
  }

  protected handleClose = (ev: Event) => {
    this.isConnected = false

    // do reconnect
    setTimeout(() => {
      this.reconnectTime *= 2 // exponential backoff

      this.open()
    }, this.reconnectTime * 1000)

    // reconnect?
    this.open()
  }

  protected handleMessage = (ev: MessageEvent) => {
    // handle message received on WS
    const data = ev.data
    if (!data) return

    // try to parse as JSON
    const msg = JSON.parse(data)

    // create new websocket event and dispatch it to listeners
    const msgEvt = new WSEvent(msg)
    this.dispatchEvent(msgEvt)
  }
}

And as a bonus here’s a React hook that lets you register an event handler for WebSocket messages:

import * as React from 'react'
import WSClient, { WEBSOCKET_EVENT, WSEvent } from './api'

// singleton
let client: WSClient

interface IUseWebSocketClientArgs {
  onEvent?: (evt: WSEvent) => void
}

const useWebSocketClient = ({ onEvent }: IUseWebSocketClientArgs) => {
  React.useEffect(() => {
    if (!client) client = new WSClient()

    // listen for events
    if (onEvent) client.addEventListener(WEBSOCKET_EVENT, onEvent as EventListener)

    // ensure client is connected
    client.open()

    // cleanup handler
    return () => {
      if (onEvent) client.removeEventListener(WEBSOCKET_EVENT, onEvent as EventListener)
    }
  })
  return { client }
}

export default useWebSocketClient

Conclusion

Like many other serverless technologies this approach is certainly not practical for every use case but it is quite reasonable for a lot of common cases. While API Gateway WebSockets kind of support binary data payloads the serverless approach is probably best suited to your application if you’re passing occasional JSON messages around and dealing with relatively low throughput and volume.

Web Application Boring Stack: 2019 Edition

At JetBridge we enjoy developing software applications with our clients that we can take pride in while expanding our areas of knowledge and expertise at the same time. Because we are frequently starting on new projects we have standardized on a harmonious and expressive set of tools and libraries and frameworks to help us rapidly lift off new applications and deliver as much value as we can with minimal repetition.

Our setup isn’t perfect or the end-all stack for every project, but it’s something we’ve evolved over years and it works quite well for us. We continue to learn about new tools and techniques and evolve our workflow so consider this more of a snapshot in time. If you aren’t reading this in July of 2019 then we have probably modified at least some parts of the stack.

Methodology

Our theory of software development is: don’t overcomplicate things.

Pragmatism and business value are the overriding concerns, not the latest and coolest and hippest frameworks or tech. We love playing with new cool stuff as much as any geek but we don’t believe in using something new just for the sake of being new or feeling unhip. Maturity and support should factor into deciding on a library or framework to base your application on, as should maintainability, community, available documentation and support, and of course what actual value it brings for us and our clients.

There is a tendency a lot of engineers have to make software more complex than it needs to be. To use non-standard tools when widely available and known tools exist that might already do the job. To try to shoehorn some neat piece of tech someone read about on Hacker News into something it isn’t really suited for. To depend on extra external services when there are already existing services that can be extended to perform the desired task. Using something too low-level when more abstraction would really simplify things, or using something too fancy and complicated when a simple system-level tool or language would accomplish things more expediently.

Simplicity is a strategy that when used wisely can greatly increase your code readability and maintainability, as well as result in easy to manage operational environments.

Frontend

By the time I am writing this all frameworks and libraries we use have likely been superseded by cool new hip JS jams and you will sneer at our unfashionable choices. Nevertheless, this is what is working well for us today:

React: Vue may have more stars on GitHub but React is still the standard and is used and supported actively by Facebook, among others. Writing apps with React hooks really feels like we are getting closer and closer to functional programming, adding a new level of composibility and code reuse that was clumsily achieved with HOCs before.
Material-UI for React is a toolkit that has almost every sort of widget and utility you might need, powerful theming and styling options, integrates CSS-in-JS very smoothly and looks solid out of the box. It is essentially an implementation of the UI paradigms promulgated by Google so working within its constraints and visual language gives you a reasonable starting point.
Create-React-App/react-scripts: This really does everything you need and configures your new React app with sane defaults. You never need to monkey around with Webpack or HMR again. We have extended CRA/r-s to spit out new frontend projects with extra ESlint and prettier options and Storybook.
Storybook: We prefer to build a component library of small and larger components implemented in isolation using mock data, rather than always coding and testing the layout and design inside the complete app. This allows UI devs to work without being blocked on completion of backend endpoints, helps to enforce the concept of reusable and self-contained components, and lets us preview the various interface states easily.
TypeScript: Everyone uses TypeScript now because it’s good and you should too. It does take some getting used to and learning how to use it properly with React and Redux requires some small amount of learning, but it’s entirely worth it. Remember: you should never need to use any. And when you think you need to use any – you probably just need to add a type argument (generic).
ESLint: ESlint works great with TypeScript now! Don’t forget to set extends: ['plugin:@typescript-eslint/recommended', 'plugin:react/recommended', 'react-app']
Prettier: Set up your editor to run Prettier on your code when you hit save. Not only does it enforce a consistent style, but it also means you can be way way lazier about formatting your code. Less typing but better formatting.
Redux: Redux is nice… I guess. You do need some central place to store your user authentication info and stuff like that, and redux-persist is super handy. In the spirit of keeping things simple though, really ask yourself if you need redux for what you’re doing. Maybe you do, or maybe you can just use a hook or state instead. Sure maybe you think at first that you want to cache some API response in redux, but if you start adding server-side filtering or search or sorting, then it really is better off just as a simple API request inside your component.
Async/await: Stop using the Promise API! Catch exceptions in your UI components where you can actually present an error to the user rather than in your API layer.
Axios: The HTTP client of choice. We use JWT for authentication and recommend our axios-jwt interceptor module for taking care of token storage, authorization headers, and refresh.
Cypress: A popular tool for writing end-to-end tests. Cypress makes it easy to mock API responses and fully test your application as an automated web browser, either headless or used interactively. Can record videos and screenshots of every state and step of your tests to review what your UI looks like and how it reacts even after automated test runs.

I don’t believe there’s anything crazy or unusual here and that’s sort of the point. Stick with what’s standard unless you have a good reason not to.

Backend

Our backend services are always designed around the 12-factor app principles and always built to be cloud-native and when appropriate, serverless.

Most projects involve setting up your typical REST API, talking to other services, and performing CRUD on a PostgreSQL DB. Our go-to stack is:

Python 3.7. Python is clean, readable, has an impressively massive repository of community modules on PyPI, active core development, and a pretty good balance of high-level dynamic features without getting too obtuse or distracting.
Type annotations and type linting with mypy. Python does have type annotations, but they are very limited, not well integrated, and not usually very useful for catching mistakes. I hope the situation improves because many errors have to be discovered at runtime in Python when compared with languages like TypeScript or Go. This is the biggest drawback to Python in my opinion, but we do our best with mypy.
Flask, a lightweight web application framework. Flask is very nicely suited to building REST APIs, providing just enough structure to your application for handling WSGI, configuration, database connections, reusable API handlers, tracing/debugging (with AWS X-Ray), logging, exception handling, authentication, and flexible URL routing. We don’t lean on Flask for much besides providing the glue to hold everything together in a coherent application without imposing too much overhead or boilerplate.
SQLAlchemy for declarative ORM. Has nice features for handling Postgres dialect features such as UPSERT and JSONB. Ability to compose mixins for model and query classes is very powerful and something we are using more and more for features like soft deletion. Polymorphic subtypes are one of the most interesting SQLAlchemy features, allowing you to define a type discriminator column and instantiate appropriate model subclasses based on its value.
Testing: subtransactions wrapping each test, pytest-factoryboy for generating fixtures from our model classes for pytest and for generating mock data for development environments. CircleCI. Pytest fixtures. Flask test client.
Flask-REST-API with Marshmallow helps succinctly define REST endpoints and serialization and validation with a minimum of boilerplate, making heavy use of decorators for a declarative feel when appropriate. As a bonus it also generates OpenAPI spec documents and comes with Swagger-UI to automatically provide documentation of every API endpoint and its arguments and response shapes without any extra effort required.
We are currently developing Flask-CRUD to further reduce boilerplate in the common cases for CRUD APIs and mandating strict data model access control checks.

In projects that require it we can use Heroku or just EC2 for hosting but all of our recent projects have been straightforward enough to build as serverless applications. You can read about our setup and the benefits this brings us in more detail in this article.

We have built a starter kit that ties together all of our backend pieces together in a powerful template to bootstrap new serverless Flask projects called sls-flask. If you’re thinking of building a database-backed REST API in Python, give it a try! You get a lot of power and flexibility in a small bundle. There isn’t anything particularly special or exotic included in it, but we believe the foundation it provides adds up to an extremely streamlined and modern development toolkit.

All of our tooling and templates are open source, and we often contribute bug reports and fixes upstream to modules that we make use of. We encourage you to try out our stack or let us know what you’re using if you’re happy with what you’re doing. Share and enjoy!

Video Encoding on AWS

Adding video encoding support to your application is relatively straightforward with Amazon’s Video On Demand encoding pipeline infrastructure template.

This CloudFormation template provides you with:

A S3 media source bucket where video files get uploaded, with an option to phase out media source files to long-term storage in Glacier.
A DynamoDB table to track the status of the encoding and store all metadata about the source and output files.
A series of Step Functions (Lambda state machines) to manage the stages of the pipeline.
MediaConvert to do the actual video encoding work.
An output S3 bucket for the encoded files and playlists, with a CloudFront CDN distribution in front.
A SNS topic which publishes events to subscribers when media ingestion begins and when it completes, as well as if there is an error.

The one deficiency in the CloudFormation template provided by AWS is that it does not include the SNS topic as a stack output, which makes it harder to tie it into other applications. JetBridge hosts a version of the stack which includes the SNS topic output at https://ext.jetbridge.com.s3.amazonaws.com/vod/video-on-demand-on-aws.template.

You can deploy the stack here:

Once the stack has finished launching, you can try uploading a video file into the source S3 bucket.

When files are added to the bucket a Lambda is automatically triggered that begins the ingestion and kicks it over to MediaConvert after generating a GUID to track the progress of the encoding.

After the encoding is complete you will have an entry in the DynamoDB table with information about the media files and the outputs, including a HLS M3U8 (HTTP Live Streaming MP3 URL UTF-8 playlist) which can be used by any web or mobile client to stream your video at adaptable bitrates.

Integrating To Your Application

The VOD encoder pipeline is a pretty nifty example of how to use ready-made stacks of infrastructure, but what if you want to integrate this pipeline into your application? Let’s look at one way you can accomplish this.

Say you are building a CMS where you want users to be able to upload videos that can be streamed by clients. You will need a user interface for performing the upload and then a way to associate the results with that object when the encoding process completes or errors.

The flow from the application’s perspective will look like this:

Register a Lambda for handling notifications from the VOD SNS topic.
Create an object in your database to store the uploaded video. A row in a video table would suffice just fine. Make up a S3 key for this row (based on the video’s ID or better, UUID) and store it in the video row as well.
Generate a pre-signed S3 PutObject request URL (Python docs) for the media source bucket.
On the browser side, upload the video file to the pre-signed S3 upload URL. Once the upload is complete the Lambda trigger will be automatically invoked, kicking off the encoding job.
Process ingestion notification received from the SNS topic. This notification includes the UUID generated by the pipeline to keep track of your job and the original S3 key of the video file that was just uploaded. Store the VOD task UUID in your video database row associated with the S3 key.
When you receive a completion or error notification from the SNS pipeline, update the video row appropriately. You now have either a HLS playlist URL associated with your video or an error message.

Registering For SNS Notifications

You can set up everything above by hand, but making reusable infrastructure is easier and more powerful. If you are using the Serverless toolkit you can use the SNS topic CloudFormation output (remember the one mentioned above that we had to add to the template?) to register a Lambda to listen for events:

functions:
  vodSnsUpdateHandler:
    handler: myapp.handler.vod_sns_update.handler
    events:
      - sns: ${cf:vod.SnsNotificationTopic}  # cloudformation output

This will invoke the function myapp.handler.vod_sns_update.handler whenever a new message is published on the SNS topic in the CloudFormation stack named vod (that’s what I called it, you can change it if you really want).

Other CloudFormation Stack Outputs

Your application will also need to know the name of the source media S3 bucket to generate the presigned upload request as well as the name of the DynamoDB table to fetch the results from. Again, this example is for Serverless:

provider:
  name: aws
  ...
  environment:
    S3_VOD_SOURCE_BUCKET: ${cf:vod.Source}
    VOD_TABLE: ${cf:vod.DynamoDBTable}

This has the effect of passing the source S3 bucket and DynamoDB table names from the VOD stack outputs into your application as environment variables.

S3 Presigned Upload

You can create a URL that you can give to a client to permit it to upload a file to a designated S3 key:

    s3 = boto3.client("s3")
    put_params = dict(Bucket=os.environ['S3_VOD_SOURCE_BUCKET'], Key=s3key)
    expire = 3600  # one hour
    url = s3.generate_presigned_url(
        ClientMethod="put_object", 
        Params=put_params, 
        ExpiresIn=expire,
    )

This URL can then be returned to a web browser which can then do a PUT to the URL with the contents of the file as the body of the request.

I recommend generating a S3 key in the form of: f"/video/{video.uuid}/media.mp4"

Processing SNS Notifications

This should be a Lambda handler that looks up the associated video entry in your database and updates it with the status published by the VOD pipeline. Some rough sample code:

import boto3
import os
import json
from myapp.db import db
from myapp.model.video import Video
from enum import Enum, unique
from typing import Optional
import logging

log = logging.getLogger(__name__)

@unique
class EncodingStatus(Enum):
    new = "new"
    ingest = "Ingest"
    complete = "Complete"
    error = "Error"

table = os.environ["VOD_TABLE"]
dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table(table)


def handler(event, context):
    records = event.get("Records", [])
    with app.app_context():  # if you use Flask-SQLAlchemy
        for record in records:
            log.debug(f"Processing VOD SNS event...")
            process_event_record(record)
        db.session.commit()
    return "ok"


def process_event_record(record: dict):
    assert "Sns" in record
    assert "Message" in record["Sns"]
    message = json.loads(record["Sns"]["Message"])

    # look up asset by key/bucket
    src_video = message.get("srcVideo")

    status = EncodingStatus(message.get("status", message.get("workflowStatus")))
    guid = message.get("guid")
    log.debug(f"Video: {src_video}, status={status}, guid={guid}")

    if not src_video:
        # this is missing in case of error
        if status == EncodingStatus.error:
            video = db.session.query(Video).filter_by(vod_guid=guid).one_or_none()
            if not video:
                log.warning(f"Got video GUID for unknown video {record}")
            else:
                video.encoding_status = status
        log.warning(f"Got video encoding without video src {record}")
        return None

    # look up video by S3 key
    video = Video.query.filter_by(s3key=src_video).one_or_none()
    if not video:
        log.warning(f"Could not find video {src_video}")
        return None

    # update video
    video.vod_guid = guid
    video.encoding_status = status
    video.vod_last_message = message
    video.hls_url = message.get("hlsUrl") if message.get("hlsUrl") else video.hls_url
    thumbnail_urls = message.get("thumbNailUrl", [])
    video.placeholder_url = thumbnail_urls[0] if thumbnail_urls else None
    video_data_info = get_video_data_info(guid)

    if not video_data_info:
        if status == EncodingStatus.complete:
            log.warning(f"Could not find data about encoding {record}")
        return asset

    src_media_info = video_data_info.get("srcMediainfo")
    encoding_details = json.loads(src_media_info) if src_media_info else None

    if not encoding_details:
        log.warning(f"Could not find encoding info {record} // {encoding_details}")
    video.duration = encoding_details["container"]["duration"]  # ms

    print(f"Media info: {src_media_info}")
    db.session.commit()

def get_video_data_info(guid: str) -> Optional[dict]:
    result = table.get_item(Key={"guid": guid})
    return result.get("Item")

Conclusion

And now you have a powerful media encoding pipeline integrated into your application. Some features to note are :

Thumbnail URLs are automatically generated.
Media info is output which contains everything from duration to dimensions to colorspace.
HLS, DASH, and MP4 outputs are produced.
Quality-Defined Variable Bitrate encoding is used by default.
Microsoft Smooth Streaming (MSS) and Common Media Application Format (CMAF) are also supported.

Hope that was helpful!