Decentralizing Social Media

No, it has nothing to do with blockchain.

What kind of language should Facebook forbid? What kind of regulations should the U.S. government promogulate regarding whom Twitter can ban?

Who cares?? ☜
Not me.

A depressing amount of energy and ink is wasted on these questions which shouldn’t even be issues in the first place. We don’t have to base our public discourse on platforms that corporations or even governments control.

The great news is that there does exist an alternative to the model of having all social media content go through a couple of companies. There is certainly no technical reason it should work that way, and there is a solution to the problem that has a foundation in technology, though there is naturally a social component as well.

What is this problem that needs a solution? I think it’s fantastically illustrated by all of these articles and experts and laws being passed to try to nudge Facebook, Youtube, Twitter to control what people are allowed to say and post. Busybodies, Concerned Citizens, corrupt politicians, think tanks, your parents, all want to petition these platforms to decide what you should be able to read or write. I view this as a problem, because I don’t think anyone should decide for me what information I should be able to share or consume. Not Mark Zuckerberg, not Donald Trump, not Jack Dorsey, not my congressional representatives, not the People’s Republic of China.

Government and private corporations in control of censorship are not the only problem here. As everyone knows these services are free, and as everyone also knows if the product is free then you are the product. Facebook and Google make almost all of their money from extracting and mining as much personal data about you as possible to sell to advertisers, PR agencies, and politicians. There is a better way.

Federated Social Media

The answer is federation. Decentralization. Distributed systems. You’re already familiar with the concept, just think of email. You don’t have an email username, you have an address. Your email address is a username on a host – mspiegelmock@gmail.com specifies the user mspiegelmock on the system gmail.com. I can write an email to someone else like rms@gnu.org, even if they don’t use gmail. I ask my provider gmail.com to send a message to the gnu.org host which is responsible for delivering it to the user rms.

No company “runs” email, yet all email servers know how to pass messages to each other. There is a vast array of different email hosts, providers, server applications and client apps. You can choose to sign up with a free provider like Google or Microsoft, your employer may provide you an account, or you can run your own server. You can use any sort of app you like with email, such as gmail.com, Apple Mail, Superhuman, Outlook, mutt, or emacs. In the earlier days of web-based mail there were a few options to choose from, like Yahoo and Hotmail, and eventually the company which provided the best user experience ended up grabbing a significant slice of email users, thanks to the wonders of competition. There were and are some issues with spam and malicious content to be sure, though a great amount of progress has been made on systems to combat it (spam and virus filters, real-time blacklists, DKIM/SPF). This is what federation looks like.

To grasp the concept behind federated social media, think of email. You sign up for an account with an instance (“host”) that you feel comfortable with, or run your own if you’re so inclined.

To follow someone you need their address, like @wooster@social.coop.

Think Twitter, but as an email address. The address denotes the username (@wooster) and the host the user is registered on (social.coop).

You have two timelines, in addition the people you follow. One timeline is the “local” timeline, which is everyone else on your instance. If you join an instance of people that share a particular hobby, language, interest, region or philosophy you get to start out with a feed that may have posts that may be relevant for you. Your instance can link, or “federate”, with other similar instances, connecting users on your instance with users on the other instances.

Moderation

Why not run a whitehouse.gov instance instead?

Just because there is no central authority for content moderation doesn’t mean that the system is full of abuse and Those Sorts Of People you would like to avoid. These things exist to be sure, as they do on any platform, but they are confined to their own instances. Moderation does exist, but unlike Facebook or Twitter you can choose your moderators. Most instances have policies about what external content they block, what types of instances they want to federate with, and what kinds of content they permit. If you disagree with their policies, you are free to join an instance that fits with your preferences, or start your own.

There are plenty of people I don’t want to hear from, there are plenty of posts our there that would decrease my quality of life, and I’m fine with outsourcing some moderation. I just don’t want this guy to be the final arbiter of all information.

Propaganda, trolls, abuse, and misinformation exist on every platform. You can find it on YouTube, LiveJournal, TikTok, Twitter, and no doubt on federated social media. Media literacy is an important skill that should be taught to help media consumers understand biases and distortions inherent in all media. A platform that helpfully provides fact-checking would be desirable to many users. But the fact remains that you cannot outsource critical thinking. There’s no getting around this.

The problem with top-down centralized structures that take it upon themselves to decide what information can or can not be spread should be plain. Corporations and politicians have bad incentives and the temptation to misuse such power to cover up misdeeds is too powerful for most to resist. We all know now what happened with Chernobyl and the misery caused by the suppression of information. Maybe if we had some supremely enlightened and benevolent information despot it would be okay to put them in charge, but I can’t really think of anyone I want to grant that authority over me.

The instance I belong to is social.coop, a social media cooperative. It’s a group of people who donate a small amount of money to pay for a server to host a Mastodon instance and volunteers who help maintain and administer it. There is an online forum for discussions and consensus-based decision-making, and a lot of smart people on it. This is just one example of the kind of self-organization that is possible in the fediverse.

Mastodon

Today the most popular software for plugging into the federated social media network is Mastodon. It’s free (AGPL) and open source naturally, and there are a number of apps you can use with it, including some slick paid apps. The Mastodon web interface looks something like this:

And the “Toot” app looks like this:

Toot for iOS

Many communities run Mastodon instances, some are public, some are cooperatives, some are private. You don’t have to use Mastodon to talk to people using Mastodon, if you use software that speaks ActivityPub then you can follow, share, post, like, comment, and communicate with anyone else in the fediverse. Just like if you use any email software, you can email anyone else using email software, which I think is pretty neat.

Mastodon is federated social media, federated social media is not Mastodon.

Technical Details (ActivityPub)

In the early days there were a number of attempts at creating social networking protocols with really obnoxious names like “pubsubhubbub.” After a bit of experimentation an official standard was published by the World Wide Web Consortium (W3C) in 2018, going by the name ActivityPub. You may know the W3C from their earlier hits like HTML, CSS, XML, and SVG.

As it’s an extremely recent standard, we’re still in the very early days of implementations. I expect there will be a number of libraries and clients popping up, along with not just standalone servers but server capabilities integrated into existing platforms and sites. Any site that lets you log in and post content could be modified to plug into the fediverse by implementing the relatively simple, JSON-based protocol. Your existing accounts could turn into ActivityPub Actor objects. New social networks can in effect bootstrap themselves by leveraging existing users, software, and federation networks, and we’ll see companies and open source projects compete to offer the best user experience.

Honestly the best way to learn about ActivityPub is to go read the standard doc. It’s in plain English with friendly cartoons and very straightforward.

Actor with messages flowing from rest of world to inbox and from outbox to rest of world

The very brief gist of it is that there is a client ↔︎ server protocol and a server ↔︎ sever protocol, much like IRC. You can read notifications that arrive in your inbox and you can publish messages to the world via an outbox. All content and objects on the platform are simple JSON documents that live at a URL (an IRI to be precise). Technically you could make a compliant ActivityPub server with a static webserver (I think? Tell me if I’m wrong).

Social media isn’t just text and image posts. The site PeerTube is a decentralized version of YouTube:

PeerTube is a free and open-source, decentralized, federated video platform powered by ActivityPub and WebTorrent, that uses peer-to-peer technology to reduce load on individual servers when viewing videos.

User Base

The network effect is what makes a social network attractive. The more people on it, the more useful it is. Having celebrities is a big draw for many people. It’s the biggest challenge that any challenger to the status quo faces.

I believe there is nothing permanent about Facebook or Twitter. I remember when it was hard to imagine anything replacing MySpace, or when everyone on the Russian-speaking internet had a LiveJournal. Fads change and great masses of people move smoothly to new platforms with ever increasing rapidity.

https://the-federation.info/ – Federation Stats

The thing that is so powerful about a social media protocol is that it seems, at least to me, like the logical conclusion of social media. Once a growing critical mass of people move to it, either because they are sick of Facebook’s shit or some cool new company’s platform happens to be powered by ActivityPub, most of the innovation around communities, software, moderation, new forms of media, organization and technology can happen within the fediverse. Because of the open and extensible nature of the underlying foundation anyone can plug in a conformant piece of software and shape their piece of it how they want while still interoperating with the wider world. It’s a perfect vehicle in which one can imagine fictional visions of the future internet taking shape, like the Metaverse from Snowcrash or the VR Net from Otherland.

https://pawoo.net – Japanese Mastodon with 625,000 users and 45 million posts

The internet is a decentralized collection of autonomous systems held together by communities defining standards and protocols. Distributed social media maps very nicely onto the architecture of the internet and promotes freedom of expression and experimentation with new forms of media, social organization, and technology. Doubtlessly it will create new problems as well, like intensifying the internet hypernormalization effect. It will likely be some time before norms and robust community structures are figured out at scale. The ability for anyone to participate in the wider social community within bounds and parameters they set for themselves will be messier but ultimately more powerful and driven by more positive incentives than the current social media monopolists.

Links

If you want to give Mastodon a spin you can check out joinmastodon.org.

To learn more about ActivityPub read the spec (really, it’s not dry). More info about the types of Actors, Activities and Objects on ActivityPub can be found here.

PeerTube homepage explaining federation of video content.

Why does decentralization matter? Mastodon op-ed.

The-federation.info – statistics and open source fediverse projects.

Will America Ever Be A Socialist Country?

“America will never be a socialist country!” some exclaim these days. For the first time in a hundred years a viable presidential candidate proudly boasts of their democratic socialistic ideals and is welcomed by many. In fact there is already a great deal of wealth redistribution and government aid programs in America, though much of it is directed to the most well-off and well-connected.

Where does socialism fit in America?

In the mainstream the fact that America is capitalist is axiomatic. America is by definition capitalist and therefore anything done by its government or people is capitalistic. This is a useful piece of propaganda both for its boosters and haters in exactly the same way calling the USSR “socialist” was.

The USSR, or Union of Soviet Socialist Republics (worker’s council) was axiomatically socialist, indicative by the name, as is Democratic Republic of North Korea. The ideal of socialism, the scientifically determined natural end state of economic development, was always just around the corner for the Soviets. Domestically it was a positive term and something the government worked hard to convince its citizens it was the system they were living under, or about to be living under. For America it was a great way to tarnish the name of socialism by pointing out how cruel and impoverished their conditions were. Calling the USSR “socialist” was instrumental in propaganda both internally and externally and so it was encouraged to become the unchallenged categorization of the economic system in the Union of Soviets, even though by any objective measure it was in no way socialist. It was a centrally controlled oligarchy with almost no discretion or devolution of government given to local workers and those in the Party or at the center got to accumulate wealth and power. Common workers had little voice in how the means of production or the fruits of their labors should be democratically distributed, and those that had any such ideas in public could face grave consequences.

America is not the USSR by any stretch, however this propaganda exercise of always explicitly describing it as capitalistic and definitely not socialistic is one aspect that it shares in common with the Soviets. It is a market economy, sort of, sometimes. It does certainly seem to have Capitalists, the sort who in earlier eras would probably sport monocles and top hats instead of Teslas and favorable capital gains tax rates. It’s my opinion however that the US federal government does engage in tremendous wealth redistribution efforts and socialized risk and capital disbursements that would have made a Soviet or Maoist central planner feel uneasy.

Image result for america will never be a socialist

When American politicians or their media mouthpieces decry socialism, it’s a worthwhile exercise to ask for whom they mean. In the American context, the technical term “socialism” means “government giving money to the underserving,” i.e. the poors. More than almost any other country, the question of whether someone deserves government handouts is a central concern when considering what our hard-earned tax dollars should be spent on. In many parts of the world, the poor, disabled, young, elderly, disadvantaged and so on are considered deserving by default and the obscenely wealthy and powerful maybe not as much. Somehow, and I don’t claim to know exactly how (although I have some ideas), these determinants have been flipped in Capitalist America.

Again speaking in the media argot, “Socialism” in America is considered Bad because the money goes to the undeserving, like poor children. In our “market economy,” unimaginable quantities of money from taxpayers and future generations is hoovered up by the government and spat out at the inconceivably wealthy and well-connected. Even a moment’s reflection can easily illuminate this but you won’t find any mention of it as socialist in the press.

To give one current example of many, there is a bill far along in the U.S. Senate to modernize America’s nuclear arsenal. This is estimated to cost around $1 trillion over ten years, which likely means $3-4 trillion given the usual Pentagon overruns. Nowhere in the media will you find a discussion of the costs vs. benefits of the proposed spending. Citizens will not find themselves pondering the “how will we pay for it?” question, never mind asking whether America really needs all 4,000 nuclear warheads in the first place. This money will of course go directly into the pockets of the wealthiest corporations and individuals to be spent on gold-plated hardware that hopefully will never be used. When a politician makes the suggestion that we should have some money spent on food stamps or health care suddenly everyone’s breaking out the eye shades and Excel to carefully examine these socialist policies.

Image result for accountant visor

“Where will the money come from for the F-35 program?” has never been posed in a debate. Rarely is this wealth redistribution upwards considered socialist, much less challenged for its basic assumptions about needs. But when someone needs to eat food, they get a vastly more stringent auditing of their finances than the president does.

At the same time a bill is passed to dole out $867 billion to (mostly) giant farm conglomerates to farm, or not, the Trump administration takes action to increase work requirements for food stamps because we don’t want socialism. The people who want to eat might not be deserving, so why give them handouts? However while making food stamps less funded and harder to access, no such limitations or work requirements were imposed on the colossal agribusinesses receiving more free government money.

The now $1 trillion budget deficit under Trump is somehow growing, all while programs for basic needs are being cut. A decent new part of the deficit is the giant tax breaks given to the wealthiest Americans and even greater military spending. The tax cuts prompted some of the largest stock buybacks in history, helpfully boosting stock prices but likely damaging everyone else in the process.

However none of these issues are new or unique to the Trump administration. Federal intervention in the marketplace and socialization of risk was of comparable scope under Obama. Thanks to state aid, pressure, and financing, entities such as AIG, Caterpillar, foreign-owned banks, and General Motors were explicitly and directly supported by the state, not to mention the $800 billion bailout of some of his biggest campaign funders, namely Wall Street.

Whether or not this was good policy is subjective, but we can’t just call it pure capitalism when the Federal government owns shares in carmakers. But when China engages in state aid of its industrial champions (like nearly every modern rich economy did at some point) we threaten a trade war and incapacitate the WTO. Again to be clear, China’s interventions are on a much grander and explicit scale and the U.S. economy isn’t equivalent to China’s. Part of what I’m saying is maybe we shouldn’t think in such black and white terms of either/or Socialist or Capitalist, and most people dunking on China for unfair trade practices conveniently forget about similar U.S. practices. Except the Libertarians who are consistent at least.

Policies of China enacted on a grand scale are instructive by nature. Under Mao’s idiotic backyard blast furnace program and state-directed agriculture tens of millions of people died unnecessarily of hunger in one of the worst atrocities in history. After Deng Xiaoping introduced some market-based reforms, more people than ever before in history were lifted out of poverty in an incredibly short time. Hopefully we can learn something from this experience. Markets should be the default solution for economic organization because the vast complexity of interactions and incentives of billions of people is impossible for humans to comprehend or organize centrally, but this principle can be abused for propaganda purposes and intrude into realms where it is inappropriate. Most importantly, markets are not the solution to everything. Some problems do not have a market-based solution. Questions of basic education, law and justice, health care, and public infrastructure do not have market solutions.

In the most recent issue of the Economist, Bernie Sanders is called out for having a foolish plan to erase all student debt and the more “reasonable” plans of candidates Joe Biden and Mayo Pete are lauded for their more measured approach. Again, how could it possibly be paid for? Where would the money come from? Education isn’t free. The U.S. federal government has backed loans from private educational institutions that fail to teach while incurring sometimes hundreds of thousands of dollars in debt with atypically high drop-out rates. Once again, socialism is directed towards for-profit universities, fully subsidized and encouraged by the government. Again socialism for the rich, but the students get loaded with stupefying amounts of debt before they even begin working, putting a massive damper on earnings potential.

This country has socialism for the rich, and rugged individualism for the poor.”

Martin Luther King, 1968

It doesn’t have to be like this. In Ukraine or Poland, a student can attend a five-year university for free. Those universities may not top world rankings but imagine what they could do if they had an economy the size of America’s. If people can get a master’s degree in a technical program for free in these countries, what’s stopping it from happening in America? As another comparison, when I had temporary residence in the Czech Republic, another poor post-soviet country, I was able to get fully covered health care for $100 per month. That covered absolutely any health-related procedure or visit that would ever need to be performed, and probably only cost that much because I was a foreigner. The quality of care may not be as good as in America but it’s a vastly poorer country. Why can’t we have these things in America? In any other country we would apply the term “corruption”, but in America we say it’s a “lack of political will.”

Sure America spends vast sums of money on health insurance and education, but the money doesn’t actually reach patients and students. It goes to vast corrupt institutions of well-connected plutocrats and middlemen who spend the money on bloated administrations and unnecessary layers of bureaucracy. Libertarian institutes like Cato argue that poor people have never had it better than today, because if you actually add in transfers, we spent tons of money on them! Well that is not factually incorrect as we do spend vast amounts of money on health insurance and educational loans and SNAP and all the rest, but the money isn’t benefiting the supposed recipients. A lot of it is eaten up by middlemen like health insurance companies and for-profit university shareholders. Again the money is funnelled to the donor classes in what we would call corruption in any other country, and the people on the bottom have to fight against constant introductions of means-testing, cuts and more and more soul-crushing bureaucracies.

Consider how much money is wasted on health insurance. Not health care, but the insurance part of it. Even small practices have specialist employees who deal with billing insurance companies, faxing paperwork and arguing over reimbursements. Students can now major in billing health insurance companies (it’s complicated), a sign that maybe resources aren’t being spent in the best way here. The Mercatus Center, a Koch-backed libertarian outfit that is absolutely not supportive of Medicare For All released a report fretting about the price tag for the program. According to the report, government expenditures would be increased to pay all health-care expenses increasing the share of spending by the federal government, but overall expenditures for human beings would decrease, by about $2 trillion over ten years. Again the scale of graft and waste in the existing system is staggering, and I think this savings estimate is on the low side, considering the source of the research and support. This is of course just focusing on the economics and ignoring the large social costs and moral questions.

This isn’t to say that fiscal responsibility should be abandoned. We should have conversations about how to fund programs we as a society deem important. We should be cautious about borrowing against future generations to pay for our expenses today and we ought to be sure that our government is spending wisely with the money it does have. I believe that there are purely economic arguments to be made in favor of single-payer health care and universal free education, not to mention moral arguments, but that isn’t really the point I’m trying to drive home here. The point is that the media and politicians in this country have an endless supply of concern for fiscal rectitude when it comes to supplying Americans with basic amenities like food, shelter, and medical care but this intense scrutiny suddenly evaporates when we’re looking at splashing out trillions of dollars for tax cuts for wealthy individuals and corporations, or military hardware, or any other of the innumerable transfers of money from the taxpayers and future generations to elites.

Absolute cretins would have you believe that market forces would be a more efficient way of distributing health care, contrary to the experience in most parts of the developed world. The concept that people are informed consumers when it comes to health care is laughable. You don’t comparison shop MRIs, you aren’t going to check the reviews of hospitals while you’re having a heart attack, nobody makes a choice to get old or be born with health issues. Most people aren’t qualified to make rational choices about comparative cancer treatment costs and don’t want to spend time figuring out how to work in their overpriced insulin into their budgets. Only someone with a chronic case of Nudge-brain could possibly think healthcare should be treated as a marketplace. Even one of the most laissez-faire capitalistic governments on the planet, Hong Kong, has free at point of service universal hospital care and one of the highest life expectancy rates at 84 years versus the USA’s 78 years and dropping. And expanding insurance to be universal doesn’t make any sort of logical sense. As I wrote before, what’s the point of insurance if it’s subsidized by the government, you aren’t allowed to price risk (i.e. pre-existing conditions), and it’s universal? Why have the insurance layer there in the first place then? Corruption, or socialism for health insurance and pharmaceutical companies.

Image result for bernie no longer asking

America is certainly fairly capitalist, but also fairly socialist in a big way already. Vast, unimaginable quantities of cash are borrowed and taxed and funnelled to the ultra-wealthy donor and plutocrat classes. If we’re going to ask hard questions about budgets and costs, maybe we should start with looking at the welfare programs for the rich instead of basic health care or education.

Studying Russian as a Foreign Language

Click here to read the Russian version of this article.

I started studying the Russian language in university for a short time, then picked up formally studying it again since living and spending a lot of time in Eastern Europe. I love languages and find them super fascinating, especially syntax and orthography. For whatever reason these things are greatly interesting to me and fun to learn about. Like all nice theoretical studies, actual practical applications and reality dampen the fun somewhat.

To study practical everyday Russian is to study exceptions and unnecessary complexity. The language it most resembles for me is Latin both grammatically and in terms of the extra work you have to do that just seems somehow extra or not adding much.

When a student begins to study the Russian language, the first word they encounter as in every other language study is “hello!” In Russian this word is “Здравствуйте!” (Zdrastvuyte!) Which is five or six syllables depending how you count that preliminary imposing consonant cluster with a rolled “R”. The student immediately gets the sense that this isn’t going to be so easy. A first impression is that some key vowels seem to be missing from most words. How does one say “zdr”? From here it goes downhill.

When you want to say you are going somewhere in Russian it’s not enough to express the concept of travel, but it’s almost required to be accompanied by information about the mode of travel. There are words for specifying walking, going by means of wheeled conveyance, sailing, flying. Unlike many other languages, not only should the mode be specified but each verb consists of a unidirectional/multidirectional pair that must be discriminated (“I went to the store on foot end of story” vs “I went on foot to the store and then somewhere else”) as well as an imperfective/perfective pair that forces the speaker to consider if the action was completed or ongoing (“I walked” vs “I was walking”). On top of having to pick the right verb for the transport modality/perfective aspect/directionality combination one also should often prefix the verb to indicate if the motion is into, around, out of, on top of, under, through, out from, up to but not inside, and so on.

Along with learning the verbs of motion another fundamental aspect of the language is the noun inflection system. There are three genders, two numbers, and six main cases. Being an Indo-European language this system roughly similar to Latin, German, Old English and lots of other languages in this family.

While Russian’s case system is not particularly unique there is the case of the particularly unpleasant Russian genitive. There are a couple of historical reasons for this; the Proto-Indo-European ablative case was folded into the modern genitive case, and the genitive is used for expressions of quantity which still retain an archaic dual number form used for numbers of things between two and four. The abessive construction (“there is no book”) also uses the genitive form. The formation of the genitive plural is famously complicated and difficult and as a student of Russian you devote considerable time and practice to this very common and challenging puzzle.

As if that wasn’t enough to keep straight, the genitive form (but not construction) comes into play when a direct object has a soul, though only if it is a masculine noun. When declining a masculine animate direct object one uses the genitive form not the accusative.

The declension of numerals in Russian is its own special challenge which is hard to convey in English but involves a lot of genitive cases. You can look at the Wikipedia section to get a little taste though.

Most numbers ending with “1” (in any gender: оди́н, одна́, одно́) require Nominative singular for a noun: два́дцать одна́ маши́на (21 cars), сто пятьдеся́т оди́н челове́к (151 people). Most numbers ending with “2”, “3”, “4” (два/две, три, четы́ре) require Genitive singular: три соба́ки (3 dogs), со́рок два окна́ (42 windows). All other numbers (including 0 and those ending with it) require Genitive plural: пять я́блок (5 apples), де́сять рубле́й (10 rubles). Genitive plural is also used for numbers ending with 11 to 14 and with inexact numerals: сто оди́ннадцать ме́тров (111 meters); мно́го домо́в (many houses). Nominative plural is used only without numerals: э́ти дома́ (these houses); cf. три до́ма (3 houses; G. sg.). These rules apply only for integer numbers. For rational numbers see below.

https://en.wikipedia.org/wiki/Russian_declension#Numerals

The Russian writing and sound system isn’t so hard in my opinion for native English speakers, once you get used to the Cyrillic alphabet. There are some things that stand out as distinct compared to English however.

The first is the letter <Ы> which one pronounces a bit like the “i” in “bill”, but pronounced much lower in your throat. There is a distinction made between this sound and “И” (sounds like “ee”) although when the two sounds are spoken it’s damn hard to hear the difference normally. Also important distinctions are made between voiced and unvoiced “sh” and palletized consonants, which again I think are extremely difficult for a native English speaker to pick up on or enunciate without a great deal of practice. There is a letter “Ё” (“yo”) which is often just written as “E” (“ye”) which a Russian student just has to accept and assume they are being tricked sometimes. Finally there are the unspoken letters “Ь” and “Ъ” which are called soft sign and hard sign. The soft sign palletizes the letter in front of it and the hard sign mostly doesn’t do anything (occasionally depalletizing or demarcating a stop), and used to be extremely common in writing until the Bolsheviks ruthlessly purged it almost entirely from the language.

The Cyrillic alphabet is named after St. Cyrill who did not invent it but did invent an alphabet called Glagolitic (“speaking”) of which a few letters were taken for the Cyrillic alphabet, though aside from those characters Cyrillic is mostly cribbed from the Greek alphabet.

One of the easiest things to learn in Russian is the conjugation of verbs. In fact there is only really one set of personal endings you have to learn (well, two conjugations but they’re the same pattern). On top of this there are only three tenses to contend with (past, present, and future). The past is formed by adding gender markers (a standard masculine/feminine/neuter or one shared plural suffix) to the root, which is refreshingly simple although it seems strange that the gender is relevant for the conjugation of verbs and only in the past. Participles do exist but as far as I can tell they are basically never encountered and only as sort of a final footnote in most textbooks.

The only real complication with verbs is the previously mentioned perfective/imperfective aspect and prefixes. Verbs can have things stuck on the front to indicate if it is perfect or imperfect, and a future tense is simply formed by using a perfective form in what would normally look like the present, but since you can’t have a perfect action in the present, it means it’s the future instead. A bit odd but does make a certain kind of sense when you think about it.

Finally one last notable difference between English and Russian is swearing. The level of expressivity and creativity that appears to go into swearing in Russian is on another level, and probably beyond any sort of deep comprehension to a non-native speaker. Swear words also have vastly more force in Russian, and would almost never be uttered in any sort of semi-polite company and some probably would get you ejected from respectable society.

I am unable to really grasp the subtleties and complexity of the different prefixed and suffixed versions of “fuck” in Russian, and Google Translate doesn’t even try.

Studying Russian is a challenge, but getting all of the grammar right is really not the most important thing for daily usage. Even if you screw up most of the inflections people can still understand your meaning, and even native speakers get things wrong frequently. For me the hardest part is not speaking Russian but understanding what people say, mostly due to my unimpressive vocabulary and the speed in which people speak, but that can be true in any language and isn’t specific at all to Russian.

Русский – Изучать Как Иностранный Язык

Click here to read the English version of this article.

Изучать русский язык – изучать исключения и ненужную сложность.

Когда студент начинает учить язык, первое слово – «Здравствуйте!»

И сразу же он знает, что не будет легко. Чтобы просто сказать “hello” нужно пять слогов, и постоянно хочется найти недостающие гласные, как будто их не хватает. Как произносить “здр-”? Это очень пугающее начало, и дальше только труднее. 

Например, когда человек хочет сказать о идти куда-то, ему нужно учитывать или “идти” это одно место или много мест – ”идти” или ходить – пешком или на колесном транспорте (ехать/ездить), “в” или “под” или “у” или “из” (уйти, выйти, прийти, …), и если совершенный или несовершенный вид (“пойти”), или на самолёте или на корабле.

После того как студент выучил “идти”, он может выучить как использовать существительные “на каждый день”. В отличие от английского, у русских существительных есть падежи. Есть простые правила, как например винительный падеж – ”я читаю книгу.”  Окей, есть три рода, и это не так трудно, например окончания слов в мужском и среднем родах похожи, но когда студент изучает дальше, у него возникают сложности побольше .

Родительный падеж (кого?/чего?) – проклятие каждого изучающего русский язык. Родительный падеж был аблатив падежом давным давно, но сейчас они оба – только родительный. Поэтому, он используется  во многих случаях. Когда чего-то нет, родительный падеж – ”нет книги.” Когда считаете, родительный падеж – ”12 стульев.”  Образовывать множественные существительные в родительном падеже – возможно самое сложное в русском языке. Студенты русского практикуют это часто.

Форма родительного падежа тоже появляется когда используется одушевлённый объект в винительном падеже (только если мужской) – ”Я вижу коня”, а когда неодушевлённый объект – “Я вижу дом”, и тоже самое с числами два-четыре – ”Я вижу три вилки.”

Использовать числа тоже очень сложно. Они склоняются конечно, а также сохраняют свои архаичные двойсвестние формы из старославянского языка для чисел 2-4. Когда несчастный студент хочет сказать о количествах, ему обязательно нужно думать или число заканчивается на один, два, три, четыре, или больше.

Один стул – (форма именительного падежа)

Два/три/четыре стула – (форма родительного падежа – двойсвестный)

Пять и т.д. стульев – (форм множественного родительного)

Дальше, «двенадцать стульев» но «двадцать один стул». Ой вей.

И если бы хотел говорить о собирательных числительных, также используют  множественный родительный даже для чисел 2-4: «двое друзей». Смотрите ещё  больше сложностей здесь

Русская орфография и фонология менее трудна, немного исключений для носителей английского языка. Буквы <Ы> (делается из «ъ + і») и её звука /ɨ/ нет в английском языке. Трудно произносить, и слышать разницу между <И> и <Ы>, так же как и между <Ш> и <Щ>. Мягкий знак <Ь> и твёрдый знак <Ъ> просто сбивают с толку. А буква <Ё> часто пишется  как <Е>, интересно, что людей которые пишут её правильно называют ёфикаторами.

Кириллица – алфавит назван в честь Кирилла и Мефодия главным образом состоит из Греческого алфавита, с немногими символами из Глаголицы «Аз Буки Веди» представляющими звуки, которых нет в Греческом алфавите (как <Ш>). Носители русского языка не верят мне, когда я говорю русский алфавит – это почти всё из Греческого алфавита. Ещё я покажу эту таблицу.

Некоторые вещи легче в русском языке чем у большинства индоевропейских языков. Например глаголы. Спряжение глагола обычно регулярное, и только одна основная форма, это легко запомнить.

Так же глаголов только три времени (прошлое, настоящее, будущее), реже используемые деепричастия, а наиболее совершенные глаголы формируются  префиксами (писать ⇾ написать).

А много слов таких же  как и в романском, немецком, латинском и английском языках. Немного примеров:

Есть – как “est” на латином (to be)

Ярмарка – как “jahr” на немецком

Проект – как “projekt” на немецком

Бровь – “brow”

Бить – “beat”

Быть – “be” 

Ложь – “lie”

Носок – “sock”

Щека – “cheek”

Пламя, плыть, плот, полёт – (flame, fleet, float, flight) – общий индоевропейский сдвиг из /ф/ от /п/ (c.f. Lat. “pater” ⇾ “father”)

О мате, я знаю мало. Не понимаю русские ругательства – это так сложно. В английском языке ругательства не очень сильные или оскорбительные, но люди советуют мне не говорить эти слова на русском. На самом деле в английской Википедии – ”In modern Russia, the use of mat is censored in the media and the use of mat in public constitutes petty hooliganism, a form of disorderly conduct, punishable under article 20.1.1 of the Offences Code of Russia.” Мало информации в английской Википедии о мате, но в русской Википедии – длинная статья, которую я не могу понять и Гугл Перевод не помогает.

Изучать русский язык – это приключение и довольно не быстро. С практикой, возможно обращаться к людям, но сложнее понимать когда они отвечают. Если я читаю со словарём  или кто-то говорит медленно, я могу понять много. Говори со мной и помогай мне учить!

Lambda Function To Route Twilio Incoming Phone Calls

The absolute fastest way to create a serverless Twilio incoming call webhook Lambda:

npm install -g serverless
sls install --url https://github.com/revmischa/slspy --name twilcall
echo "twilio" > requirements.txt

handler.py:

from twilio.twiml.voice_response import VoiceResponse
from urllib.parse import parse_qsl


def hello(event, context):
    call = dict(parse_qsl(event['body']))

    resp = VoiceResponse()
    resp.say("hello world!", voice='alice')

    response = {
        "headers": {"content-type": "text/xml"},
        "statusCode": 200,
        "body": str(resp),
    }

    return response

Deploy, then set webhook handler for your phone number:

sls deploy

That’s all!

Customize your response with TwiML.

See this gist to check out more advanced handling with loading the Twilio API key from Secrets Manager, doing lookups with Twilio Add-Ons to detect spam/robocalling, and detailed caller lookup info that is output to a slack channel every call.

Making Use Of AWS Secrets Manager

One of the many new services re-invented at AWS’s re:invent conference was the storage of secrets for applications. Secrets in essence are generally things your application may need to run but you don’t really want to put in source control. Things like API keys, password salt, database connection strings and the like.

Current ways of providing secrets to applications are things like configuration files deployed separately, Heroku’s config which exposes them as environment variables, HashiCorp Vault, CloudFormation variables, and plenty of other solutions. AWS already had a dedicated mechanism for storing secrets and making them available via an API call in it’s SSM Parameter Store service, in which you could store values, with optional encryption.

AWS Secrets Manager is slightly but not very different from SSM Parameter Store; it adds secret rotation capabilities and isn’t as buried deep down inside the obliquely named “AWS Systems Manager” service. I think those are the main features. Oh it also lets you package a set of key/value pairs into one secret, whereas Param Store makes you create separate values that require individual API calls to retrieve.


Using Secrets With Serverless

To store encrypted secrets in the AWS Secrets Manager and make them available to your serverless application, you need to do the following:

  • Create a secret in Secrets Manager.  Select “Other type of secrets” unless you are storing database connection info, in which case click one of those buttons instead.
  • Select an encryption key to use. Probably best to create a key per application/stage.
  • Create key/value pairs for your secrets.
  • After creating the secret, there will be sample code for different languages that shows you how to read it in in your application.

Encryption Key Access

To create encryption keys, look in IAM (click “Encryption Keys on the bottom left in IAM) and assign admins and users of the keys. I manually assign the lambda role that’s been created as a user of the encryption key it needs access to in the console. There may be a way to automate this but it feels like an appropriate step to have some small manual intervention.

Your lambda’s role will look like $service-$stage-$region-lambdaRole

IAM Role

In addition to granting your lambda role access to decrypt your secret, you also need to grant it the ability to access your secret.:

plugins:
  - serverless-python-requirements
  - serverless-pseudo-parameters

provider:
  name: aws
  runtime: python3.7
  stage: ${opt:stage, 'dev'}  # default to dev if not specified on command-line

  iamRoleStatements:
    - Effect: Allow
      Action: secretsmanager:GetSecretValue
      Resource: arn:aws:secretsmanager:#{AWS::Region}:#{AWS::AccountId}:secret:myapp/${self:provider.stage}/*

  environment:
    LOAD_SECRETS: true

This is taken from my serverless.yml file. Note the secret:myapp/dev/* bit – it’s a good idea to use prefixes in your key names so that you can restrict access to resources by stage (i.e. dev only has access to dev secrets, prod only has access to prod secrets). In general prefixing AWS resources with application/stage identifiers is a crude but readable and simple way to create IAM policies for access groups of related resources.

serverless-pseudo-parameters is a handy plugin that lets you interpolate things like AWS::Region in your configuration so that you don’t need to do an unwieldy CloudFormation Fn::Join array to generate the ARN.

Reading Secrets In Your Application

Now that your application has access to GetSecretValue and is a user of the encryption key used to encrypt your secrets, you need to access the secrets in your application.

Sample code is provided in the Secrets Manager console to read your secret. One thing to note is that if you store key/value pairs in your secret, which is most likely what you’ll want to do almost all the time, you get it back as JSON. The sample code provided doesn’t decode it. The general idea is something like:

import boto3
import base64
import json


def get_secret(secret_name):
    """Fetch secret via boto3."""
    client = boto3.client(service_name='secretsmanager')
    get_secret_value_response = client.get_secret_value(SecretId=secret_name)

    if 'SecretString' in get_secret_value_response:
        secret = get_secret_value_response['SecretString']
        return json.loads(secret)
    else:
        decoded_binary_secret = base64.b64decode(get_secret_value_response['SecretBinary'])
        return decoded_binary_secret

And then you can do as you please with the secrets.

App Configuration

I like consuming secrets directly into my application’s configuration at startup. With Flask it looks something like this:

    if os.getenv('LOAD_SECRETS'):
        # fetch config secrets from Secrets Manager
        secret_name = app.config['SECRET_NAME']  # dev, prod
        secrets = get_secret(secret_name=secret_name)
        if secrets:
            log.debug(f"{len(secrets.keys())} secrets loaded")
            app.config.update(secrets)
        else:
            log.debug("Failed to load secrets")
  • First, only try to load secrets if we’re executing under serverless/lambda. LOAD_SECRETS is defined as true in the serverless.yml snippet above. This should not be true when running tests, as you don’t want tests to access secrets and you don’t want to pay for all the additional accesses anyhow.
  • SECRET_NAME is in application config and can be different for dev and prod.
  • app.config.update(secrets) adds whatever you stored in Secrets Manager to your app config.
  • I should probably make a dumb PyPI module to do this automatically.

Serverless Python API Development

In a previous article I discussed how to interact with the serverless AWS Lambda platform using only tools provided by Amazon. This was a valuable experiment that I suggest applying to any new technology or interesting new system you’d like to learn. Start with the basics and try doing a project without too many extra tools or abstractions so that you can get an idea of how the underlying system works and what’s unpleasant or boilerplate-y or requires too much effort. Once you have an idea of how the pieces fit together you can have a much better appreciation for the abstractions that go on top because you understand how they work, what problems they are solving, and what pain they are saving you from.

AWS services are powerful but generally need to be put together in coherent ways to achieve your goals. They’re modules that provide the functionality you need but still require some glue to make a nice developer experience. Fortunately because the entire platform is scriptable, software tools and additional layers of abstraction are rapidly increasing the capabilities of software engineers on their own to manage configuration without the need for any hardware or humans in between. CloudFormation (CF) allows declaration of your infrastructure with JSON or YAML. CF templates like the Serverless and CodeStar transforms make it easier to write less CloudFormation code to describe a serverless configuration. And then tools like the Serverless toolkit add another layer of automation on top of CF and provide a really excellent developer experience. Not to be outdone, Amazon provides an even higher level toolkit called Amplify (subject of a future article) to further increase the leverage of effort to available hardware and software muscle.

Serverless Toolkit

After going through the process of building some toy applications using AWS SAM and the Serverless CF transform, I quickly saw some of the drawbacks of not using a more advanced system to automate things:

  1. Viewing logs. Looking at CloudWatch logs in the AWS Console is not a great way to view the output of your application in real time, or in any time really. 
  2. It wasn’t clear to me how to save some pieces of a serverless application architecture for re-use in later projects. I posed a question to the Flask mailing list and IRC channel about how to make an extension based around it and didn’t get a useful response.
  3. Defining stuff like API gateways, S3 buckets for code, and domains in CF is tedious. It can be automated further.
  4. It would be nice to have some information readily available, such as what URL my application is deployed at.
  5. Deployments, including to different stages.
  6. Telling me when a deployment is finished, especially when using CodeStar.
  7. Invoking functions for testing and via automation.
  8. Managing dependencies.

And some other general stuff like keeping track of the correct AWS configuration profile and region. 

As happens so often in the field of Computers, I’m not the first one to encounter these issues and some other people have already solved most of the problem for me. 

To ensure a steady supply of confusion when discussing the relatively recent trend of serverless application architecture, there exists a collection of tools called Serverless, which resides on serverless.com. This should not be confused with serverless the adjective or the Serverless Application Model (SAM) or the AWS Serverless CF transform.

Every one of the issues mentioned above is simply handled by Serverless. I believe it’d be an unnecessary expenditure of time and effort to continue to develop serverless applications without it, based upon my recent experience trying to do so. Unless you’re just starting out and want to get a feel for the basics first, that is.

I won’t reiterate the Serverless quickstart here, go try it out yourself on their site. It takes very little effort, especially if you already have AWS credentials set up. I will instead talk about what advantages it gives you:

Logging

This is easy.  You can view (and tail) the logs for any function with

sls logs -f myfunction -t

Reusability

# immediately create a Flask app based on my template
sls install --url https://github.com/revmischa/serverless-flask --name myapp

Some of what people have been doing is going the same route as Create-React-App and creating templates for Serverless projects that can be accessed with “sls install.” On the one hand this does make it very easy to create and share reusable setups and allows for divergence as templates evolve, but it makes it much harder for projects started with older templates to incorporate new refinements. In the realm of Flask and Python, I don’t feel this problem is solved just by templates and some sort of python module that can co-evolve is needed. Something analogous to the react-scripts package that goes along with Create-React-App would likely be the way to go.

Configuration And CloudFormation

Now you declare your resources and functions in the serverless.yml configuration file, along with lots of other useful stuff

Nearly all of the boilerplate CF needed for serverless like a S3 bucket for code, IAM permissions for invoke and CloudWatch, API Gateway, etc are totally hidden from you and you never need to care about them. Only the minimum configuration and CF needed to describe what’s unique about your setup is required from you. On a scale of sendmail.conf to .emacs, serverless.yml is fairly high on the configuration file sublimity scale.

Info

This is easy. Where’d I park my domain again?

$ sls info
Service Information
service: myapp
stage: dev
region: eu-central-1
stack: myapp-dev
api keys:
  None
endpoints:
  ANY - https://di1baaanvc.execute-api.eu-central-1.amazonaws.com/dev
  ANY - https://di1baaanvc.execute-api.eu-central-1.amazonaws.com/dev/{proxy+}
  GET - https://di1baaanvc.execute-api.eu-central-1.amazonaws.com/dev/openapi
functions:
  app: myapp-dev-app
  openapi: myapp-dev-openapi
Serverless Domain Manager Summary
Domain Name
  myappmyapp.net
Distribution Domain Name
  dcwyw3gslhqw1.cloudfront.net

Deployment

This is easy too! Too easy!

$ sls deploy
$ sls deploy -s prod # specify stage

This bundles requirements if needed, packages the service, uploads to S3, and kicks off a CloudFormation stack update. 

Notice that sweet Serverless Domain Manager Summary section?
That, my friend, is the serverless-domain-manager plugin. If you want your endpoints to be deployed under a domain name you already have in a Route53 zone (and hopefully have an ACM certificate in us-west-1 to go with it) you can have Serverless automatically fire up the domain or subdomain for you along with a CloudFront distribution and API Gateway domain mapping.

I discovered an issue with the domain manager plugin selecting the ACM certificate for your domain at random among a list of matching domain names. This was picking an expired previous certificate, so I fixed it to filter out any unusable certificates. My PR was quickly and politely merged. Always a positive sign.

Waiting / Notifications

The aforementioned deploy command tells you when it’s done. Then you can test it out right away. You can speed it by only deploying a specific function, or using the S3 accelerate option to speed up uploading of your artifacts. Don’t waste time deploying stuff you don’t need or watching the CodeStar web UI.

Invoking Functions

AWS SAM is pretty easy, and so is Serverless. If developing a python webapp with the serverless-wsgi plugin, you can also serve your app up locally.

Managing Dependencies

(This part is python-specific)

How to manage dependencies for your python lambda? Well, just stick them in requirements.txt. Duh, right? With Serverless, more or less right. Remember that any dependencies have to be bundled in your lambda’s zip file. Need to build binary dependencies and not on a linux amd64 platform? Just add “dockerizePip: true” to the serverless-python-requirements plugin configuration in serverless.yml and you’re good to go.

Note that if invoking functions locally or starting the WSGI server, you still need a local virtualenv. One wacky non-Serverless template I looked at used pipenv instead to manage both local and lambda dependencies, but I couldn’t advise it; it’s pretty weaksauce.

Extending Serverless

Mostly what I’ve been doing with AWS Lambda is making small web API services using Python and the Flask microframework. With serverless providing exactly the tooling I need, I also want to be able to start new projects with a minimum of effort and have some pieces already in place that I can build on for my application.

I forked a serverless-flask template I found and started building on top of it. I made it not ask if you want to use python 2 or 3 (why not ask if I want UTF-8 or EBCDIC while you’re at it?) and defaulted dockerizing pip to false.

If building an API server in Flask, your life can be made much nicer with the addition of marshmallow to handle serializing and deserializing requests, flask-apispec to integrate marshmallow with OpenAPI (“swagger”) and Flask, and CORS. My version of the template includes all of this to make it as easy as humanly possible to make a documented serverless python REST API with the absolute minimal amount of effort and typing. And as a bonus it generates client libraries for your API from the OpenAPI definition in any language you desire.

Instructions for using the template and getting started quickly can be found here.

Serverless? Why Not

This article is a marker in the path our journey so far has taken us. Improving how we build applications and services is an ongoing process. Our previous milestone was unassisted AWS services, this present adventure was improved tooling for those services, and the next level to up may be AWS Amplify and GraphQL. Or maybe not. Stay tuned.