projectM: OpenGL and Shader Modernization

I wrote recently about the interesting work being done on the open source music visualizer projectM, and that progress is being made on modernizing the graphics capabilities of this library. Since it was first created, there have been changes made in the way OpenGL is to be used for widespread compatibility and support for GPU shader programs.

Now the efforts are in the home stretch; OpenGL ES support is close to complete, and there is support for handling a decent percentage of the shader programs found in the visualization presets. The vast majority of the work has been done (by @deltaoscarmike), and mostly bug fixing remains.

After setting up vertex buffers and attributes and reworking the texturing and sampling code, the final step for full modern compatibility was dealing with the HLSL shaders. MilkDrop is the visualizer for Milkdrop that projectM is built for compatibility with. It was made with DirectX and the shader language that it supported in presets is HLSL, the language used by DirectX. Since projectM is designed to be run on other systems besides Windows, some form of support for HLSL was needed. Previously it was handled by the Cg toolkit from nVIDIA, but this project has been long abandoned and it was an unpleasant and sizeable dependency. The only other solution was to convert the HLSL code into GLSL, the OpenGL shader language. Use is now being made of hlsltranslator to transpile HLSL to GLSL, which is no small feat. Additional support for a #define preprocessor directive needed to be added due to its liberal use in previous code and presets.

Default warp and composite fragment shaders were created, switching to the preset versions if they are present and compile without problems. The blur shader was reimplemented in GLSL, and some helper routines recreated in GLSL for the presets that expect things like computing angle or texture sampling. The combination of running preset shader code on the GPU with the recent intel SSE optimizations for CPU per-pixel math brings performance improvements as well.

The real amazing news is that projectM now builds on a raspberry pi. This is notable not just because of the popularity of them, including running media centers, but because if it builds on a raspberry pi it should (with some effort of course) eventually run anywhere. One of the biggest issues was that projectM could only be run on a desktop or laptop computer. But now by using modern graphics operations it can run on accelerated hardware on mobile and embedded devices via OpenGLES, opening up many new opportunities, and potentially making it much easier to re-integrate into downstream projects like Kodi and VLC. Running it on the web via Emscripten, compiling to wasm and using WebGL is doable again. It’s amazing what possibilities modernizing software can bring.

It should be mentioned that the vast majority of the OpenGL work was done by superstar deltaoscarmike who apparently really knows what he’s doing.

Some screenshots of the glsl branch:

 

The Internet Hypernormalization Effect

Tribalism, echo chambers, groupthink, fake news, splintering. More and more there is a vague awareness of the effect the internet has on shaping peoples’ perceptions of the world.

There is a modern phenomenon of internet hyper-normalization, in which individuals can stumble into communities of like-minded folk, with the eventual effect of distorting and warping reality. This has only been possible with the rise of the internet, as many types of communities were virtually un-assembleable in meatspace. With the need for human and physical social interaction diminishing every day thanks to remote jobs, Amazon, and more and more entertainment and daily life accessible online, these communities and the people in them get elevated to a seemingly representative view of the modern world and norms, mores, opinions, and culture.

Having spent a great deal of time on the internet, working at LiveJournal, chatting with depraved individuals on IRC, and rubbernecking the information superhighway accidents, I have some insight into this situation.

 

Do you know what furries are?

furry cable car
Furries go past my apartment on the California St. cable car

Furries are people that believe they have the souls of non-human animals, and act out this belief and fantasy by wearing anthropomorphic costumes, creating art, music, conferences and much more. This being the internet it goes without saying that there is an unbounded furry fetish pornography and distasteful works as well. The furry community could only exist with the internet. Why?

Imagine if you don’t have the internet. Would you ever know a furry community exists? Probably not. If you don’t know this community exists, your desire to join or create it is likely to be very small. If you do know the community exists, the likelihood you desire to join it grows proportionally with the extent of the community. Many people crave belonging to a group, any group. They want to have an identity. They are willing to change and adopt new customs to gain friends, and the undercurrent of fetishization implies a possibility of gaining sexual partners as well. Any established group, no matter its character, attracts members looking for something to belong to.

Now imagine if you don’t have the internet and you really are a furry, without ever knowing such a community exists. Say you think you’re a wolf. You want to howl at the moon. You want to hunt in packs with your wolf buddies, write wolf poetry, wear a wolf suit, yiff with other wolves. Well, how do you find other like-minded people? You have no option. Maybe you bring it up with your friends, or post a classified ad, or look for furry conferences. Imagine telling your peers you think you’re really a wolf. They’d tell you you’re a fucking goofball and ostracize you. Now consider if you have the internet. You can find others like you! You can share your drawings on deviantart and people post comments telling you how great it is. You can organize meetups and conferences and meet other furries. Suddenly, you have the possibility to connect with others and influence impressionable young people who happen to stumble into your community.

This person should face great disapprobation and social stigma.

 

More than just making new friends

If consenting adults want to all be insane goofballs together, who is to say it’s a bad thing? So what if the internet has now enabled groups of weird people to get together and do whatever bizarre shit they’re into together. What’s the problem?

When humans operate in a functioning society they are checked by ostracism, judgement, ability to make friends and meet romanic partners and approval of their peers. These constraints can have a positive effect on people, preventing them from all sorts of harmful behavior and enabling positive interaction. We have a public road system for example that we can all share more or less harmoniously, because if you decide to drive on the sidewalk you will face considerable opprobrium. We can enjoy shops and jobs and entertainment events and all kinds of great benefits to living in modern societies, because we all work together as a culture to delineate boundaries of acceptable behavior. If you want to beat your kids in public or play the bagpipes in an apartment building at 4am or try to go skiing on the mall escalator wearing nothing but a bathrobe, someone’s likely to say something. We all learn what is acceptable and unacceptable within our society because we learn the rules from our interpersonal interactions. But this now becomes less and less of a check for greater and greater numbers of people moving to internet communities for acceptance and socialization.

Much like cultures in different parts of the planet, many communities on the internet have very deep systems of interaction and discourse. Whether you’re in an IRC channel, message board, forum, LiveJournal community, twitter social group, youtube channel, facebook group, or tumbler cross-section, you may have your own norms, language, rules, values and beliefs. The danger that I have observed from this is that humans aren’t really wired to give these relatively tiny and insular communities the appropriate weight when extrapolating the rest of human society.

 

Internet Hypernormalization Effect

I wish there was a term for this but there isn’t, so I’ll make one up. Call it Internet Hypernormalization Effect. It’s when individuals join a small and insular community and lack perspective on its place in larger human society. Because humans are wired to learn rules about their society from the people they interact with on a daily basis, they end up incorrectly projecting the norms of their small social group onto humanity at large, resulting in a profoundly warped and distorted view of humanity.

I don’t believe this is something conscious people do, or really fully recognized by people who haven’t been trawling the stranger parts of the internet for most of a lifetime, but it’s there.

If you hang out with furries online, or juggalos, or people into Harry Potter fanfic, or vegans, or UFO hunters, or channers, or /r/thedonald, or any other insular online clique, over time you begin to ascribe their values and properties to an inappropriately large segment of the population. It’s impossible to maintain perspective on the world when you don’t get out, don’t socialize with people in your IRL (in real life) community, and primarily communicate with your group day in and day out. IHE spills over into the echo-chamber effect on social media that more and more researchers and political scientists warn about.

Typical group dynamics are magnified on the internet here; rejecting views that don’t comport with the group, praising ideas, links, media, thoughts, commentary that align with the group’s values, and lowering in status or even humanity the out-group.

If you’re really deep into the juggalo facebook group, you subconsciously think that maybe 20, 30% of humanity is a juggalo. Not consciously, but your brain gets this impression because it’s whom you primarily interact with. It’s your community. It distorts your perception of the world. Drinking faygo and wearing clown face paint is a normalized activity and without a proper sense of perspective lent by interaction with cross-sections of typical human beings. By eschewing normie gatherings in favor of online and juggalo gatherings, you are no longer constrained by browbeating and censure. This freedom certainly has its merits but with the freedom can come a serious ungrounding and reality distortion field.

 

<Programming>: A Peek At Academia

IMG_1747

This week the 2nd conference was held in Nice, France. The topic: the art, science, and engineering of programming.

It was my first brush with the world of serious academic computer scientists. I don’t have a degree myself, though I did study CS for a couple of years at USF before deciding work was more fun and interesting and lucrative. I was by far the least qualified person at the conference; most were Ph.D students or professors or postdocs. They all came from a world whose details I was hazy on, at best. It turned out that my world, the world of “industry” as they term it, also is something of a faraway land to them.

Throughout the workshops and presentations I was struck by a few things: how well-read and smart and knowledgable the participants were, the fun abstract and interesting nature of the problems they were solving, and the complete lack and regard of any (at least to me) practical applications of their work and research.

Partly due to being in an unfamiliar realm out of my depth on many topics, and maybe partly with a little bit of jealousy of the intellectual playground they get to spend their days in, I tried to keep an open mind about the talks and learn what I could. I did really want to know if there were applications of the problems they were solving outside the world of academic research into solving problems of other academic researchers, but I felt like it’d be improper to inquire. Let me give some concrete examples.

IMG_1743

Researchers at Samsung built a prototype for sending code to be executed on other devices and making use of their resources. One demo was showing a game being played on a phone, and then having the game display seamlessly transfer to another device (that could be a smart TV, in theory). Pretty neat!

Well for one, this was for Tizen, an operating system that exists purely as a bargaining chip for Samsung and a backup strategy to not be solely dependent on Android. So there is no real-world application for this to run on any real devices. Furthermore, giving other devices the ability to make use of Tizen device resources is a huge avenue for security problems, as the presenter readily acknowledged. When combined with the fact that Tizen has more holes than swiss cheese this is doubly worrying. Additionally, one of the major unsolved huge problems with IoT (besides security) is interoperability between devices of different manufacturers. When I asked if there was any interest or plan to submit their work to a standards track, the presenter and host of the session got very confused.

IMG_1741

Another talk was a study of running safe C/C++/Fortran code. It included implementations to provide safety of memory management, bounds checking, variadic arguments length checking, use-after-free, and double-free errors. Awesome! Fantastic! Just one catch – it’s only for running said code on the JVM. Since people don’t usually run C++ code on the JVM, this is of limited use, except possibly for tooling for people running Ruby on the JVM, one attendee told me. The talk and the paper had nothing to say on the performance overhead. Research was sponsored by Oracle Labs.

Actually, a vast amount of the research topics were relating to Java and the JVM, a situation I found scandalous. I had hoped and even assumed that academics would be proponents of Free Software, because of the massive contributions to learning, understanding, and implementation, while being unencumbered by profit-driven abuses of the legal system to the determent of progress. And yet they all live and breathe Java! Java is of course NOT free (not as in free beer, but as in libre – a French word meaning “Richard Stallman”), as was recently affirmed by a US district court which found Google liable for potentially 8 or 9 billion dollars for writing header files mimicking an interface of Java. Java is probably the least free language out there now.

In the first workshop I went to there was a session where we would all get to play with a new attribute-based grammar to compose a basic C language parser. Cool! But it was all done in Java. I said “sorry, I don’t have a JDK” and the entire room burst out laughing. “Who the fuck uses JAVA?” I asked, incredulous. “Uh, everybody!” came the smart-ass reply. Since there was no functioning internet at the conference venue, I couldn’t download the JDK, so I left. What a sad state of affairs for academia, to be so beholden to the most evil corporation in software today.

Research was presented on improving the efficiency of parsing ambiguities resulting from deep priority conflicts. An interesting and thoughtful study of helping compilers do a better job of catching a certain type of ambiguity and resolving it in an optimal fashion. They applied their analysis to 10,000 Java files on GitHub and 3,000 OCaml files, and found three conflicts in two Java files, but a great many in the OCaml source files.

Screen Shot 2018-04-15 at 16.15.54.png

So for all the folks out there doing serious work with OCaml, you’re in luck!

IMG_1750
My favorite talk and the winner of an award at the conference was simply about Lisp, Jazz, and Aikido. And how they’re all cool and similar.

IMG_1758

Another enjoyable and award-winning talk was about how academics talk about Monads. Whether they are railroad tracks, boxes, or sometimes burritos.

IMG_1762

One of the student research projects sounded at first like it might be getting dangerously close to some sort of potentially useful application one day. One student talked about his system for dynamic access control for database applications. Unfortunately it requires using a contract language of his own devising in a lisp-lisp environment.


Don’t get me wrong, I enjoyed the conference and was frankly intimidated by all the super smart folks there. But it left me with a feel that so much talent, brains and time was being spent solving problems purely for the benefit and respect of other academics, instead of trying to solve serious problems facing us vulgārēs who have deadlines and business objectives and real-world problems to solve. The best part of the conference by far was just talking to the people there. I had a lot of interesting and thoughtful conversations. The research, eh.

IMG_1756

Travel: Amsterdam

Amsterdam’s a fine place; good cheese, markets, art, bicycles, weed, idyllic empty parks.

Seems like it’d be a chill place to live, but quite pricey.

Hemp, Marijuana, and Hash Museum

First day there I nearly got run down by a dozen bicycles, not realizing you have to look both ways before crossing the sidewalk.

Rembrandt’s House

 

projectM Music Visualizer Status Update

As I’ve ended up with de facto maintainership of the illustrious projectM open source music visualizer I’ve seen a fair bit of interest in the project. I think I at least owe a blog post to update folks on where it’s at, what needs working on, and how to help make it better.

 

What is projectM?

projectM is a music visualizer program. In short it makes cool animations that are synchronized and reactive to any music input. I say music and not audio because it includes beat detection for making interesting things happen on the beat.

Screen Shot 2014-08-25 at 12.31.07 AM

History

Some of you may remember the old windows mp3 player WinAmp. It contained a supremely amazing and innovative music visualizer called Milkdrop written by a gentleman from nVidia named Ryan Geiss, known just as Geiss. The visualizer was not a single set of rules for visualizing audio but rather a mathematical interpreter that would read in “preset” files which were sets of equations. You can read the very illuminating description here of how the files are defined if you’re interested. In short there is a set of per-frame equations describing colors and FFT waveforms and simple transformations, and there is a set of per-vertex equations for more detailed transformations and deformations.

Due to the popularity of WinAmp and Milkdrop there have been many thousands of presets authored and shared with really stunning and innovative visual effects ranging from animated fractals to dancing stick figures to bizarre abstract soups. The files are often named things like:

  • shifter – cellular_Phat_YAK_Infusion_v2.milk
  • [dylan] cube in a room -no effects – code is very messy nz+ finally some serious stfu (loavthe).milk
  • NeW Adam Master Mashup FX 2 Zylot – In death there is life (Dancing Lights mix)+ Tumbling Cubes 3d.milk
  • suksma + aderassi geiss – the sick assumptions you make about my car [shifter’s esc shader] nz+.milk
  • flexi + cope – i blew you a soap bubble now what – feel the projection you are, connected to it all nz+ wrepwrimindloss w8.milk

And so on.

Screen Shot 2014-07-18 at 2.15.36 PM

As I understand it, possibly incorrectly, there were two major problems with Milkdrop. First that it was implemented with DirectX, win32 APIs and assembler, and secondly that it was not open source (though it was made open source fairly recently). So some enterprising folks in 2003 created projectM as an open source reimplementation that would be Milkdrop preset-compatible.

I didn’t work on projectM originally and I am not responsible for the vast majority of it. However the previous authors and contributors have for whatever reason mostly abandoned the project so it was left to random people to make it work. The code is quite old although the core Milkdrop preset parsing, beat detection, most of the OpenGL (more on that later) calls, and rendering is in fine shape. projectM is really just a library though, designed to be used by applications. In the past there have been XMMS and VLC plugins, a Qt application, pulseaudio and jack-based applications, and more.

 

OSX iTunes Plugin

Not really having a good solution for OSX I went ahead and ported the ancient iTunes visualizer code to work on a then-modern version of iTunes and voila! projectM on OSX. Though I did have to deal with the very unfortunate Objective-C++ “language” to make it work. Not Objective-C, Objective-C++. No I didn’t know that existed either.

Screen Shot 2014-08-25 at 12.33.50 AM

I tried to submit the plugin to the Mac App store as a free download. Not to make money or anything, just to make it easy for people to get it. The unpleasantness of this experience with Apple and their rejection is actually what spurred me to start this blog so I could complain about it.

Much to my, and apparently a number of other people’s dismay, a very recent version of iTunes or macOS caused the iTunes visualizer to stop working as well as it did. It appears to be related to drawing and subviews in the plugin.

 

Cross-Platform Standalone Application

I decided that what would be better is a cross-platform standalone application that simply listens to audio input and visualizes it. This dream was made possible by a very recent addition to the venerable cross-platform libsdl2 media library adding support for audio capture. I quickly hacked together a passable but very basic SDL2-based application that runs on Linux and macOS and in theory windows and other platforms as well. Some work needs to be done to add key commands, text overlays (preset name, help, etc), better fullscreen support and easy selection of which audio input device to use.

The main application code demonstrates how simple libprojectM is to use. All one must do is set up an OpenGL rendering context, set some configuration settings, and start feeding in audio PCM data to the projectM instance. It automatically performs beat detection and drawing to the current OpenGL context. It’s really ideal for being integrated into other applications and I hope people continue to do so.

Screen Shot 2018-02-18 at 20.49.03.png

You can obtain source, OSX and linux builds from the releases page. This is super crappy and experimental and needed some configuration tuning to make it look good, and you need to drop the presets folder in. But it’s a start.

Build System

In their infinite wisdom the original authors chose the cmake build system. After wasting many hours of my life I will not get back and almost giving up on the software profession altogether I decided it would be easier to switch to GNU autotools, the same build system almost all other open source projects use, than to deal with cmake’s bullshit. So now it uses autotools (aka the “./configure && make && make install” system everyone knows and loves).

 

Needed Efforts

This is where you come in. If you like music visualizers and want to help the software achieve greater things there is some work to be done modernizing it.

The most important task by far is getting rid of the OpenGL immediate-mode calls and replacing them with vertex buffer object instructions. VBO is a “new” (not new at all) way of doing things that involves creating a chunk of memory containing vertices and pushing it to the GPU so it can decide how and when to render your triangles. The old-school way was “immediate mode” where you would tell OpenGL things like glBegin(GL_QUADS) (“I’m going to give you a sequence of vertices for quadrilaterals”) and give it vertices one at a time. This is tremendously inefficient and slow so it isn’t supported on the newer OpenGL ES which is what any embedded device (like a phone or raspberry pi) supports, as well as WebGL.

I believe that projectM would be most awesome as a hardware device with an audio input and an HDMI output, but making a reasonably-sized and -priced solution would mean using an embedded device. It would be great to have a web application (I attempted to do this with Emscripten, a JavaScript backend for llvm) but that requires WebGL. Having an open source app for Android and iOS would be amazing. All of this requires the small number of existing immediate-mode calls to be updated to use VBOs instead. Somebody who knows more about this stuff or has more time than me should do it. There aren’t a lot of places in the code where they are used; see this document.

Astute readers may note that there already are iOS and Android projectM apps. They are made by one of the old developers who has made the decision to not share his modern OpenGL modifications with the project because he makes money off of them.

rms3.png
why the fuck hoarders don’t share their code back

Another similar effort is to replace the very old dependency on the nVidia Cg framework for enabling shaders. Cg was used because it matches Directx’s shader syntax. GLSL, the standard OpenGL shader language is not the same, and requires manual conversion of the shaders in each preset.

The Cg framework has been deprecated and unsupported for many years and work needs to be done to use the built-in GLSL compilation calls instead of Cg and convert the preset shaders. I already did some work on this but it’s far from finished.

 

The Community

The reason I’m writing this blog post is because of the community interest in the project. People do send pull requests and file issues, and we definitely could use more folks involved. I am busy with work and can’t spend time on it right now but I’m more than happy to guide and help out anyone wishing to contribute. We got an official IRC channel on irc.freenode.net #projectm so feel free to hang around there and ask any questions you have. Or just start making changes and send PRs.