A lovely seaside town. Excellent weather, strange hamburgers, Roman ruins. It’s on the coast of Croatia, just across from Italy. A popular destination for Eastern European tourists. Surrounded by insanely huge cliffs.
The main touristic area is inside and around the place of Diocletian, a Roman Emperor.
This week the 2nd conference was held in Nice, France. The topic: the art, science, and engineering of programming.
It was my first brush with the world of serious academic computer scientists. I don’t have a degree myself, though I did study CS for a couple of years at USF before deciding work was more fun and interesting and lucrative. I was by far the least qualified person at the conference; most were Ph.D students or professors or postdocs. They all came from a world whose details I was hazy on, at best. It turned out that my world, the world of “industry” as they term it, also is something of a faraway land to them.
Throughout the workshops and presentations I was struck by a few things: how well-read and smart and knowledgable the participants were, the fun abstract and interesting nature of the problems they were solving, and the complete lack and regard of any (at least to me) practical applications of their work and research.
Partly due to being in an unfamiliar realm out of my depth on many topics, and maybe partly with a little bit of jealousy of the intellectual playground they get to spend their days in, I tried to keep an open mind about the talks and learn what I could. I did really want to know if there were applications of the problems they were solving outside the world of academic research into solving problems of other academic researchers, but I felt like it’d be improper to inquire. Let me give some concrete examples.
Researchers at Samsung built a prototype for sending code to be executed on other devices and making use of their resources. One demo was showing a game being played on a phone, and then having the game display seamlessly transfer to another device (that could be a smart TV, in theory). Pretty neat!
Well for one, this was for Tizen, an operating system that exists purely as a bargaining chip for Samsung and a backup strategy to not be solely dependent on Android. So there is no real-world application for this to run on any real devices. Furthermore, giving other devices the ability to make use of Tizen device resources is a huge avenue for security problems, as the presenter readily acknowledged. When combined with the fact that Tizen has more holes than swiss cheese this is doubly worrying. Additionally, one of the major unsolved huge problems with IoT (besides security) is interoperability between devices of different manufacturers. When I asked if there was any interest or plan to submit their work to a standards track, the presenter and host of the session got very confused.
Another talk was a study of running safe C/C++/Fortran code. It included implementations to provide safety of memory management, bounds checking, variadic arguments length checking, use-after-free, and double-free errors. Awesome! Fantastic! Just one catch – it’s only for running said code on the JVM. Since people don’t usually run C++ code on the JVM, this is of limited use, except possibly for tooling for people running Ruby on the JVM, one attendee told me. The talk and the paper had nothing to say on the performance overhead. Research was sponsored by Oracle Labs.
Actually, a vast amount of the research topics were relating to Java and the JVM, a situation I found scandalous. I had hoped and even assumed that academics would be proponents of Free Software, because of the massive contributions to learning, understanding, and implementation, while being unencumbered by profit-driven abuses of the legal system to the determent of progress. And yet they all live and breathe Java! Java is of course NOT free (not as in free beer, but as in libre – a French word meaning “Richard Stallman”), as was recently affirmed by a US district court which found Google liable for potentially 8 or 9 billion dollars for writing header files mimicking an interface of Java. Java is probably the least free language out there now.
In the first workshop I went to there was a session where we would all get to play with a new attribute-based grammar to compose a basic C language parser. Cool! But it was all done in Java. I said “sorry, I don’t have a JDK” and the entire room burst out laughing. “Who the fuck uses JAVA?” I asked, incredulous. “Uh, everybody!” came the smart-ass reply. Since there was no functioning internet at the conference venue, I couldn’t download the JDK, so I left. What a sad state of affairs for academia, to be so beholden to the most evil corporation in software today.
Research was presented on improving the efficiency of parsing ambiguities resulting from deep priority conflicts. An interesting and thoughtful study of helping compilers do a better job of catching a certain type of ambiguity and resolving it in an optimal fashion. They applied their analysis to 10,000 Java files on GitHub and 3,000 OCaml files, and found three conflicts in two Java files, but a great many in the OCaml source files.
So for all the folks out there doing serious work with OCaml, you’re in luck!
My favorite talk and the winner of an award at the conference was simply about Lisp, Jazz, and Aikido. And how they’re all cool and similar.
One of the student research projects sounded at first like it might be getting dangerously close to some sort of potentially useful application one day. One student talked about his system for dynamic access control for database applications. Unfortunately it requires using a contract language of his own devising in a lisp-lisp environment.
Don’t get me wrong, I enjoyed the conference and was frankly intimidated by all the super smart folks there. But it left me with a feel that so much talent, brains and time was being spent solving problems purely for the benefit and respect of other academics, instead of trying to solve serious problems facing us vulgārēs who have deadlines and business objectives and real-world problems to solve. The best part of the conference by far was just talking to the people there. I had a lot of interesting and thoughtful conversations. The research, eh.
As I’ve ended up with de facto maintainership of the illustrious projectM open source music visualizer I’ve seen a fair bit of interest in the project. I think I at least owe a blog post to update folks on where it’s at, what needs working on, and how to help make it better.
What is projectM?
projectM is a music visualizer program. In short it makes cool animations that are synchronized and reactive to any music input. I say music and not audio because it includes beat detection for making interesting things happen on the beat.
Some of you may remember the old windows mp3 player WinAmp. It contained a supremely amazing and innovative music visualizer called Milkdrop written by a gentleman from nVidia named Ryan Geiss, known just as Geiss. The visualizer was not a single set of rules for visualizing audio but rather a mathematical interpreter that would read in “preset” files which were sets of equations. You can read the very illuminating description here of how the files are defined if you’re interested. In short there is a set of per-frame equations describing colors and FFT waveforms and simple transformations, and there is a set of per-vertex equations for more detailed transformations and deformations.
Due to the popularity of WinAmp and Milkdrop there have been many thousands of presets authored and shared with really stunning and innovative visual effects ranging from animated fractals to dancing stick figures to bizarre abstract soups. The files are often named things like:
shifter – cellular_Phat_YAK_Infusion_v2.milk
[dylan] cube in a room -no effects – code is very messy nz+ finally some serious stfu (loavthe).milk
NeW Adam Master Mashup FX 2 Zylot – In death there is life (Dancing Lights mix)+ Tumbling Cubes 3d.milk
suksma + aderassi geiss – the sick assumptions you make about my car [shifter’s esc shader] nz+.milk
flexi + cope – i blew you a soap bubble now what – feel the projection you are, connected to it all nz+ wrepwrimindloss w8.milk
And so on.
As I understand it, possibly incorrectly, there were two major problems with Milkdrop. First that it was implemented with DirectX, win32 APIs and assembler, and secondly that it was not open source (though it was made open source fairly recently). So some enterprising folks in 2003 created projectM as an open source reimplementation that would be Milkdrop preset-compatible.
I didn’t work on projectM originally and I am not responsible for the vast majority of it. However the previous authors and contributors have for whatever reason mostly abandoned the project so it was left to random people to make it work. The code is quite old although the core Milkdrop preset parsing, beat detection, most of the OpenGL (more on that later) calls, and rendering is in fine shape. projectM is really just a library though, designed to be used by applications. In the past there have been XMMS and VLC plugins, a Qt application, pulseaudio and jack-based applications, and more.
OSX iTunes Plugin
Not really having a good solution for OSX I went ahead and ported the ancient iTunes visualizer code to work on a then-modern version of iTunes and voila! projectM on OSX. Though I did have to deal with the very unfortunate Objective-C++ “language” to make it work. Not Objective-C, Objective-C++. No I didn’t know that existed either.
I tried to submit the plugin to the Mac App store as a free download. Not to make money or anything, just to make it easy for people to get it. The unpleasantness of this experience with Apple and their rejection is actually what spurred me to start this blog so I could complain about it.
I decided that what would be better is a cross-platform standalone application that simply listens to audio input and visualizes it. This dream was made possible by a very recent addition to the venerable cross-platform libsdl2 media library adding support for audio capture. I quickly hacked together a passable but very basic SDL2-based application that runs on Linux and macOS and in theory windows and other platforms as well. Some work needs to be done to add key commands, text overlays (preset name, help, etc), better fullscreen support and easy selection of which audio input device to use.
The main application code demonstrates how simple libprojectM is to use. All one must do is set up an OpenGL rendering context, set some configuration settings, and start feeding in audio PCM data to the projectM instance. It automatically performs beat detection and drawing to the current OpenGL context. It’s really ideal for being integrated into other applications and I hope people continue to do so.
You can obtain source, OSX and linux builds from the releases page. This is super crappy and experimental and needed some configuration tuning to make it look good, and you need to drop the presets folder in. But it’s a start.
In their infinite wisdom the original authors chose the cmake build system. After wasting many hours of my life I will not get back and almost giving up on the software profession altogether I decided it would be easier to switch to GNU autotools, the same build system almost all other open source projects use, than to deal with cmake’s bullshit. So now it uses autotools (aka the “./configure && make && make install” system everyone knows and loves).
This is where you come in. If you like music visualizers and want to help the software achieve greater things there is some work to be done modernizing it.
The most important task by far is getting rid of the OpenGL immediate-mode calls and replacing them with vertex buffer object instructions. VBO is a “new” (not new at all) way of doing things that involves creating a chunk of memory containing vertices and pushing it to the GPU so it can decide how and when to render your triangles. The old-school way was “immediate mode” where you would tell OpenGL things like glBegin(GL_QUADS) (“I’m going to give you a sequence of vertices for quadrilaterals”) and give it vertices one at a time. This is tremendously inefficient and slow so it isn’t supported on the newer OpenGL ES which is what any embedded device (like a phone or raspberry pi) supports, as well as WebGL.
Astute readers may note that there already are iOS and Android projectM apps. They are made by one of the old developers who has made the decision to not share his modern OpenGL modifications with the project because he makes money off of them.
Another similar effort is to replace the very old dependency on the nVidia Cg framework for enabling shaders. Cg was used because it matches Directx’s shader syntax. GLSL, the standard OpenGL shader language is not the same, and requires manual conversion of the shaders in each preset.
The Cg framework has been deprecated and unsupported for many years and work needs to be done to use the built-in GLSL compilation calls instead of Cg and convert the preset shaders. I already did some work on this but it’s far from finished.
The reason I’m writing this blog post is because of the community interest in the project. People do send pull requests and file issues, and we definitely could use more folks involved. I am busy with work and can’t spend time on it right now but I’m more than happy to guide and help out anyone wishing to contribute. We got an official IRC channel on irc.freenode.net #projectm so feel free to hang around there and ask any questions you have. Or just start making changes and send PRs.
At this year’s FOSDEM in Brussels, Jan Tobias Mühlberg gave a talk on the latest work on Sancus, a project that was originally presented at the USENIX Security Symposium in 2013. The project is a fully open-source hardware platform to support “trusted computing” and other security functionality. It is designed to be used for internet of things (IoT) devices, automotive applications, critical infrastructure, and other embedded devices where trusted code is expected to be run.
A common security practice for some time now has been to sign executables to ensure that only the expected code is running on a system and to prevent software that is not trusted from being loaded and executed. Sancus is an architecture for trusted embedded computing that enables local and remote attestation of signed software, safe and secure storage of secrets such as encryption keys and certificates, and isolation of memory regions between software modules. In addition to the technical specification [PDF], the project also has a working implementation of code and hardware consisting of compiler modifications, additions to the hardware description language for a microcontroller to add functionality to the processor, a simulator, header files, and assorted tools to tie everything together.
Many people are already familiar with code signing; by default, smartphones won’t install apps that haven’t been approved by the vendor (i.e. Apple or Google) because each app must be submitted for approval and then signed using a key that is shipped pre-installed on every phone. Similarly, many computers support mechanisms like ARM TrustZone or UEFI Secure Boot that are designed to prevent hardware rootkits at the bootloader level. In practice, some of those technologies have been used to restrict computers to boot only Microsoft Windows or Google Chrome OS, though there are ways to disable the enforcement for most hardware.
In somewhat of a contrast to more proprietary schemes that some argue restrict the freedom of end-users, the Sancus project is a completely open-source design built explicitly on open-source hardware, libraries, operating systems, crypto, and compilers. It can be used, if desired, in specialized contexts where it is of critical importance that trusted code runs in isolation, on say an automobile braking actuator attached to a controller area network bus, or a smart grid system such as the type that was hacked in Ukraine during the attack by Russia. These are the opposite of general-purpose devices; instead, one specific function must be performed and integrity and isolation are critical.
The problem is that many medical devices, automotive controllers, industrial controllers, and similar sensitive embedded systems are made up of limited microcontrollers that may have software modules from different vendors. Misbehaving or malicious software can interfere in the operation of those other modules, expose or steal secrets, and compromise the integrity of the system. Integrity checks based in software are bypassed relatively easily compared to gate-level hardware checking; those checks also add considerable overhead and non-deterministic performance behavior.
Sancus 2.0 extends the openMSP430 16-bit microcontroller with a small and efficient set of strong security primitives, weighing in at under 1,500 lines of Verilog code and increasing power consumption by about 6%, according to Mühlberg. It can disallow jumps to undeclared entry points, provide memory isolation, and attestation for software modules.
Besides providing a key hierarchy and chain of trust for loading software modules, Sancus has a simple metadata descriptor for each module that stores the .text and .data ranges in memory; it then ensures that a .data section is inaccessible unless the program counter is in the .text range of the appropriate module. This is a simple but effective process isolation mechanism to ensure that secrets are not accessible from other software modules and that one module cannot disturb the memory of other modules.
Mühlberg mentioned that there is ongoing work on creating secure paths between peripherals for secure I/O, integration with common existing hardware solutions such as ARM TrustZone or Intel SGX, formal verification, and ensuring suitability for realtime applications.
To give a feel for the system in action, Mühlberg showed a demonstration video comparing two simulated automotive controller networks with malicious code running on a node. One can see the unsecured system behave erratically when receiving invalid messages, whereas the Sancus system gracefully slows down and safely disengages.
Much has been written about the upcoming IoTpocalypse: the lack of security in critical infrastructure and general despair about the dismal state of easily exploitable embedded systems as they multiply and get connected to the internet. A project based on open-source building blocks and free-software ethos that attempts to provide a layer of integrity and deterministic behavior to microcontrollers should be lauded and considered by anyone building hardware applications where security and reliability are strong requirements.
Budapest is one of my most favorite places of all. The architecture is some of the best in the world, there’s a mix of cultures from Austria-Hungary to Turks to Magyars. Everything I ate there was delicious. Great public transit. Turkish-style baths everywhere. Fun boat tour on the Danube river, separating Buda and Pest.
Good to visit as an American, too. At immigration most foreigners were being fingerprinted and questioned for long periods of time. When they saw my US passport they just waved me through.
Only real complaint is that all written words in Budapest are in Hungarian, a language I have absolutely zero chance of ever learning or understanding. It belongs to the Uralic family along with Finnish and Estonian, containing basically no relation to any other Greek, Latin, Romance, or Slavic tongues or words. Besides the unfortunate language it’s a great place.