I am developing a simple HTML5 game with Unity. The experience reminded me of my post about Rust and led me to coin a new term that I'd like to discuss today.

Ladies and gentlemen, in the spirit of Object-Oriented Programming I present to you Internet-Oriented Software Development (IOSD): a style of software development in which the official way to program is by trying something, giving up, asking strangers on the internet, and hoping for the best.

You may wonder: how is this new, seeing as that's what we are all already doing? The keyword here is "official". If you want to program with (say) Keras during a long trip with unreliable internet, you could do it with an offline version of the API reference alone. Sure, getting the offline docs will take a bit of work, but at least there's an official repository of knowledge that you can always go to. Of course you can search on the wider internet for help too, but you don't have to.

IOSD is different: when you release IOSD software you publish an okay guide to your software, and that's it. There is no need for you to keep it up to date nor for it to be useful, because your documentation is Internet-Oriented: if someone has a problem, they can ask the internet, IRC channels, their co-workers, anyone but you.

Rust came to mind because, as I complained before, that's how their intended development cycle apparently works: if you don't know how to do something, you are encouraged to either search the forums or ask the developers. In the later case, "we don't know, we won't do it, and we won't tell you who is doing it right" is a possible response. TensorFlow used to be like that too (ask me about the Seq2Seq tutorial!), but they reversed course and their current official suggestion is that you use Keras (no, seriously).

But Unity is an even better example. For starters, the official docs are essentially useless because they tell you what something does but neither why nor what to use it for. Can you guess in which situations is a Sprite renderer required, and what do you need it for? Because I can't. One might argue that Unity Learn is where you should look for answers, in which case one would be wrong. Taking the first course in the "Game development" section, for instance, gets you this tutorial which is only valid for an outdated version of Unity.

No, the real source of answers are YouTube tutorials. Sure, sometimes they refer to windows that aren't there anymore and/or changed their name, but you can always add a "2019.4" to your search and try again.

I am not entirely a beginner with Unity, as I worked with it for my PhD projects. Even then, the list of resources I needed to complete my current project so far includes 5 YouTube tutorials, 2 forum threads, and zero links to official documentation. Is this a problem? Is IOSD better than the thick manuals we had before? Am I the only one getting outdated answers for trivial problems? I have no idea. So I propose a compromise: I will point at the situation and give it a name, let someone else answer the hard questions, and we will share the credit.

Gephi is a tool for graph visualization that's great for plotting big datasets. If you have large, complex graphs to plot, Gephi has plenty of layouts that you can apply to your data and turn a mess of a graph into something that hopefully makes sense.

But there's something Gephi can't do: animations. The layout algorithms at work look awesome, and yet there's no easy way to create a video out of them. Capturing the screen is an alternative, but the final resolution is not really high.

It has been suggested that you could export images of the individual networks at every step and then put them together. Of course, the people who suggested this in the past have also...

• ... sworn that they have scripts to do it,
• ... promised that they'll release them after a quick cleanup, and
• ... disappeared from the internet without having published their solutions.

Therefore, this post.

Note: this post is not an introduction to Gephi, and I'll assume that you know the basics. If you don't know them, these slides look quite good.

Requirements

1. Gephi (obviously)
2. ImageMagick (to post-process images in bulk)
3. FFmpeg (to create the final video)

I am also assuming that you are using Linux. This is not a requirement, but it does make my life easier.

Preparing your data

The first step is to have some data to visualize. For the purposes of this exercise, you can download this zip file containing two files named nodes.csv and edges.csv. They form a simple directed graph that I generated from a sample of directories in my computer.

Now, load them into Gephi:

1. Start a new Project
2. Load the nodes table: File → Import spreadsheet → nodes.csv. Follow the steps, but remember to select "Append to existing workspace" in the last one.
3. Repeat the previous step, this time using the edges.csv file.

Next, choose a layout. In my experience it is better to try them all first, see which one gives the best result, and then generate the final graph from scratch. Generating the final graph will take a while, so it's better to do it only once you are sure about which parameters to use.

Exporting all frames

It is now time to run our first script. If you don't have the Scripting Plugin installed, you can do it now via Tools → Plugins. We will use it to write a very simple Python script that does the following:

1. Run a single step of a layout
2. Take note of the relative size of the graph (we'll come back to this)
3. Export it to a PNG file
4. Return to step 1.

In case you want to copy-paste it, this is the script I used. Don't forget to remove the comments first, because Gephi doesn't like them.

def make_frames(iters, layout, outdir):
    # As many iterations as frames we want to generate
    for i in range(iters):
        # Run a single step of our layout
        runLayout(layout, iters=1)
        # Calculate the bounding box of this specific graph
        min_x = 0
        max_x = 0
        min_y = 0
        max_y = 0
        for node in g.nodes:
            min_x = min(min_x, node.x)
            max_x = max(max_x, node.x)
            min_y = min(min_y, node.y)
            max_y = max(max_y, node.y)
        width = max_x - min_x
        height = max_y - min_y
        # Generate a file and include the graph's bounding box
        # information in its filename
        exportGraph("%s/%05d-%d-%d.png" % (outdir, i, width, height))

Once you have copied this script into the console, you can generate all animation frames with the command make_frames(100, FruchtermanReingold, "/tmp/"). This will run the FruchtermanReingold layout for 100 iterations, and will save the generated frames in the /tmp/ directory. Of course, you can choose other layouts (see the documentation for more info) and you can run the script for a larger number of steps. You can also customize the layout parameters in the regular Layout tab, and the script will pick them up.

The script will block Gephi entirely, so don't go for a really high number of steps from the beginning. Start with 50-100, and only then move on.
For a nicer effect, make a single run of the "Random" algorithm first. This will put all your nodes in a very small space, and the final effect will be like an explosion of nodes.

Generating the animation

A further issue to deal with is the changing size of the image canvas. If we set Gephi to generate a 1024x1024 output image but we only have two nodes close to each other, those two nodes will look huge. If we have thousands of disperse nodes, however, the individual nodes will be barely visible. Therefore, if you made a video with the images we generated in the previous section directly, you would almost certainly get a zoom effect where the nodes would get bigger and smaller as the graph gets denser or sparser respectively.

To avoid this, we need to scale all pictures proportionally. The Bash script below calculates the maximum theoretical size of our graph (based on those variables we added to the filenames before), scales all images down to the proper size (as defined by canvas_w and canvas_h), and places them in the center of a single-color canvas (see the rgb() call).

# Directory where all individual frames are
SOURCE_DIR=/tmp
# Directory where final frames will be stored
TARGET_DIR=/tmp/outdir

# Obtain the theoretical maximum width and height in the PNG frames
max_width=`ls ${SOURCE_DIR}/*png | cut -f 2 -d '-' | sort -n | tail -1`
max_height=`ls ${SOURCE_DIR}/*png | cut -f 3 -d '-' | cut -f 1 -d '.' | sort -n | tail -1`

# Give your desired canvas size
canvas_w=1024
canvas_h=1024

# Scaling factor for the frames, based on the largest theoretical dimension
if (( $max_width > $max_height ))
then
    factor=`bc -l <<< "$canvas_w/$max_width"`
else
    factor=`bc -l <<< "$canvas_h/$max_height"`
fi

# Generate the new frames
for file in ${SOURCE_DIR}/*png
do
    # Obtain the properties of the image
    frame=`echo $file | xargs -n 1 basename | cut -f 1 -d '-'`
    width=`echo $file | cut -f 2 -d '-'`
    height=`echo $file | cut -f 3 -d '-' | cut -f 1 -d '.'`
    # Calculate how the image should be scaled down
    new_width=`bc -l <<< "scale=0; ($width*$factor)/1"`
    new_height=`bc -l <<< "scale=0; ($height*$factor)/1"`
    # Put it all together
    convert $file -scale ${new_width}x${new_height} png:- | \
        convert - -gravity Center -background "rgb(255,255,255)" \
        -auto-orient -extent ${canvas_w}x${canvas_h} \
        ${TARGET_DIR}/${frame}.png
done

Once you have generated this second set of frames, you can generate your final video going to your TARGET_DIR directory and running the command

ffmpeg -framerate 30 -i %05d.png -c:v libx265 -r 24 final_video.mp4

If your video is too slow, a higher framerate value will do the trick (and vice versa). The final result, heavily compressed for the internet, can be seen below:

Final thoughts

I hope you'll find this guide useful. I'm not going to say that it's easy to follow all these steps, but at least you can set them up once and forget about it.

Some final points:

• For an alternative method of graph generation involving nodes with timestamps, this script looks like the way to go.
• I'm interested in unifying everything under a single script - I chose Python because it was easier than Java, but maybe developing a proper multi-platform plugin is the way to go. I can't promise I'll do it, but I'll think about it. In that same vein, perhaps the script should center in a specific node?
• If you plan to publish your video on the internet by yourself, this post gives a nice overview of which standards to use. If you want to tweak the video quality, this SE question provides some magic incantations for FFmpeg.
• Finally, and speaking of magic incantations, special thanks to this post for providing the right ImageMagick parameters.

Here is an idea that I had and that I don't have time to work on. I read somewhere about the following job interview question:

What happens after you write a URL in your browser and press Enter?

If you think about this for a moment, you might realize that what this question really means is "tells us everything you know about computers". I have yet to find a topic that wouldn't be involved in giving a full response. Off the top of my head, and in roughly chronological order, you would have to explain...

• ... how your keyboard sends signals, including the difference between pressing and releasing a key. Also, how your computer display works.
• ... how to turn a series of electric impulses into a character.
• ... how to parse a URL, including the difference between Unicode and ASCII.
• ... how the internet works: DNS, TCP/IP, IPv4 vs IPv6, routing, etc
• ... how the browser and server negotiate the type of content they want. It might also include an introduction to the GZIP compression algorithm.
• ... a primer on HTTPS, including cryptography and handling certificates.
• ... what is a web server and how it works. Same for load balancers, proxies, and pretty much all modern server infrastructure.
• ... how your operating system renders anything on screen.
• ... how your web browser renders content.
• ... the standards involved in receiving content: HTML, CSS, JavaScript, etc.

I imagine that this would be a great idea for a Wiki: the main page would simply present the general question, and you could go deeper and deeper until you reach your motherboard's buses, your microprocessor's cache, the specifics of BGP, or pretty much anything that was ever used in an internet-connected computer.

I was never asked this question, which is a bit of a missed opportunity: I don't know exactly how many hours I could waste on this question, but I'm willing to bet it would be more than what any reasonable interviewer is willing to spend. More realistically, I imagine the point of the question is both to check whether you know about computers and, more important, whether you know when to stop talking about computers.

The Council of the European Union has released this Draft in which they call for what is effectively a ban on End-to-End Encryption (E2EE). The document itself is unsurprisingly vague, but if you follow the parallel document about "Exceptional Access" you'll see a bunch of proposed solutions, all of which require the interception of your private communications. As it is to be expected, the documents pinky swears that this is the only way that terrorists and child predators will be stopped.

There are several reasons why this is a stupid idea. Today's post will briefly detail the main two.

First, this is technically impossible. The entire point of E2EE is that no one (not you, not me, not the NSA) can decrypt their content without the right key. And yet, the proposal that has been passed around in the last years is the idea of a "master key", a key that only authorities have and that would be "carefully" used by the authorities to legally decrypt content between two parties that they consider suspicious. So let's assume that WhatsApp implements this idea. They now have a single key that only the EU can access. Well, two keys - Australia has legally mandated backdoors, so they need their own. And China will need one too. The US wouldn't need one, simply because some of WhatsApp servers are in the USA and therefore the NSA can use a National Security Letter to force WhatsApp to reveal the other keys while forbidding everyone to talk about this. As you can see, the "one single key" idea is flawed from the very beginning.

And then there are the hackers: if it comes out that there is a secret key that breaks WhatsApp's encryption, it is now a race between WhatsApp's engineers to keep it safe against every single government in the world trying to break it.

The second main point is: if you ban secure communications, then only criminals will have access to secure communications. We already have unbreakable encryption and it is trivial for any criminal organization to deploy their own. So they are not the ones whose communications will get intercepted. The only wiretapped ones will be us, the law-abiding citizens. Instead of keeping us safe from criminals, the Council of the European Union is delivering us into the data collection efforts of the NSA and friends.

A call for action

Do you remember when the European Unions imposed sanctions against the NSA for their illegal data collection? Me neither, because that didn't happen. And I don't see why this time it would be any different. Well, there is that one time when Angela Merkel told Obama that she was angry he wiretapped her phone. I'm sure he felt really bad about that. But my point is: I wouldn't expect our politicians to stand up for our privacy, in particular when they are the ones creating the problem to begin with.

We have once again a proposal that will not stop any criminals, is technically impossible, and that is being written without asking anyone who knows what they are talking about. If you are in the EU I ask you to contact your representatives - I am not aware at the time of any movement against this, but I bet at least the Pirate Party will have something to say (edit Nov. 25: they do). The tech industry already lost the DRM fight (as exemplified by the ongoing youtube-dl saga) and the fight against Article 13. And there are lobbying efforts underway to bring software patents to Europe.

Don't let your privacy go away too.

Further reading

• Whenever someone swears that they can keep the "master key" secret, remind them of that time the secret NSA luggage keys ended up in the Washington Post.
• A Hacker News thread with more than 650+ comments discussing several other points with much more details.

In today's weekend posting, two recommendations about things that are not free (which is a first) and a rant (which is very much in brand for this blog).

Drawing faces with JLJ

On a previous blog entry I complained that it's very difficult to find a good drawing tutorial because many, many teachers will suggest something as useless as "do whatever comes natural". So imagine my surprise when I found a course on drawing faces that makes none of those mistakes.

The course in question is titled "How to draw a portrait" and is taught by an illustrator from Florida called Joshua L Johnson. The course guides you through the steps of framing your drawing, identifying the main features, refining the details and, finally, adding shadows. The course can be found on Skillshare following this link.

I like this course for a couple reasons. First, each step is actionable: when he wants you to draw an eye, he explains that a generic eye is composed of 7 segments and explains where to place each one. Second, the workflow itself is designed in a smart way, first delimiting "areas" of work and then refining them step by step. The course ends with a 40 minutes, real-time lesson on how to draw a specific face from beginning to end which I found really helpful. So if your faces are as bad as mine, you should consider taking a look.

Solutions and other problems

It is hard for me to express to you how ridiculously funny Allie Brosh is. Her blog Hyperbole and a half is the only website I can remember where I had to stop reading for minutes at a time because I couldn't stop laughing. Some of the most well-known entries are probably This is why I'll never be an adult which gave rise to the "all the things" meme, and the creation of the Alot. Unsurprisingly, her first book collecting some of these stories ended up being a New York Times best-seller.

Perhaps more well-known are her two posts on depression (part 1, part 2) where she manages to put in words the feelings of thousands of people. I have seen an actual therapist recommend these posts to people, and the almost 10K collective comments in those entries alone seem to agree.

And the reason I am bringing up these two sides of her blog is because I recently read her second book, and let me tell you, it is a roller coaster: it is funny, it is sad, and sometimes it's both at the same time. It is the best thing I read all year, and I think everyone should do the same. To say that I recommend it would be an understatement. It would be more accurate for you to imagine me grabbing you by your clothes while yelling "READ THIS BOOK".

Disclaim all of the things

I didn't want to leave this post as it is without complaining about how difficult it is to make an honest recommendation on the internet.

I have a subscription to Skillshare because I like the quality of their courses, but I am really, really annoyed at their marketing showing up everywhere. With so many youtubers doing paid promotions for courses they don't care about, I feel slightly dirty making a recommendation just like them, even if no one is paying me for doing it. I thought for a second about pointing you to a free mirror, but that would be unfair to the course's creator.

Similarly, someone on Allie Brosh's publishing team had the brilliant idea of creating fake Reddit accounts and using them to market the book. People like them make it impossible for me to recommend almost anything in good conscience. I have decided to make an exception for this specific book, but I don't see that happening again anytime soon.