Monday, August 4, 2014

Still alive and kickin'

Apologies that it's been a while since I've written a proper release announcement. As many of you know, I'm working full-time at System76 now. With less time for Novacut, I decided to forgo the release announcements for the sake of eking out just a bit more development each month.

But we've still been chugging along, and have continued to deliver our monthly releases like clockwork.

We've been quietly making big improvements in three critical areas: Data Safety, Security, and Performance.

Data Safety

Dmedia makes the bold promise of "making file management go way" for creative professionals. That's pretty much like promising a unicorn. And if you promise a unicorn... well, you better deliver a unicorn, and your unicorn better work.

As they say, the proof is in the pudding (er, unicorn), and our pudding works:

For every file in your library, Dmedia will try to maintain three known-good copies, each on a different physical drive. This is the equilibrium point toward which Dmedia will always strive.

Think of Dmedia like a marble in bowl. If you pull the marble up the side up the bowl and release it, gravity will pull the marble back down toward the center, and friction will slow the back-and-forth motion till the marble comes to rest.

Dmedia has only three behaviors, or compulsions if you will, that drive its quest for equilibrium:

  1. Create new copies
  2. Delete redundant copies
  3. Verify existing copies

Think of the first behavior (create new copies) as the way Dmedia moves the marble toward the center. Think of the second behavior (delete redundant copies) as what Dmedia does when it overshoots the center.

And for the third behavior (verify existing copies), imagine someone is holding the bowl and tilting it side to side, introducing unpredictable external influences. Dmedia does constant reality checks, and when reality doesn't match its metadata, Dmedia updates its metadata to match reality.

(The "marble in a bowl" metaphor isn't perfect, but hopefully it helps you visualize what Dmedia does.)

Although Dmedia has had all these behaviors turned-on for some time, Dmedia now handles extreme abuse with ease. It can gracefully recover when there are unexpected crashes in any of its background tasks. It has multiple, independent mechanisms by which most data-safety tasks can be accomplished. In general, Dmedia has greatly matured and is almost ready for the main stage.

However, with great data comes great responsibility :D So I'm still not comfortable giving Dmedia the "production ready" seal of approval. We still need some additional UI for communicating certain data-safety-sensitive things with the user. And we also need a way to run automated, multi-node simulations for validating each monthly Dmedia release. Once I declare Dmedia as "production ready", we aim to keep that promise release, after release... after release.

For details on our current manual testing procedures, and our plans for long-running simulation testing, please see Testing Dmedia.

Still, I think it's a great time to give Dmedia a spin and see what all the fuss is about. Just please do so with files that are safely backed up elsewhere.


I strongly feel that passwords aren't a viable authentication mechanism going forward, especially for the so called "Internet of Things".

For example, I think it's a super bad idea to use password authentication to locate and unlock your car from anywhere on the Internet. (In their defense, I think Tesla is an extraordinary company, but they need to up their game here, to match their excellence elsewhere.)

If you want to authorize a new phone to be able to unlock your car, it's reasonable to require you to be physically in your car with said phone. And with this constraint, Tesla could use something like the Firefox Sync device peering, or the Dmedia device peering (which was heavily inspired by FireFox Sync, but Dmedia uses direct P2P communication on the local network instead of mediation through a 3rd party server).

I'd love to see more well-vetted, standard solutions emerge in this space, and that's a big part of why I've split out the internal Dmedia HTTP server into a new library called Degu. Degu includes an HTTP server and a matching HTTP client, and is squarely aimed at implementing REST APIs for device-to-device communication on the local network.

I plan to implement the next generation of the Dmedia peering protocol and identity framework in Degu with the aim of it being generally useful for many types of applications. I hope Degu can be a productive research platform for security and usability in the coming Internet of Things. If you're interested in being involved with this work, or building something atop Degu, please shoot us an email or stop by the #novacut IRC channel.

We've also modernized our SSL configuration (partly thanks to Python 3.4). We're now using TLSv1.2 with ECDH-based perfect-forward-secrecy. And we're now using 4096-bit RSA keys with sha384 signatures for our machine and user certificates (up from 2048-bit RSA keys with sha1 signatures).


Performance is closely related to data safety as the time it takes Dmedia to converge at its equilibrium point largely depends on how well Dmedia uses the available IO capacity. Files must be downloaded from one peer to another, copied from one drive to another, etc. And remember, all this can be happening simultaneously among several devices, so it needs to be done in a way such that different peers don't needlessly create new copies of the same fragile file at the same time.

Yet Dmedia is fundamentally a lock-free and completely distributed system. A surprisingly effective way to keep peers from stepping on each other toes has simply been to randomize the order in which fragile files are dealt with, and we use this technique in a number of places.

We've also made big performance improvements when it comes to our metadata layer. Much of this improvement is thanks to the Degu HTTP client, which is a fair bit faster than the http.client module in the Python3 standard library.

Degu 0.7 also brings the first step in replacing the common HTTP parser and IO abstractions shared between the server and client with a high-performance C extension.

Other news

David Jordan has been making great progress on importing Novacut edits into Blender:

For example, this was edited in Novacut, finished in Blender:

Dmedia (and Novacut) are far more compelling if the underlying Dmedia platform can be used across a broad ecosystem of applications.

Install Novacut 14.07

Packages are available in ppa:novacut/stable for Ubuntu 14.04 LTS.

To install Novacut on Ubuntu 14.04, just open a terminal and run these three commands:

sudo apt-add-repository ppa:novacut/stable
sudo apt-get update
sudo apt-get install novacut

If you want to help develop Novacut, it's best to install from ppa:novacut/daily.

Note if you've added both the daily and the stable PPAs, the versions in the daily PPA will supersede the stable versions. For more details on the PPAs, read about our Monthly Release Process.

Source code

You can download the source code from each component's Launchpad project page:

Saturday, April 19, 2014

Security alert: Dmedia vulnerable to Heartbleed

Dmedia (and therefor Novacut) are affected by the Heartbleed bug in the OpenSSL library. This bug is very serious as it allows an attacker to capture the private keys Dmedia uses, which then allows an attacker to steal both your Dmedia library metadata and the files it contains.

Please see USN-2165-1 for details about the OpenSSL fix in Ubuntu.

What you need to do

To correct this problem, first make sure your packages are up-to-date:

sudo apt-get update
sudo apt-get dist-upgrade

Then you'll need to force Dmedia to generate new user and machine certificates:

rm ~/.local/share/dmedia/user-1.json
rm ~/.local/share/dmedia/machine-1.json
restart dmedia

You should do this on all your computers running Dmedia before peering them again.

The next time you open Dmedia or Novacut, you'll see this screen:

On your first computer, click New Account. On any additional computers, click Connect to Devices and then accept the peering offer on the first computer.

More details

It's easy for an attacker on the local network to use the Heartbleed bug to attack Dmedia on systems running a vulnerable version of OpenSSL. This includes when you're using, for example, a public WiFi network at a coffee shop. This is true even when you only have a single Dmedia device on a given network.

In practice it's probably very difficult for a remote attacker to exploit Heartbleed in Dmedia from across the Internet. Most home routers use NAT to prevent direct access to your computers from across Internet. Also, each time Dmedia starts, it runs on a different, random port. Dmedia uses Avahi to advertise this random port to other Dmedia devices on the local network. Dmedia does not advertise this random port to any outside servers. That said, remote attacks could sill be possible if, for example, your router was compromised.

As Dmedia is not yet widely used, it's probably not yet a common attack target. However, to play it safe, please follow the above procedure to generate new Dmedia SSL certificates.