Monday, August 4, 2014

Still alive and kickin'

Apologies that it's been a while since I've written a proper release announcement. As many of you know, I'm working full-time at System76 now. With less time for Novacut, I decided to forgo the release announcements for the sake of eking out just a bit more development each month.

But we've still been chugging along, and have continued to deliver our monthly releases like clockwork.

We've been quietly making big improvements in three critical areas: Data Safety, Security, and Performance.

Data Safety

Dmedia makes the bold promise of "making file management go way" for creative professionals. That's pretty much like promising a unicorn. And if you promise a unicorn... well, you better deliver a unicorn, and your unicorn better work.

As they say, the proof is in the pudding (er, unicorn), and our pudding works:

For every file in your library, Dmedia will try to maintain three known-good copies, each on a different physical drive. This is the equilibrium point toward which Dmedia will always strive.

Think of Dmedia like a marble in bowl. If you pull the marble up the side up the bowl and release it, gravity will pull the marble back down toward the center, and friction will slow the back-and-forth motion till the marble comes to rest.

Dmedia has only three behaviors, or compulsions if you will, that drive its quest for equilibrium:

  1. Create new copies
  2. Delete redundant copies
  3. Verify existing copies

Think of the first behavior (create new copies) as the way Dmedia moves the marble toward the center. Think of the second behavior (delete redundant copies) as what Dmedia does when it overshoots the center.

And for the third behavior (verify existing copies), imagine someone is holding the bowl and tilting it side to side, introducing unpredictable external influences. Dmedia does constant reality checks, and when reality doesn't match its metadata, Dmedia updates its metadata to match reality.

(The "marble in a bowl" metaphor isn't perfect, but hopefully it helps you visualize what Dmedia does.)

Although Dmedia has had all these behaviors turned-on for some time, Dmedia now handles extreme abuse with ease. It can gracefully recover when there are unexpected crashes in any of its background tasks. It has multiple, independent mechanisms by which most data-safety tasks can be accomplished. In general, Dmedia has greatly matured and is almost ready for the main stage.

However, with great data comes great responsibility :D So I'm still not comfortable giving Dmedia the "production ready" seal of approval. We still need some additional UI for communicating certain data-safety-sensitive things with the user. And we also need a way to run automated, multi-node simulations for validating each monthly Dmedia release. Once I declare Dmedia as "production ready", we aim to keep that promise release, after release... after release.

For details on our current manual testing procedures, and our plans for long-running simulation testing, please see Testing Dmedia.

Still, I think it's a great time to give Dmedia a spin and see what all the fuss is about. Just please do so with files that are safely backed up elsewhere.


I strongly feel that passwords aren't a viable authentication mechanism going forward, especially for the so called "Internet of Things".

For example, I think it's a super bad idea to use password authentication to locate and unlock your car from anywhere on the Internet. (In their defense, I think Tesla is an extraordinary company, but they need to up their game here, to match their excellence elsewhere.)

If you want to authorize a new phone to be able to unlock your car, it's reasonable to require you to be physically in your car with said phone. And with this constraint, Tesla could use something like the Firefox Sync device peering, or the Dmedia device peering (which was heavily inspired by FireFox Sync, but Dmedia uses direct P2P communication on the local network instead of mediation through a 3rd party server).

I'd love to see more well-vetted, standard solutions emerge in this space, and that's a big part of why I've split out the internal Dmedia HTTP server into a new library called Degu. Degu includes an HTTP server and a matching HTTP client, and is squarely aimed at implementing REST APIs for device-to-device communication on the local network.

I plan to implement the next generation of the Dmedia peering protocol and identity framework in Degu with the aim of it being generally useful for many types of applications. I hope Degu can be a productive research platform for security and usability in the coming Internet of Things. If you're interested in being involved with this work, or building something atop Degu, please shoot us an email or stop by the #novacut IRC channel.

We've also modernized our SSL configuration (partly thanks to Python 3.4). We're now using TLSv1.2 with ECDH-based perfect-forward-secrecy. And we're now using 4096-bit RSA keys with sha384 signatures for our machine and user certificates (up from 2048-bit RSA keys with sha1 signatures).


Performance is closely related to data safety as the time it takes Dmedia to converge at its equilibrium point largely depends on how well Dmedia uses the available IO capacity. Files must be downloaded from one peer to another, copied from one drive to another, etc. And remember, all this can be happening simultaneously among several devices, so it needs to be done in a way such that different peers don't needlessly create new copies of the same fragile file at the same time.

Yet Dmedia is fundamentally a lock-free and completely distributed system. A surprisingly effective way to keep peers from stepping on each other toes has simply been to randomize the order in which fragile files are dealt with, and we use this technique in a number of places.

We've also made big performance improvements when it comes to our metadata layer. Much of this improvement is thanks to the Degu HTTP client, which is a fair bit faster than the http.client module in the Python3 standard library.

Degu 0.7 also brings the first step in replacing the common HTTP parser and IO abstractions shared between the server and client with a high-performance C extension.

Other news

David Jordan has been making great progress on importing Novacut edits into Blender:

For example, this was edited in Novacut, finished in Blender:

Dmedia (and Novacut) are far more compelling if the underlying Dmedia platform can be used across a broad ecosystem of applications.

Install Novacut 14.07

Packages are available in ppa:novacut/stable for Ubuntu 14.04 LTS.

To install Novacut on Ubuntu 14.04, just open a terminal and run these three commands:

sudo apt-add-repository ppa:novacut/stable
sudo apt-get update
sudo apt-get install novacut

If you want to help develop Novacut, it's best to install from ppa:novacut/daily.

Note if you've added both the daily and the stable PPAs, the versions in the daily PPA will supersede the stable versions. For more details on the PPAs, read about our Monthly Release Process.

Source code

You can download the source code from each component's Launchpad project page:

Saturday, April 19, 2014

Security alert: Dmedia vulnerable to Heartbleed

Dmedia (and therefor Novacut) are affected by the Heartbleed bug in the OpenSSL library. This bug is very serious as it allows an attacker to capture the private keys Dmedia uses, which then allows an attacker to steal both your Dmedia library metadata and the files it contains.

Please see USN-2165-1 for details about the OpenSSL fix in Ubuntu.

What you need to do

To correct this problem, first make sure your packages are up-to-date:

sudo apt-get update
sudo apt-get dist-upgrade

Then you'll need to force Dmedia to generate new user and machine certificates:

rm ~/.local/share/dmedia/user-1.json
rm ~/.local/share/dmedia/machine-1.json
restart dmedia

You should do this on all your computers running Dmedia before peering them again.

The next time you open Dmedia or Novacut, you'll see this screen:

On your first computer, click New Account. On any additional computers, click Connect to Devices and then accept the peering offer on the first computer.

More details

It's easy for an attacker on the local network to use the Heartbleed bug to attack Dmedia on systems running a vulnerable version of OpenSSL. This includes when you're using, for example, a public WiFi network at a coffee shop. This is true even when you only have a single Dmedia device on a given network.

In practice it's probably very difficult for a remote attacker to exploit Heartbleed in Dmedia from across the Internet. Most home routers use NAT to prevent direct access to your computers from across Internet. Also, each time Dmedia starts, it runs on a different, random port. Dmedia uses Avahi to advertise this random port to other Dmedia devices on the local network. Dmedia does not advertise this random port to any outside servers. That said, remote attacks could sill be possible if, for example, your router was compromised.

As Dmedia is not yet widely used, it's probably not yet a common attack target. However, to play it safe, please follow the above procedure to generate new Dmedia SSL certificates.

Tuesday, July 23, 2013

What Ubuntu Edge is *Really* about

Canonical's Ubuntu Edge campaign on Indiegogo is breaking records left and right, and if it reaches its goal by August 21st, will break the all-time crowdfunding record by a huge margin. In just 12 hours, the campaign raised more than any other on Indiegogo and enough to place within the top 10 on Kickstarter. The phone Mark Shuttleworth has proposed contains some serious hardware and a gorgeous design. Clearly thousands of people want a piece of this vision, but like every project it has its detractors.

Ubuntu Edge is an ambitious project by a large company, two strikes against it in some people's books.
Why doesn't Canonical just fund it themselves and sell it once it's ready? Don't Canonical have more than enough? These people suggest Canonical should just take the risk themselves and release it when its ready.
For those who fear taking a risk on a project that might not succeed or think crowdfunding should be about the little guy, these ideas make sense. And yet, they completely miss the point of Ubuntu Edge!
This crowdfunding campaign isn't really about whether Canonical can afford to produce a limited-run phone. More than the phone itself, Ubuntu Edge is about showing the world that Ubuntu Phone has legs, that there is real demand for this. Even if Mark Shuttleworth can afford to fund Ubuntu Edge, he shouldn't. This campaign may be the greatest statement the community can possibly make to support Ubuntu Touch.
The Ubuntu community is being asked to show that Ubuntu isn't just a great open source project with a track record for excellent software engineering, but a community willing to support hardware development.  The community is proving to OEMs that there is a viable market for hardware with Ubuntu Touch and proving to carriers that people want to buy this hardware.
If tens of thousands of people will pay unsubsidized prices for a high-end smartphone that won't ship for almost a year, breaking all crowdfunding records in the process, that sends a powerful message to OEMs, to carriers, to the media, and to app developers. There is a long road ahead for Ubuntu Edge, but already, the community is sending a clear message. In less than a day, the Ubuntu community has proved that there are thousands of people willing to take a chance on Ubuntu Phone.

I'm also quite confident in Canonical's ability to deliver on their ambitious plan. Canonical has already proven themselves quite capable at bringing Ubuntu up on existing hardware, so their ability to make Ubuntu work isn't really even a question. This will be their first original piece of hardware, but Canonical's OEM partners will probably handle much of the electronics design and manufacturing. Canonical's keen eye for design and concern for usability details is apparent in the hardware decisions revealed so far, and they will no doubt be very involved with the development of the hardware. However, they will likely be working with partners who will know how to engineer the hardware and manufacture the device in volume, which is no doubt the plan.

Right now, the greatest uncertainty for Ubuntu Edge is whether or not the Ubuntu community can step up, break the crowdfunding record, and reach the goal Canonical set for the campaign. The first day was a tremendous success,  reaching more than 10% of the total, but campaigns generally slow down in the middle, and it will be a long road ahead. Canonical and the Ubuntu community have grabbed the attention of mainstream media with a lot of positive momentum already. No doubt OEMs are watching carefully now. Success will almost certainly guarantee future Ubuntu Touch devices, while failure here would hurt Ubuntu's chances with OEMs.
Pre-orders are more than just a chance to get a cutting-edge phone with Ubuntu Touch, Android, and desktop-morphing capabilities. With every pre-order backers vote with their wallets for Ubuntu's success. The people who back this campaign are saying “I believe in Ubuntu Touch, and I'm willing to take the risk of not knowing exactly what it will be or when it will arrive because this is worth it.” Only a small fraction of the potential market for Ubuntu phones will be willing to buy an unsubsidized device sight unseen, and so the effect of every pre-order is magnified because it signals many more potential customers down the road. There may never be another time when voting with your wallet will make more difference for Ubuntu than right now.
This more than anything is why Mark Shuttleworth is asking for the community's help. It's entirely possible he could fund a production run of 50,000 phones and sell them all, but that wouldn't send a very strong a message to OEMs and carriers, much less the mainstream media or the general public. Mark Shuttleworth and the rest of Canonical aren't sending a plea for help but a call to arms, rallying for everyone to join together in support of a greater cause.

And that is the point.

-David Jordan

Sunday, June 2, 2013

Novacut 13.05

This is a very exciting release because all the Dmedia automation behaviors have now been turned on. Although note that we still don't consider Dmedia production ready, so please only test Dmedia with files you have safely backed up elsewhere.

This screencast walks you through installing Novacut on Ubuntu 13.04, and gives you a tour of all the new features:

This is also an exciting release because the Version One Hashing Protocol has been finalized and is now used by default. The only downside is that because V1 uses our Dbase32 encoding, it's not possible to support Version Zero alongside Version One.

This screencast explains how to upgrade your V0 Novacut library to V1:

In a nutshell, if you've used Novacut prior to this month's release, after you install Novacut 13.05 you'll need to open a terminal and run this command to upgrade your library:


As it seems very few have existing Dmedia libraries they care about, we didn't put much effort into this user experience (obviously), but we did put a lot of effort into making sure the upgrade works correctly. However, if anyone out there has problems upgrading or needs help, please email, and we'll make sure you get your data migrated.

Special Thanks

Thanks to Marco Buono for making this great Novacut intro sequence:

Thanks to saras fox, David Jordan, and James Raymond for helping me test the V0 to V1 upgrade this month.

And thanks to the many people who have patiently reviewed the many protocol iterations leading up to Version One, especially Hagen F├╝rstenau, Robert von Burg, and David Jordan.

Install Novacut 13.05

Packages are available for Ubuntu 13.04 in ppa:novacut/stable.

If you want to help develop Novacut, it's best to install from ppa:novacut/daily.

Note if you've added both the daily and the stable PPAs, the versions in the daily PPA will supersede the stable versions. So same effect as if you only added the daily PPA. For more details on the PPAs, read about our Monthly Release Process.

Source code

You can download the source code from each component's Launchpad project page:

Thursday, February 28, 2013

Novacut 13.02, with caution

This month's release is a bit different. We're in the process of turning on the remaining Dmedia automation behaviors, and switching from the interim Version Zero hashing protocol and schema to Version One.

Please note that we don't yet consider Dmedia "production ready", so please don't yet trust Dmedia as the only place where you store your files. But turning on these remaining automation behaviors means I feel we're almost there.

We want to be very cautious during this transition, and so we recommend that most users skip the Novacut 13.02 and 13.03 releases. To make this more than just a suggestion, 13.02 has only been released for Ubuntu Raring (the Ubuntu development version, which will be released as Ubuntu 13.04 on April 25th).

Most of our PPA users are on Ubuntu 12.04 LTS (Precise) and Ubuntu 12.10 (Quantal), while we have only a handful on Raring. This is a good thing while we validate these changes. We likewise don't recommend our users upgrade to Ubuntu Raring yet... wait till after the stable release on April 25th.

In the mean time, Precise and Quantal users can continue to use the Novacut 13.01 release. You'll be able to migrate your Dmedia library and Novacut edits from V0 to V1 whenever you upgrade to Ubuntu 13.04.

For details on how we'll do the V0 to V1 migration, please see this post.

What's New

After more feedback and testing, I've decided that we are indeed switching to the Dbase32 encoding I proposed last month. The V1 protocol now uses Dbase32 instead of standard RFC-3548 Base32 encoding, although V0 is still the active protocol.

Most of the work this month was in preparing FileStore and Dmedia for the migration. Dropping Precise and Quantal support also meant we could drop support for Python 3.2, and start taking advantage of some of the richer POSIX features available in Python 3.3. There was also a critical fix so that Dmedia downgrades a drive when metadata about the drive is missing.

For details, see the release notes for FileStore 13.02 and Dmedia 13.02.

Special Thanks

Thanks to Simon Wells, saras fox, and James Raymond for diligently reviewing my many merge proposals.

Thanks to Robert von Burg for continuing to provide feedback on the Dbase32 encoding and the V1 Protocol Specification.

And thanks to David Jordan for his work on porting Graffik to Ubuntu, which is Dynamic Perception's open source control app for nanoMoCo, an open hardware stepper motor controller:

Plus David had a chance to play with the Ubuntu Touch SDK:

As always, if you want to get involved with Novacut design or development, please stop by the #novacut IRC channel on freenode and introduce yourself.

Install Novacut 13.02

Packages are available for Ubuntu Raring in ppa:novacut/stable. But remember, we recommend that most users skip the Novacut 13.02 and 13.03 releases.

If you want to help develop Novacut, it's best to install from ppa:novacut/daily.

Note if you've added both the daily and the stable PPAs, the versions in the daily PPA will supersede the stable versions. So same effect as if you only added the daily PPA. For more details on the PPAs, read about our Monthly Release Process.

Source code

You can download the source code from each component's Launchpad project page:

Saturday, February 2, 2013

Dropping support for Ubuntu Precise and Quantal

Novacut and Dmedia users be warned: in either the 13.02 (February) or 13.03 (March) release, we're going to drop support for Ubuntu Precise (12.04 LTS) and Quantal (12.10).

I know this will be a bit frustrating for folks, for which I apologize. But on the upside: (1) you'll still be able to keep running 13.01 on Precise and Quantal, (2) the next few releases will likely be very boring from a user's perspective anyway, and (3) you'll be able to upgrade directly from 13.01 without loosing your current Dmedia library or Novacut edits.

We're dropping support for Precise and Quantal because, at long last, Dmedia is going to be officially "production ready". Over the next few months, we'll be turning on the remaining automation behaviors, including those scary but deeply important "copy reducing" behaviors.

The biggest reason we're only supporting Raring is we need a smaller target when validating Dmedia. By our current PPA stats, we have very few Raring users, and that's a good thing while we turn on the remaining Dmedia features, just in case we make a goof.

By only supporting a single Ubuntu release initially, we can do deeper validation of Dmedia. Validating Dmedia is going to take multiple physical computers running an extended simulation (at least one week per run). Right now, I don't have the time to personally do this properly for more than one Ubuntu release. I'd rather have higher confidence in supporting a single Ubuntu version than less confidence in it working on 3 versions.

By focusing on Raring, we can have a higher-quality Dmedia release ready when Ubuntu 13.04 starts shipping on hardware, just in case Dmedia starts shipping on hardware around that time (wink, wink).

And although it would be nice to keep supporting Novacut and Dmedia on Ubuntu 12.04 LTS, I think it's far more important to be forward looking and focus on making Novacut and Dmedia amazing on Ubuntu 14.04 LTS.

If you have questions or concerns about any of this, please ask!

Thursday, January 31, 2013

13.01, first release of the new year

What's New

Thanks to Simon Wells, the Dmedia browser got a nice makeover and now displays useful metadata:

Other than that, this was a slow month as far as changes that users will notice. However, huge progress was made in finalizing the version one Dmedia Hashing Protocol, including drafting a formal protocol specification. (See the FileStore 13.01 release notes for more details.)

Before finalizing the protocol, we'd like to have at least one independent implementation. Robert von Burg has started work on a Java implementation, and has already provided very valuable feedback on things that are unclear in the specification. Thanks, Robert!

Now all that sounds very geeky, but there is a good reason why we've put so much work into this part. I felt it was deeply important that Novacut work without the cloud, that it be easy to move assets between different cloud providers, and that ferrying hard drives across the sneaker-net be a first-class way to share the assets needed for collaboration.

Cloud services can be unprecedented traps for customer lock-in, and I don't want artists to check-out of one roach motel only to become permanent residents at another. Unless you can get your data out of the cloud, and you have access to the software needed to use your data without the could, you're boned.

Although it's not being used yet, there is a new software component this month: D-Base32. It's an experimental base-32 encoding designed for document-oriented databases. Unlike standard Base32 encoding, it preserves the sort order (so sorting by the binary IDs gives you the same order as sorting by the D-Base32 encoded IDs). We haven't decided yet whether we're going to switch, and we'd love feedback on this. Please see the D-Base32 Design Rationale if you're interested.

As always, if you want to get involved with Novacut design or development, please stop by the #novacut IRC channel on freenode and introduce yourself.

Special Thanks

Special thanks to Greg McQueen for taking the time to interview me for Digital Gleu:

Install Novacut 13.01

Please follow these instructions to install Novacut on Ubuntu 12.04 LTS (Precise), Ubuntu 12.10 (Quantal), or Ubuntu Raring (the development version). Note that if you've already installed a previous version of Novacut, you'll automatically get Novacut 13.01 the next time the Ubuntu Update Manager runs.

If you're trying Novacut for the first time, you probably want to start by watching this so you understand a bit about Dmedia, and then watch this to get a good tour of Novacut.

Source code

You can download the source code from each component's Launchpad project page: