CassaFS

Paul Dwerryhouse (paul@dwerryhouse.com.au)

Introduction

CassaFS is a FUSE-based filesystem using the Apache Cassandra distributed database system as a backend

Currently, CassaFS is experimental, proof-of-concept software. It has not been tested thoroughly, and definitely has bugs in the implementation of some calls, although it now passes all of the Tuxera POSIX Test Suite tests. Do not use it in production environments, even if it looks like it might be working. You WILL lose data. Also, I reserve the right to change the underlying database structure at any time, without providing an upgrade path.

CassaFS is licenced under the Free Software Foundation's General Public Licence, version 3. This means it is free software.

Known Limitations

• Due to a limit on the number of columns per row in Cassandra, the maximum number of files per directory is 2147483647. Similarly, the maximum file size is 2147483647 x 8192 byte blocks. I might see if I can use composite keys to get around this, at a later time.
• Even if the system is writing a full 8192 byte block to a file, CassaFS will still read the block from the database (in this particular case, it doesn't need to). So, it's slow.
• It's written in Python. So, again, it's slow. I will rewrite it in C at a later time.
• I can't think of any sensible way to return filesystem statistics with statfs, so it returns zero for almost everything. This means running the df command on this filesystem is useless.
• Similarly, the keyspace name is hardcoded, so this means only one CassaFS filesystem per server. This will be fixed when a run-time configuration file is added, too.
• There are write races; I'm attempting to use ZooKeeper to remove these.

Download

The source code for CassaFS is here.

Debian/Ubuntu deb packages are available below. The package is architecture independent, but I've only been able to package python-thrift (a required dependency) for amd64 and i386, at this stage.

Quickstart

If, after all the warnings, you still want to try CassaFS out, here's a quickstart for Debian and Ubuntu users:

Add the following lines to your /etc/apt/sources.list:

		deb http://www.apache.org/dist/cassandra/debian 11x main
		deb http://dwerryhouse.com/software/cassafs unstable main
	

Install cassafs and its prerequisites:

		apt-get update
		apt-get install cassandra python-zookeeper zookeeper python-pycassa python-thrift python-fuse python-cassafs
	

If not already started, run Cassandra (make sure it's listening on 127.0.0.1:9160, which should be the default):

		service cassandra start
	

Start Zookeeper (ensure that it's listening on port 2181):

		/usr/share/zookeeper/bin/zkServer.sh start
	

Put the following in your /etc/fstab:

		cassafs         /mnt            fuse    allow_other,noauto,attr_timeout=0 0    0
	

Make the Cassandra filesystem:

		mkcassafs 127.0.0.1:9160 CassaFS
	

Then mount the filesystem:

mount /mnt

Voila, it should now be working. You can copy files in, create directories, generally do most of the things you expect to be able to do with a POSIX-compliant filesystem (except for the small matter of those things that don't work yet).

When you fill up your Cassandra server, just throw more servers at it (fixing up the hardcoded ip address, of course, as noted above) and you'll be able to expand your filestore easily. There, your own personal - but incredibly unreliable - Dropbox.

History

Version 0.0.17 - 2013/05/07

CassaFS now passes all of the relevant Tuxera Posix Test Suite tests (that is, everything except for the chflags tests).

This does not mean that it's stable, however; there are definitely known bugs in the CassaFS handling of rename.

Version 0.0.12 - 2012/10/04

CassaFS now uses ZooKeeper to handle locking. Configuration file updated to handle this:

[main]
servers=10.1.8.209:9160,10.1.8.210:9160
keyspace=CassaFS

[zookeeper]
servers=10.1.8.209:2181,10.1.8.210:2181
	

Version 0.0.8 - 2012/10/01

New support for a configuration file - default name is /etc/cassafs.conf. If you want to use a different file, then use the config option in fstab:

		cassafs /mnt fuse allow_other,noauto,attr_timeout=0,config=/path/to/cassafs.conf 0    0
	

At this stage, the configuration file only takes two options: 'servers' (a comma-separated list of host:port entries for Cassandra servers), and 'keyspace', the keyspace where your filesystem data is stored. Example:

[main]
servers=10.1.8.209:9160,10.1.8.210:9160
keyspace=CassaFS