At LCA 2012 (Linux Conf AU)

This year I was not expecting to be heading to Linux Conference Australia (LCA), as two of the Serval Project team were expecting to be there, and I am already looking at the likelihood of somewhere north of six bouts of international travel this year, on top of a very full itinerary for myself in our software development program, especially in the first half of the year.

However, circumstances changed just before LCA2012 with one of our team unable to travel due to health, and another being overseas at an IEEE 802.11 meeting as part of our effort to get the ad-hoc WiFi standard improved. So I found myself flying in for just over 24 hours to present one of the talks.

The two Serval Project talks this year were covering Serval Rhizome, our store-and-forward message and file distribution system, and the Serval Mapping Service, an infrastructure-free map and crowd-sourced information gathering and visualisation platform that is something like a cross between GoogleMaps, Ushahidi and Twitter.

Update: Here are the videos of the talks.  The authorships are wrong way around, however:

The talks were well attended and many people asked a wide variety of questions. Several themes that emerged in the question time were that we should think about how we might interface our work with that of the ham radio community, and a general discussion of the Open Street Maps project that we use to supply the cacheable maps for the Serval Mapping Service. Questions were also asked about whether we were looking to branch out from Android to other platforms, including low-cost “feature phones”, which are common in the developing world. We have been targeting the Symbian/Belle S60 platform for a long time now. We also explained how we intend to seek the manufacture of custom feature and smart phones that contain a pluggable extra radio for mesh communications.

This goal received a substantial boost when we discovered that Andrew Tridgell (of SAMBA fame) has already written frequency-hopping firmware for HopeRF radio modules similar to the ones we are intending to use. This will likely save us months of effort, and also enabled us to learn from Andrew's experience about what the realistic capabilities of the modules are. The firmware and adapter boards that Andrew has created will also make it much easier for us to quantify the transmission range that we can achieve with these devices in practise, rather than relying on fairly simplistic link-budget calculations.

Also from talking with Andrew we have begun to look more seriously about using small model drone aircraft to loft mesh repeaters (possibly just mesh-enabled Android phones) and have them loiter over an area to provide significant coverage where it would otherwise be possible.

The combination of the longer-range radios and an established ability to loft mesh devices each have the potential to greatly increase the utility of our technology in a variety of settings, including disaster response and in both built-up areas and open country where either obstacles or low population density make WiFi meshing difficult.

It was also great to hear about David Rowe's continuing progress on his open-source codec, called Codec2, which while still alpha, already is able to demonstrate veryacceptable performance at 2500bps and 1400bps, i.e., as low as 175 bytes per second for speech of a quality that is more than acceptable for a phone call, and is substantially better than the now rather aged GSM codec that requires almost 10x the bandwidth. The challenge is how to make effective use of such a low bit-rate coded, since as David pointed out during his talk, the IP, UDP and RTP headers in a typical VoIP application would consume probably 5x more bandwidth than the voice stream itself.

This is a welcome problem we have been anticipating for some time now, and have some thoughts on how we might address it. First, once we get our new overlay mesh operating, we will use packet aggregation to defray some of the headers. Second, we will setup pseudo virtual circuits between nodes that will allow the voice to be directed using one-byte channel identifiers that are local to each node-pair. Third, we can build more efficient transport structures once we branch out to custom radio interfaces, such as in the ISM915 band building on the work that Andrew Tridgell described earlier in this post.

For example, we could use a one-byte packet type identifier plus the one-byte channel identifier described to provide an overall header size of just two bytes. Add to this perhaps 2 more bytes of synchronisation data for the cryptography layer, and the total overhead is 4 bytes per 40ms, for a total of 100 bytes of overhead, and a total data requirement in each direction of 275 bytes per second, or just under 2400bps. Allowing for guard bands and the like on the radio interface, we should be able to easily accommodate a full duplex voice call over a 5kbs – 10kbs radio channel, and multi-hop voice calls over perhaps 20kbs – 30kbs*. Such a low bit rate, as I have explained previously, offers the potential of very long range compared with WiFi, probably hundreds of metres per hop in built-up urban areas, and perhaps a few kilo-metres in open country, and even better with an elevated transmitter. We do hope to quantify these range figures in some realistic situations over coming months.

(* Each channel into and out of each node requires about 2400bps. In a multi-hop network, we do not want any two adjacent, two-hop or three-hop neighbours transmitting at the same, as they will cause interference to each others (the interference range is typically about double the effective transmission range, so one node talking can be heard by its immediate neighbours, and cause interference to it's two-hop neighbours). Also, we have two voice streams, one each way, so we need to schedule this in an appropriate manner. Let us explore how this might work if we have four nodes, A, B, C and D, involved as successive hops in a multi-hop call.

At time 0 transmits to B (A → B), so B must be listening, and C cannot talk, else it would interfere with B's reception of A's transmission. In fact, even D cannot talk, for fear of interfering with B's reception of A's transmission, as B may be in the interference range of D. Thus in this entire system of four nodes to maximise range we need to have only one of the four nodes talking at any time. As it takes three hops to carry the audio in each direction, we need six time slices. Some savings can, however, be gained by having B talk to A and C at the same time with a double-length frame, and similarly C talk to B and D with a double length frame, thus saving two guard bands. Thus we need two 2400bps channels and two 4800bps channels (a total of 14400bps), plus four guard bands. At 20kps this would leave approximately 25% of air time available, including for use guard bands, or to allow for the occasional carriage of additional data.

At 30kps fully half of air time would be available, which would also allow for some forward error correction, and a more realistic allowance for additional data, especially if some creative time slicing and signalling is employed. For calls with more than four hops, additional bandwidth is not required, as nodes more than three hops from one another have reasonable prognosis (though no guarantee) of not interfering with one another.)

Meanwhile, also at LCA, we have been demonstrating the ability of the Serval software to share itself via bluetooth and WiFi to other phones, thus simulating deployment in a disaster zone or rural/remote context where internet access is not available. While there have been some issues, it has been tremendous to see many people install and try our software. It seems that practically everyone at LCA this year is aware of Serval and what we are trying to do.

As part of my talking about the Serval Mapping Service, Jeremy and I setup a “scavenger hunt” using the Serval Mapping Service to place geotagged pins on the map and Serval Rhizome to distribute photographs of the location of each small 2.5cm square cardboard token that we hid around the sprawling Ballarat University Campus. Similar to geocaching, the tokens were placed so as to be near impossible to find without knowing their location. LCA attendees can use the Mapping and Rhizome software to locate the tokens and return them to Jeremy. Each token is redeemable for two free coffee vouchers to help encourage participation. Apart from a bit of fun, we hope that this will provide us with some real-life experience and feedback on the use of these technologies.

Jeremy also succeeded in setting up an configuration-free gateway between Serval Mesh phones and a free SIP to PSTN service that is running at the conference, and left some voice mail on my cell phone to prove it.

We also had some interesting contact with several businesses that may be able to make use of our technology.

Much credit must go to Jeremy for getting the software together, and engaging in many of the conversations that have enabled us to have such a successful time at LCA so far. Also Rob Thomas has been a tireless advocate of Serval at LCA and elsewhere, and together with Jeremy have helped many people to install our software on their phones at the conference this year.

Thus, while I was not expecting to attend LCA this year, I am very glad that I managed to get there.

Demonstrating the Serval Rhizome Store-and-Forward MeshMS SMS Service

(Please take a look at our crowd-funding campaign at igg.me/at/speakfreely.)

Update: We trialled Rhizome on a training exercise in New Zealand recently.  Follow the link to read the blog post about it.

SMS messages on a cellular network get delivered via the cell towers and message centre infrastructure.

On a mesh, it is possible for us to deliver messages to other phones that are reachable on the mesh at the same time.  This works great, and we have had this capability in the Serval BatPhone software for a while now.

However, it does have some limitations, in particular, if there is no link all the way from the sender to the receiver on the mesh at the instant the message is sent.  This is rather unfortunate, as we usually think of SMS as being the most reliable fall-back on a mobile communications network.

So we set about replicating this resilience by creating a store-and-forward SMS-like service on top of the Serval Rhizome mesh file distribution framework.  We call this MeshMS, for Mesh Message Service.

This service uses direct mesh links if they are available by making use of the existing SMS capability in the Serval BatPhone software.  However, if there is no direct link, then it uses a store-and-forward scheme, that asks any passing phones to copy the message and distribute it to other phones on the mesh, until it (hopefully) eventually reaches its intended destination.  The cartoon below shows how this works.

The man wishes to send the message "Prepare the Zeppelin" to his hench-person, but there is no direct mesh link at that time due to the challenging topology.  The message gets picked up (stored) by the compatible phone in the pram, without any action on the part of the woman.  The woman then keeps walking along, and eventually the phone in the pram is able to automatically deliver (forward) the message to the ultimate destination, whereupon the minion knows to ready his master's zeppelin, to do the weekly grocery shopping, one trusts. Of course, there could be many intermediate steps, instead of just the one shown here.

The great thing about this approach is that it doesn't require a complete path to the destination at the time of sending, but can propagate progressively across the mesh as it is able, and can make use of nodes that move between otherwise isolated mesh networks, creating an asynchronous link between them where it would not otherwise be possible.

While the delay in such a service is huge, the bandwidth is also great, as potentially gigabytes of data can be transferred between nodes.  One side use of this protocol that we intend to exploit is to provide an efficient means of distributing updates to the Serval software suite, so that field updates can occur without dependence on infrastructure, and with much greater aggregate bandwidth than any single-cast cellular or wireless approach.  In fact, this will even allow the update of software, maps and other resources during a disaster.

We have even demonstrated it to deliver an SMS message between South Africa and Australia, using nothing but compatible phones to carry the message more than 10,000km:

This feature has profound utility in allowing the exchange of messages, files, and all manner of data in what are otherwise very difficult settings.

It also provides the capabilities required to enable disadvantaged communities, perhaps in remote locations, war zones or informal settlements, to create their own infrastructure, carrier and cost-free SMS networks using compatible handsets, so that they can enjoy the benefits of digital communications that many of us take for granted.

Serval is in finals for the Ashoka Foundation Change Makers Citizen Media Competition!

It seems that this last month has been one of the Serval Project gaining international recognition as leading the mesh communications space.  First some thoughts on that, and then some of the exciting things that have been happening.

This is incredibly gratifying and humbling, and I think also says something about the value of the open development and free software model, and also about the value of pragmatism and choosing the right applications and approach.  Ultimately it must spur us onto continuing to advance not only the technology, but also its use.

We are quite aware that there are other more experienced players in this space, who have in some cases invested many millions of dollars into mesh networking.  However, the closed-handed commercial approach that many such group have taken means that many innovations in the mesh networking space do not get well known, and that people feel isolated from what is happening in this space -- which is quite counterproductive, since mesh is by it's nature a word-of-mouth and peer-to-peer technology, and that character bleeds over into how it is best used, valued, evangelised and adopted.

We are also quite aware that there have been significant efforts put into making robust mesh routing protocols and standards, for example 802.11s, and yet in a short space of time we seem to be accelerating our traction at a much faster rate.  I think that there are a few interrelated causes for this.  First, we are extremely pragmatic in our approach, and this is partly enabled by us driving a vision forward, rather than seeking consensus to generate a moderated output.

The unfortunate reality for mesh networking is that it is still in such infancy that the various components have to be well matched to perform well, and thus the traditional compromise approach to standards generation, without an working example to inform the standard, is prone to produce a sub-optimal result, and take a very long time to produce.

This is also related to the lack of clear commercial application (or more specifically, of commercial return) of mesh networking in the eyes of many companies.  This is because when viewed as a traditional network, mesh networks are very poor cousins; they have poor reliability, poor bandwidth, poor quality of service, limited range and so on. All these make it very difficult to layer existing commercial models over a mesh network.  For example, adapting Google's advertising model onto a mesh network has trouble getting the advertisements over the mesh to each device, let alone the search data.

It takes a fresh approach to mesh networking to understand where the substantial commercial opportunities are hiding.  This requires gaining an understanding of the strengths of mesh networks, rather than their weaknesses.  The natural broadcast nature of wireless networks actually works well for broad dissemination of data.  The ability of these networks to operate without supporting infrastructure allows the provision of services when not ordinarily possible.

These services do not need to be data-poor services, either.  Serval has already created a prototype mapping application that could easily support navigation, geotagged advertising by businesses, crowd sourcing of points of interest and a variety of other features -- and would continue to be available when infrastructure is not available.

In this kind of setting, the limitations of mesh networking become strengths. The limited range of mesh networks means that locally relevant data is available where it is relevant, and doesn't flood the rest of the world or distant parts of the mesh.  The broadcast nature of wireless communications allows multiple receivers to receive these locally relevant data, whether it be advertisements, maps, communications between nearby friends or otherwise.  Also, by focussing on hop-by-hop dissemination of data, the traditional bandwidth degradation of mesh networks can be significantly avoided -- at the extreme it is single-hop communications which can be performed at the order of 100mbit between nearby WiFi enabled devices, which is way faster than my home ADSL internet connection.

But back to some of the recent successes we have had in our work and leadership being recognised, including the headline of this post:

The Serval Project is one of 11 finalists from a field of around 500 entries in the Ashoka Foundation's Change Maker Citizen Media competition.  This is tremendous recognition of the value of our work in the citizen empowerment space, and we are delighted to engage with this community, and with the other amazing finalists.  We'd also love you to vote for us if you think we should be one of the winners.  You can see the other finalists and vote using the links below:

We have also made it to the finals of the World Embedded Software Competition in Korea, for the second year running.

Meanwhile, we have begun to engage with the IEEE 802.11 standards process, with Romana in Atlanta at their meeting as I type.  It continues to amaze me the generally positive reception and voice that we are granted at events such as this, and we are hopeful that we can effect positive change in the standards to better support mesh networking.

We are also beginning to form partnerships with Universities, companies, not-for-profit and other entities, not just in Australia, but around the world.  More on some of those as they mature, but it is exciting for me to see our efforts multiplied as we gain partners.  As I have previously mentioned, we are keen to not only create resilient software, but produce it in a resilient manner, and that means with multiple teams in multiple countries around the world.  While this process has begun, we would love to expand this even further, so if you are interested, drop me a line.

Closer to home, but from my perspective very satisfying is that two of our students won their sections of the departmental end of term student expo.  Congratulations to both Swapna Palaniswamy and Corey Wallis for your great work. Swapna's work has also resulted in an upcoming publication at the International Conference on Wireless Technology for Humanitarian Relief later this year, which is an unusual achievement for a student in her course.  Corey has similarly had a presentation based on his work accepted  at Linux Conference Australia in January (we also have another talk at LCA on our Rhizome mesh file distribution protocol).

So it is a tremendously exciting time, and we are incredibly grateful to the Shuttleworth Foundation, Flinders University, NLnet and the Awesome Foundation for their enabling support of our activities.

So now it is back to driving the technology side of our activities so that we can deliver on the promise.

We have two primary areas of activity on this front at present.  First, we are working to consolidate the existing software to make it generally more usable and stable.  Second, we are working on incorporating the Rhizome file and messaging system so that we can provide what we think will be a very powerful mesh messaging service, that will even make possible the delivery of messages when there is no path immediately available through the mesh, and for the sender to receive confirmation when the message has been received.  But more on that when we have something concrete to show.

Early Prototype of the Serval Mesh File Distribution System, the Rhizome Retriever

Romain one of our team has been working hard to get the Rhizome Retriever together as part of our work in supporting citizen journalism in a community in Nigeria, that I have blogged about previously.

They are already using our software in that community to make free mesh telephone calls, but from the outset the plan has been to enable them to be citizen journalists, and to share news and information in their community.

To enable this, we are making our Rhizome Retriever (Serval RR), which basically lets you pick files on your phone, and put them up on the mesh, from where they will be automatically distributed to other phones on the mesh.

Mass file distribution on a mesh is usually fraught with difficulties, not the least of which is that the available bandwidth on a mesh halves (or worse) for each extra hop that you try to traverse.

So you can often get decent bandwidth to nearby nodes, but poor or intermittent connection to more distant nodes.

Also, the more people trying to transfer things many hops across the network reduces the available bandwidth for everyone.

So we decided to make the Serval RR only ever send data one hop to the nearest neighbours.  They can then in turn send data to their neighbours later on at their convenience.

As a result, Serval RR can already distribute even relatively large files, certainly into the several mega-byte range, without great difficulty, and without making the network unusable for everyone else.

Here is a quick demo of the initial prototype of the Serval RR software in action:

The importance of communications during emergencies

Early 2011 saw the worst floods in Queensland, Australia in 40 years.

Not surprisingly, this placed significant stresses on response efforts, and for a variety of reasons, the Queensland Government has issued an interim report on the event and what responses should be made.

You can read the interim report here.

I have been reading through this from a telecommunications perspective, and found the following items:


2.24  Seqwater should give consideration to posting information about current and future releases on its website during flood events as one method of ensuring accurate and timely information is available to the public. 


This may seem like a small matter, but by making the information available somewhere where someone in the community could access it, and possibly disseminate it further, perhaps by word of mouth, or perhaps by feeding it into a crowd-source incident reporting system such as Ushahidi or the Serval Rhizome/Mapping combination, where it could be made available to others who need it.  Indeed, this process could be automated, providing a very valuable community service.  However, if the information is not shared in the first instance, then such broad and means of distribution will be troublesome to implement, and potentially disadvantage many.

In short, putting information out where the public can get to it, and make further use of it supports resilience.



5.21 The Queensland Fire and Rescue Service should ensure that rescue technicians on deployment are provided with individual radios, rather than sharing a communications pack


Presumably the sharing of communications packs is related to their cost and bulk.
Using off-the-shelf meshing mobile phones would allow all rescue technicians to have a personal means of communications that makes use of the shared communications pack, relaying traffic back through that resource.



5.29 The Queensland Fire and Rescue Service should consider isolating repeaters during a large scale emergency response. If this solution is found to be feasible, it should be implemented as protocol as soon as possible. If it is not, the Queensland Fire and Rescue Service should explore other solutions to the issue of the fire communications network being overloaded and firefighters resorting to localised networks during large scale emergency response situation.


This seems to be responding to the issue of excessive traffic being relayed across the region, congesting the system and preventing it being of effective use.  The Serval Mesh system has the potential to localise message delivery to the local teams that need it (whether or not they are locally based), and still retaining the ability to broadcast messages over the whole region when suitably addressed.

This is not a particular mode we had considered greatly in the past, so this is a helpful thing for us to discover.


7.2 Lockyer Valley Regional Council should investigate the feasibility of installing alarm-activating gauges in the creeks at Spring Bluff, Murphys Creek and other communities where communication systems are poor and there is a risk of rapid and unexpected water rise.


This is interesting for us to read, as we are already contemplating creating a very similar system for use in the Mekong River Basin on conjunction with a Lao doctoral student here in the department.  The characteristics of the system are having a local mesh broadcast an alarm to all mesh nodes when one or more mesh-equipped sensors indicates a high water level.

It is possible that those sensors could be low-cost mobile telephones connected to a device such as the IOIO, creating a very simple and low-cost solution, provided the device can be powered, perhaps using a solar panel and modest storage battery, such as an old car battery no longer able to supply the peak load required to start a car, but more than sufficient for the couple of watts required to run a phone and sensor.

"SMS alerts are also not a reliable method of providing flood warnings in parts of Queensland which experience problems with telephone coverage. That difficulty is compounded during a flood, when telephone reception can be affected by flood related power outages and congested telecommunications networks."


This is precisely the kind of short coming with cellular networks that Serval has sought to find and create solutions for, that complement the existing infrastructure.  In particular, the ability of the Serval Rhizome mesh file/message distribution system to work asynchronously, and automatically share information carried in phones as they more from region to region has particular value in these sorts of situations.

Also, the ability place local calls on the mesh, and where possible, setup connections from the mesh to the outside world offers a significant capability that is otherwise a considerable liability in these situations.

The ability for Serval technology to allow the rapid setup of a local mobile telephone network, and then connect it to the global telecommunications network using the nearest functional mobile phone tower was demonstrated in Brisbane in late January 2011, just a couple of weeks after the flooding had subsided.

We did this by placing a phone running the Serval software at a high vantage point where it could communicate with the nearest functional mobile phone tower (we used a helium balloon, but a roof top, or even a long fishing pole gaffer taped to a car would suffice).  This then allowed the phones on the ground that were also running our software to relay calls out to the world.  Here is the video we took of us doing exactly this, and watching people use it:

Photos and Update from Nigeria - and introduction to Rhizome

We have a partner organisation working in Nigeria looking at citizen-sourced journalism and using our mesh network as the connectivity among a number of camera-equipped cell phones, so that citizen journalists there can make a short video or news piece and then have it distribute automatically over the mesh using our Rhizome protocol.

Rhizome is our mesh file distribution protocol that allows user-created content and software updates to spread hop-by-hop over the mesh.  This hop-by-hop approach elegantly solves many of the problems with mesh-wide file transfer, in particular bandwidth starvation and instability of long links.

We have called it rhizome because it is like the vegetable reproductive process of the same name, because we think it is a good analogy for the way that Rhizome allows data to spread by small hops, but such that over time it can spread over great distances.

We are busy working on making Rhizome do all that the team in Nigeria needs, but in the meantime, the team in Nigeria have sent us some photos showing the mesh in action.

First, they sent photographic evidence of 10 IDEOS U8150 phones forming a mesh with our software:

10 Serval BatPhones on the mesh (and a call in progress!)

They also sent us this picture which I really love, because it shows a community getting started with out technology thousands of kilometres from where we are writing the software here in Australia.  This is what it is all about, people using and benefiting from the software that we are creating and releasing.

Finally, just a week after they got 10 phones on the mesh, they sent this shot showing 15 phones on the mesh.  This was just a case of them getting 15 phones within range, as the underlying BATMAN mesh protocol should be able to handle a couple of hundred devices without any difficulty:

One of the challenges we are looking at solving with this group is auto-update of the mesh software, over the mesh.  That is, we want to be able to load a signed software update onto any one of the phones on the mesh, and have the phones on the mesh share it automatically with one another.

This is a great idea, and the Rhizome protocol will allow us to do this, however it introduces some security challenges.

In particular, we don't want someone or some party seizing or cracking our signing key, and then being able to spread a trojan update that destroys the Serval mesh.

One approach we are looking at is making the update require that the new version be signed by a majority of keys from a pool, where each key would be held by a different mesh-friendly organisation.   The pool of trusted keys could then be similarly updated by a majority vote of existing trusted keys.

This will allow us to spread the keys among countries and jurisdictions and provide the kind of distributed resilience that the Serval Project is predicated on, and even allows the software to continue to be refined and distributed if the Serval Project as an organisation ceases to exist.

If you think that your organisation might like to be one of our key custodians, let me know.