Monday, December 19, 2011

Misleading "Out of Memory" Error in SQLite3

(Please take a look at our crowd-funding campaign at

The following is recorded in the hope that it may save someone else the hassle of tracking down this kind of error.

While working on Serval 0.08 I have been working with SQLite3 for the first time.

I had the following fairly innocent query causing SQLite to report an "out of memory" error:

int count=sqlite_exec_int64("SELECT COUNT(*) FROM FILES WHERE id='%s' AND datavalid<>0;",hash); 

After spending a bit of time hunting everywhere to find where I could possibly have leaked enough RAM for SQLite3 to get upset, I finally realised that the column datavalid did not exist in the FILES table.  I had meant to update the CREATE TABLE statement, but obviously hadn't gotten around to it.

I updated the CREATE TABLE statement to include a definition for the datavalid column, and the "out of memory" errors are now a thing of the past.

I can only guess that somewhere in SQLite3 when it tries to find the column, it gets a NULL pointer returned, which some other part of itself then interprets as a failure to allocate memory to retrieve the column, rather than the failure of the column itself to exist.

Shopping Serval Style

We are currently preparing to provide the latest Serval mesh telephony technology for use in a disaster response training exercise with a major relief agency in late February 2012 -- about two months from now.  In that time we need to not only get our prototype Field Communications Unit (FCU) kit together, but also finish some software development that is key for making the technology useful for the relief agency.  You can watch a demo of one of the features we are working on in the video below, where I send a photo from one phone to another, without infrastructure.  This is the same technology we have used to send an SMS more than 10,000km without infrastructure.

On the hardware side, we have ordered 50 Huawei IDEOS X1 phones for AUD$71.10 each -- pretty cheap for a fully featured Android smart phone, complete with digital compass, GPS, accelerometers, a capacitive screen and enough CPU and RAM to meet our needs. These phones are network locked, but that doesn't worry us as we won't ever be putting a SIM card in them, because they will be used solely as mesh communications devices.

It turns out that buying 50 phones at a time is an interesting exercise.  People want to know that you are not planning to resell them or export them.  Some shops will even try to limit you to buying five or less at a time.  Those tactics are legal in some places, but not here in sunny South Australia, where we have an excellent Fair Trading Act that correctly classifies such terms of trade as discriminatory, or in some cases as tantamount to bait-and-switch.

In fact, until quite recently the South Australian consumer protection and gambling laws were so strong that cereal-packet competitions could not require a South Australian resident to purchase a product to participate in a trade promotion lottery, because it would be considered gambling, i.e., pay for chance to win.  So when I was growing up it was quite common to see terms and conditions looking quite normal, and then have sections specially for South Australians, that would often say something along the lines of:
"10. To enter, cut the bar code from a packet of XYZ and include it in your entry as proof-of-purchase.  Residents of South Australia are permitted to submit a hand drawn facsimile of the proof-of-purchase.
17. Only one entry per household per day.  Residents of South Australia may enter more than once per day."

There were certainly occasions in my childhood when we would go to the effort of getting the coloured pencils out to draw a barcode by hand to get the chance to win a prize.  I think we sometimes did it just because we could.  Sadly, with the introduction of poker machines (Australia has fully 21% of the worlds poker or slot machines --- a statistic that is disturbing given that Australia has only about 1% of the developed-world population) and the general liberalisation of gambling in South Australia the situation has changed, and I believe we have lost our little parochialism.

Anyway, I digress.

After the difficulties I encountered initially in sourcing 50 mobile phones, I eventually went to our local post office to see if they would order me 50, since they had the model I wanted in their catalog.  Not only were they willing to order them (I promised to pay up-front, since it was a large purchase), they also kindly sliced 10% off the price.  So I now have a box of 44 phones in my lounge room, with the remaining 6 on back order:

At this price, the phones don't come with SD cards, which we do need for the Serval Rhizome software to do anything useful, as store-and-forward requires somewhere to store!  So I hunted around, and after some luke-warm experiences with suppliers either being unwilling to give me a decent price for what is clearly a wholesale quantity, or failing to stock enough, I tried OfficeWorks.  They had enough in stock, but were charging $16 for 4GB cards, which I had quotes from other local stores for $7.  So I loaded my basket with 50 memory cards, and proceeded to try their price-matching policy:

However, the large price difference and quantity did trigger concern from their store manager.  He was very friendly and amiable, but did point out that to get the price matched, I had to find a local supplier with the quantity I wanted actually in stock.  But that wasn't easy to arrange, as most of the cheaper suppliers only had a few on hand.  There is of course a silliness in their rule, as I could have bought them in small batches such that each batch was small enough that the cheaper shop held sufficient stock.  But in the end I did manage to find a cheap supplier on eBay who had plenty in stock at $5.80 a piece for SANDisk 4GB class 4 microSD memory cards.  Now we just have to see if they will arrive.

So that covered the phones and memory cards.  The next challenge was to come up with a way to easily charge 50 phones.  Not only did it need to work in my home, but it also had to have the potential to work in a real disaster zone, and one which might not be in Australia.  Having enough power boards to plug 50 little USB charger bricks was not an attractive option, especially since I need to fit all of this into my 23kg luggage limit when I fly, and 50 phones alone are not that light.

What I have decided to do is to chop up a pile of USB leads and wire them up to some automotive 5v accessory power supplies that are commonly used to power portable DVD players and the like.  Apart from being the cheapest 5v high-current power supplies I could find (AUD$40 for 5A), they have the benefit of running on 12v, or in the case of some models, on 12v-24v, so that they can even work in a truck.  This means that in-vehicle mass-charging is possible when the power grid is off.  It also means that they can all be charged using a single cheap 13.8v power supply, which are also quite cheap; you could even use an old computer power-supply.  You can see my prototype here:

So now I just need to get back to writing the software that needs to go on these phones ...

Thursday, December 15, 2011

Serval Developer Release 0.07, and our Roadmap for 2012

It has been some time since our last developer release, and so it is with great pleasure that I am able to announce that Serval BatPhone Developer Release 0.07 is now available for download from the Android Market.

This version offers compatibility with many more models of Android handsets, fixes to a large number of bugs, and some general usability improvements.  Full functionality still requires root permissions on your phone, but it does its' best to work on phones without root access.

Personally, what I am most happy about is that we have substantially improved our software development and release process, which has already resulted in a much better quality in the current release, and I am confident will result in increasing quality in future releases.

In the longer-term this is the first of several planned releases that will hopefully bring us to a 1.00 public release around September 2012.  The planned intervening releases and their headline improvements, subject to resourcing, slippages and all the other usual hazards of software development are:

0.08 - Serval Rhizome/MeshMS store-and-forward file distribution core technology.  Automatic update of Serval software over the mesh.

0.09 -  Revamped and consolidated MeshMS API and functionality.  Integrated UI for sending and receiving MeshMS messages, including using the Rhizome store-and-forward facility.The underlying technology has already been demonstrated by sending an SMS between Africa and Australia without reliance on infrastructure.

0.10 - Revamped UI for entire Serval Mesh application.  Voicemail and Push-to-talk group communications.

0.11 - Serval Mapping Integrated (using Rhizome to get map tiles without internet access).  Voice calls moving from SIP+RTP via SipDroid and Asterisk to Serval MSIP and MDP, which will improve performance on lossy wireless networks, allowing clear calls over multi-hop links each with 50% packet loss, and greatly reducing the APK size by removing the full Asterisk deployment it currently contains.

0.12 - Consolidate existing work, and address outstanding issues for first public release.  0.12 will become 1.00RC1 when accepted.

Together, these releases will culminate in a software package that will let you do most mobile-phone type things (voice calls, SMS/MMS, voice mail, push-to-talk) when there is no infrastructure, or for free when infrastructure is available, such as transferring and sharing data among friends and community without having to pay up to $2/MB for SMS traffic or $2/MB for cellular data services.

This is good news for people with very low purchasing power, such as large portions of the population in developing countries as well as for those living in more developed countries such as Australia, where infrastructure is not universally present, especially for those living in rural and remote areas.

But for now, it is back to working on Serval Mesh release 0.08 ...

Thursday, December 1, 2011

Shuttleworth Foundation Intro Video

As many readers will know, my work on the Serval Project is substantially funded by the Shuttleworth Foundation.  They provide my income, thus freeing my time, co-investment in the project, and loads of support in a variety of ways, including a great team of other fellows and support staff.  

Without their support the Serval Project wouldn't be what it is.

The foundation are always on the look out for other people with world changing ideas, and have released a new explanatory video that describes how their fellowship model works.

So if you have an idea that might change the world given half a chance, then I would recommend getting in touch with them, and possibly applying for a fellowship.

Wednesday, November 23, 2011

TEDx Adelaide Talk Is Now On Youtube

As I mentioned in a previous post, I recently presented at TEDx Adelaide, which had the theme "On The Edge".  They have now put the videos up on YouTube, so here it is:

The bottom line? 

Just as MP3 files and digital music distribution has changed the market place for the recording industry, so to decentralised digital mobile communications is on the verge of similarly disrupting the mobile telecommunications industry, by similarly adding value to the everyday experience of many people. 

The question is, will the mobile carriers see the opportunities that change brings, or will they work against their own interests by failing to embrace the change?

Only time will tell.

Tuesday, November 22, 2011

Demonstrating the Serval Rhizome Store-and-Forward MeshMS SMS Service

(Please take a look at our crowd-funding campaign at

Update: We trialled Rhizome on a training exercise in New Zealand recently.  Follow the link to read the blog post about it.

SMS messages on a cellular network get delivered via the cell towers and message centre infrastructure.

On a mesh, it is possible for us to deliver messages to other phones that are reachable on the mesh at the same time.  This works great, and we have had this capability in the Serval BatPhone software for a while now.

However, it does have some limitations, in particular, if there is no link all the way from the sender to the receiver on the mesh at the instant the message is sent.  This is rather unfortunate, as we usually think of SMS as being the most reliable fall-back on a mobile communications network.

So we set about replicating this resilience by creating a store-and-forward SMS-like service on top of the Serval Rhizome mesh file distribution framework.  We call this MeshMS, for Mesh Message Service.

This service uses direct mesh links if they are available by making use of the existing SMS capability in the Serval BatPhone software.  However, if there is no direct link, then it uses a store-and-forward scheme, that asks any passing phones to copy the message and distribute it to other phones on the mesh, until it (hopefully) eventually reaches its intended destination.  The cartoon below shows how this works.

The man wishes to send the message "Prepare the Zeppelin" to his hench-person, but there is no direct mesh link at that time due to the challenging topology.  The message gets picked up (stored) by the compatible phone in the pram, without any action on the part of the woman.  The woman then keeps walking along, and eventually the phone in the pram is able to automatically deliver (forward) the message to the ultimate destination, whereupon the minion knows to ready his master's zeppelin, to do the weekly grocery shopping, one trusts. Of course, there could be many intermediate steps, instead of just the one shown here.

The great thing about this approach is that it doesn't require a complete path to the destination at the time of sending, but can propagate progressively across the mesh as it is able, and can make use of nodes that move between otherwise isolated mesh networks, creating an asynchronous link between them where it would not otherwise be possible.

While the delay in such a service is huge, the bandwidth is also great, as potentially gigabytes of data can be transferred between nodes.  One side use of this protocol that we intend to exploit is to provide an efficient means of distributing updates to the Serval software suite, so that field updates can occur without dependence on infrastructure, and with much greater aggregate bandwidth than any single-cast cellular or wireless approach.  In fact, this will even allow the update of software, maps and other resources during a disaster.

We have even demonstrated it to deliver an SMS message between South Africa and Australia, using nothing but compatible phones to carry the message more than 10,000km:

This feature has profound utility in allowing the exchange of messages, files, and all manner of data in what are otherwise very difficult settings.

It also provides the capabilities required to enable disadvantaged communities, perhaps in remote locations, war zones or informal settlements, to create their own infrastructure, carrier and cost-free SMS networks using compatible handsets, so that they can enjoy the benefits of digital communications that many of us take for granted.

Monday, November 21, 2011

Building The Serval BatPhone Software

We have been putting some significant effort into making the Serval BatPhone software easier to build from scratch, especially first time around.

The procedure for Mac/Linux has now been reduced to (assuming you want the wifi-autodetect branch, which for now you probably do):

1. git clone git:// -b wifi-autodetect
2. cd batphone
3. ./BUILD.txt

After that, you should be able to compile the software from in Eclipse without difficulty, as all the messy stuff has been taken care of (git sub-modules, JNI/NDK compilation, NaCl preparation, among others).

This easy build process will be pushed up to the master branch fairly soon, but for now, it is only available with the wifi-autodetect branch.

Note that building on Windows remains unsupported, as end of line markers in files get messed up if great care is not taken.  If you want to build on Windows, we recommend you install a Linux virtual machine, e.g., using the free VirtualBox software, and run the build process from in there.

Sunday, November 20, 2011

A Serval Project Progress Report

It is time to reflect on the past six months of Serval, and where we intend to head in the next six months.

Six months ago we did not have a developer software offering, or even a single Android application that contained all the components for Serval. Today we do.

On-site in South Africa after testing the Serval MeshMS store-and-forward SMS Service, delivering files and SMS messages up to 11,000km without infrastructure, or a continuously connected mesh.
Six months ago we did not have MeshMS (Mesh SMS service), interactive mapping,  Rhizome file distribution or store-and-forward SMS working.  Today, we do, and have even sent photos and SMS messages more then 10,000km between Australia and South Africa using mobile phones as the only infrastructure (more on this in a blog post and video in the next few days).  See this blog entry for an infographic and video showing the sending and receiving of this first inter-continental mesh SMS message.

Demonstration of the Serval Rhizome Mesh-Based Infrastructure-Free File Distribution System, which is also the basis of the store-and-forward MeshMS/SMS service.

Six months ago we had a number of legal issues to work through with regard to giving Flinders University an appropriate holding in the Serval Project's commercial arm, and the Shuttleworth Foundation had not yet come on board.  Today, we have just about finalised the details for these arrangements, with the final execution to occur in the coming weeks.  It will be fantastic to get this settled.

TEDxAdelaide Presentation November 2011. (Video coming soon) “The democratisation of online music is one example where the internet has eroded the monopoly power of music labels returning some portion of that power to the general public, without bankrupting those enterprises.” Paul will explore what is essentially a Napster equivalent that will compete with the mobile telecommunication giants. He believes that the advent of powerful, programmable, portable digital devices (e.g. smartphones) are a key factor in this eventual democratisation of mobile telecommunications.
Six months ago we were not greatly recognised internationally for our work on mesh communications.  Today, we are increasingly recognised as the leaders internationally in this space, which is a tremendous honour and also acts to spur us along to deliver on all the potential that mesh networking offers.  Recognition has included support by the Shuttleworth Foundation (South Africa), reaching the finals of the World Embedded Software Competition for University students (South Korea), strong engagement by and with the IEEE 802.11 standards process (USA and international), presenting at the Adelaide Festival of Ideas and TEDx Adelaide (Australia), and also reaching the finals of the Ashoka Foundation World Changer's Citizen Media competition (international) from a field of more than 400 entrants.  We are also seeing university and other partnership opportunities in Australia, Europe, the Middle East, Africa, the Americas, and Asia begin to take shape.  Now all we need is to be able to clone ourselves a few times over so that we can engage fully with the various opportunities!

Our plans for the next six months are to fill out the feature set that we have generated (voice, SMS, MMS, interactive mapping, file and software distribution), and add in the missing pieces, primarily the security and authenticity components that we have planned from the outset.  This will take us to mid-2012 when we hope to focus on maturing all of this into a general public release, as compared to the early-access developer software we have released to date.

Adelaide Festival of Ideas, 2011, JIM BETTISON ORATION: KICKING THE INFRASTRUCTURE HABIT: [listen to MP3]  "Modern communications systems use extensive and expensive infrastructure to deliver services we could only dream of a few decades ago. This works for those who enjoy peace and sufficient wealth, but fails to reach the last billion people in poorer countries, as well as those in remote, emergency or disaster situations. Now modern mobile phones have the potential to communicate directly, to form networks without reliance on any infrastructure. The Serval Project based at Flinders University is turning this dream into a reality. It is working to make communications available to everyone, anywhere, any time especially to those who need it most."

We are also looking seriously about getting advanced mesh support built into one or more models of mobile phone to offer energy efficiency and/or range improvements over the basic service.

Simultaneously, we are beginning to explore appropriate trials of the technology that if they go ahead will gain us valuable feedback, and refine the software to maximise it's utility for the general community, including in South Africa (assisting in the distribution of educational material without cost or dependence on cellular or school IT infrastructure), North America and Australia.

So all in all, the next six months should keep us fairly busy.

Wednesday, November 9, 2011

Serval is in finals for the Ashoka Foundation Change Makers Citizen Media Competition!

It seems that this last month has been one of the Serval Project gaining international recognition as leading the mesh communications space.  First some thoughts on that, and then some of the exciting things that have been happening.

This is incredibly gratifying and humbling, and I think also says something about the value of the open development and free software model, and also about the value of pragmatism and choosing the right applications and approach.  Ultimately it must spur us onto continuing to advance not only the technology, but also its use.

We are quite aware that there are other more experienced players in this space, who have in some cases invested many millions of dollars into mesh networking.  However, the closed-handed commercial approach that many such group have taken means that many innovations in the mesh networking space do not get well known, and that people feel isolated from what is happening in this space -- which is quite counterproductive, since mesh is by it's nature a word-of-mouth and peer-to-peer technology, and that character bleeds over into how it is best used, valued, evangelised and adopted.

We are also quite aware that there have been significant efforts put into making robust mesh routing protocols and standards, for example 802.11s, and yet in a short space of time we seem to be accelerating our traction at a much faster rate.  I think that there are a few interrelated causes for this.  First, we are extremely pragmatic in our approach, and this is partly enabled by us driving a vision forward, rather than seeking consensus to generate a moderated output.

The unfortunate reality for mesh networking is that it is still in such infancy that the various components have to be well matched to perform well, and thus the traditional compromise approach to standards generation, without an working example to inform the standard, is prone to produce a sub-optimal result, and take a very long time to produce.

This is also related to the lack of clear commercial application (or more specifically, of commercial return) of mesh networking in the eyes of many companies.  This is because when viewed as a traditional network, mesh networks are very poor cousins; they have poor reliability, poor bandwidth, poor quality of service, limited range and so on. All these make it very difficult to layer existing commercial models over a mesh network.  For example, adapting Google's advertising model onto a mesh network has trouble getting the advertisements over the mesh to each device, let alone the search data.

It takes a fresh approach to mesh networking to understand where the substantial commercial opportunities are hiding.  This requires gaining an understanding of the strengths of mesh networks, rather than their weaknesses.  The natural broadcast nature of wireless networks actually works well for broad dissemination of data.  The ability of these networks to operate without supporting infrastructure allows the provision of services when not ordinarily possible.

These services do not need to be data-poor services, either.  Serval has already created a prototype mapping application that could easily support navigation, geotagged advertising by businesses, crowd sourcing of points of interest and a variety of other features -- and would continue to be available when infrastructure is not available.

In this kind of setting, the limitations of mesh networking become strengths. The limited range of mesh networks means that locally relevant data is available where it is relevant, and doesn't flood the rest of the world or distant parts of the mesh.  The broadcast nature of wireless communications allows multiple receivers to receive these locally relevant data, whether it be advertisements, maps, communications between nearby friends or otherwise.  Also, by focussing on hop-by-hop dissemination of data, the traditional bandwidth degradation of mesh networks can be significantly avoided -- at the extreme it is single-hop communications which can be performed at the order of 100mbit between nearby WiFi enabled devices, which is way faster than my home ADSL internet connection.

But back to some of the recent successes we have had in our work and leadership being recognised, including the headline of this post:

The Serval Project is one of 11 finalists from a field of around 500 entries in the Ashoka Foundation's Change Maker Citizen Media competition.  This is tremendous recognition of the value of our work in the citizen empowerment space, and we are delighted to engage with this community, and with the other amazing finalists.  We'd also love you to vote for us if you think we should be one of the winners.  You can see the other finalists and vote using the links below:

We have also made it to the finals of the World Embedded Software Competition in Korea, for the second year running.

Meanwhile, we have begun to engage with the IEEE 802.11 standards process, with Romana in Atlanta at their meeting as I type.  It continues to amaze me the generally positive reception and voice that we are granted at events such as this, and we are hopeful that we can effect positive change in the standards to better support mesh networking.

We are also beginning to form partnerships with Universities, companies, not-for-profit and other entities, not just in Australia, but around the world.  More on some of those as they mature, but it is exciting for me to see our efforts multiplied as we gain partners.  As I have previously mentioned, we are keen to not only create resilient software, but produce it in a resilient manner, and that means with multiple teams in multiple countries around the world.  While this process has begun, we would love to expand this even further, so if you are interested, drop me a line.

Closer to home, but from my perspective very satisfying is that two of our students won their sections of the departmental end of term student expo.  Congratulations to both Swapna Palaniswamy and Corey Wallis for your great work. Swapna's work has also resulted in an upcoming publication at the International Conference on Wireless Technology for Humanitarian Relief later this year, which is an unusual achievement for a student in her course.  Corey has similarly had a presentation based on his work accepted  at Linux Conference Australia in January (we also have another talk at LCA on our Rhizome mesh file distribution protocol).

So it is a tremendously exciting time, and we are incredibly grateful to the Shuttleworth Foundation, Flinders University, NLnet and the Awesome Foundation for their enabling support of our activities.

So now it is back to driving the technology side of our activities so that we can deliver on the promise.

We have two primary areas of activity on this front at present.  First, we are working to consolidate the existing software to make it generally more usable and stable.  Second, we are working on incorporating the Rhizome file and messaging system so that we can provide what we think will be a very powerful mesh messaging service, that will even make possible the delivery of messages when there is no path immediately available through the mesh, and for the sender to receive confirmation when the message has been received.  But more on that when we have something concrete to show.

Tuesday, November 8, 2011

A Simple Illustration of A Multi-Hop Telephone Call

From time to time I get asked to provide a simple explanation of what the Serval Project actually does.

It allows you to make multi-hop phone calls using only phones, with the signal bouncing from phone to phone 

What does a multi-hop mobile mesh telephone call look like?  

Well, it can look a bit like this:

Of course this photo is a bit staged, and in reality the people in the middle of the call don't have to have their phones out, or do anything -- the mesh routing software automatically works out how to get the call between the parties.  

The only limitation is that their be a path between the calling parties.

Also, the phones can be quite a bit further apart than shown here, around 100 - 200m with line of sight at present, and potentially kilometres apart once we get the ISM915 band meshing happening in the future.

In a future post I will explain how we can enable SMS type services, even when there isn't a reliable path.

Tuesday, October 25, 2011

Early Prototype of the Serval Mesh File Distribution System, the Rhizome Retriever

Romain one of our team has been working hard to get the Rhizome Retriever together as part of our work in supporting citizen journalism in a community in Nigeria, that I have blogged about previously.

They are already using our software in that community to make free mesh telephone calls, but from the outset the plan has been to enable them to be citizen journalists, and to share news and information in their community.

To enable this, we are making our Rhizome Retriever (Serval RR), which basically lets you pick files on your phone, and put them up on the mesh, from where they will be automatically distributed to other phones on the mesh.

Mass file distribution on a mesh is usually fraught with difficulties, not the least of which is that the available bandwidth on a mesh halves (or worse) for each extra hop that you try to traverse.

So you can often get decent bandwidth to nearby nodes, but poor or intermittent connection to more distant nodes.

Also, the more people trying to transfer things many hops across the network reduces the available bandwidth for everyone.

So we decided to make the Serval RR only ever send data one hop to the nearest neighbours.  They can then in turn send data to their neighbours later on at their convenience.

As a result, Serval RR can already distribute even relatively large files, certainly into the several mega-byte range, without great difficulty, and without making the network unusable for everyone else.

Here is a quick demo of the initial prototype of the Serval RR software in action:

Sunday, October 23, 2011

The importance of communications during emergencies

Early 2011 saw the worst floods in Queensland, Australia in 40 years.

Not surprisingly, this placed significant stresses on response efforts, and for a variety of reasons, the Queensland Government has issued an interim report on the event and what responses should be made.

You can read the interim report here.

I have been reading through this from a telecommunications perspective, and found the following items:

2.24  Seqwater should give consideration to posting information about current and future releases on its website during flood events as one method of ensuring accurate and timely information is available to the public. 

This may seem like a small matter, but by making the information available somewhere where someone in the community could access it, and possibly disseminate it further, perhaps by word of mouth, or perhaps by feeding it into a crowd-source incident reporting system such as Ushahidi or the Serval Rhizome/Mapping combination, where it could be made available to others who need it.  Indeed, this process could be automated, providing a very valuable community service.  However, if the information is not shared in the first instance, then such broad and means of distribution will be troublesome to implement, and potentially disadvantage many.

In short, putting information out where the public can get to it, and make further use of it supports resilience.

5.21 The Queensland Fire and Rescue Service should ensure that rescue technicians on deployment are provided with individual radios, rather than sharing a communications pack

Presumably the sharing of communications packs is related to their cost and bulk.
Using off-the-shelf meshing mobile phones would allow all rescue technicians to have a personal means of communications that makes use of the shared communications pack, relaying traffic back through that resource.

5.29 The Queensland Fire and Rescue Service should consider isolating repeaters during a large scale emergency response. If this solution is found to be feasible, it should be implemented as protocol as soon as possible. If it is not, the Queensland Fire and Rescue Service should explore other solutions to the issue of the fire communications network being overloaded and firefighters resorting to localised networks during large scale emergency response situation.

This seems to be responding to the issue of excessive traffic being relayed across the region, congesting the system and preventing it being of effective use.  The Serval Mesh system has the potential to localise message delivery to the local teams that need it (whether or not they are locally based), and still retaining the ability to broadcast messages over the whole region when suitably addressed.

This is not a particular mode we had considered greatly in the past, so this is a helpful thing for us to discover.

7.2 Lockyer Valley Regional Council should investigate the feasibility of installing alarm-activating gauges in the creeks at Spring Bluff, Murphys Creek and other communities where communication systems are poor and there is a risk of rapid and unexpected water rise.

This is interesting for us to read, as we are already contemplating creating a very similar system for use in the Mekong River Basin on conjunction with a Lao doctoral student here in the department.  The characteristics of the system are having a local mesh broadcast an alarm to all mesh nodes when one or more mesh-equipped sensors indicates a high water level.

It is possible that those sensors could be low-cost mobile telephones connected to a device such as the IOIO, creating a very simple and low-cost solution, provided the device can be powered, perhaps using a solar panel and modest storage battery, such as an old car battery no longer able to supply the peak load required to start a car, but more than sufficient for the couple of watts required to run a phone and sensor.

"SMS alerts are also not a reliable method of providing flood warnings in parts of Queensland which experience problems with telephone coverage. That difficulty is compounded during a flood, when telephone reception can be affected by flood related power outages and congested telecommunications networks."

This is precisely the kind of short coming with cellular networks that Serval has sought to find and create solutions for, that complement the existing infrastructure.  In particular, the ability of the Serval Rhizome mesh file/message distribution system to work asynchronously, and automatically share information carried in phones as they more from region to region has particular value in these sorts of situations.

Also, the ability place local calls on the mesh, and where possible, setup connections from the mesh to the outside world offers a significant capability that is otherwise a considerable liability in these situations.

The ability for Serval technology to allow the rapid setup of a local mobile telephone network, and then connect it to the global telecommunications network using the nearest functional mobile phone tower was demonstrated in Brisbane in late January 2011, just a couple of weeks after the flooding had subsided.

We did this by placing a phone running the Serval software at a high vantage point where it could communicate with the nearest functional mobile phone tower (we used a helium balloon, but a roof top, or even a long fishing pole gaffer taped to a car would suffice).  This then allowed the phones on the ground that were also running our software to relay calls out to the world.  Here is the video we took of us doing exactly this, and watching people use it:

Saturday, October 22, 2011

Serval is going to Atlanta

Well, Romana (co-founder of Serval Project) is.  And that is because she will be representing the Serval Project's interests at the IEEE 802 plenary meeting there in just a couple of weeks (November 2011).

Romana will be putting forward use cases that reveal deficiencies in the current 802.11 family of WiFi standards for mesh and ad-hoc communications.

If all goes well, we may have the opportunity to input into a process of looking to address these issues, which is tremendously exciting for us.

It could mean much longer range mesh hops using ordinary mobile phones, by using the cellular radio to implement some kind of WiFi in the ISM 915MHz band.

It could also spell the end of frustrating incompatibilities in the 802.11 ad-hoc WiFi standard, including the bizarre lack of specification of ad-hoc in 802.11n, preventing standards conforming devices from using the higher speeds and other benefits that 802.11n brings.

Either or both of these have the potential to make citizen generated infrastructure-free communications systems mainstream, thus our excitement.

So Atlanta will be the official first turning of clod in a process that could extend up to four years, and represents a huge personal investment on the part of Romana, as well as a substantial financial investment on the part of the Serval Project, and thus the Shuttleworth Foundation whose support makes this possible.

We would invite any other open efforts that have concerns with aspects or omissions of the IEEE 802.11 WiFi standards to get in touch with us, so that we can do our best to represent the needs of more than just ourselves.

Tuesday, October 18, 2011

Oh the fun of the Android linker

So this week I decided that I would finally do the initial integration of the excellent NaCl crypto library into DNA, but with the view of making it available from Android so that it can be used as part of our forthcoming Rhizome browser (this is for browsing Rhizome offerings, not the web -- more on this thing in a future post).

My main objective was to have only one copy of the NaCl library in our Android application so that the APK stays as small as possible, and so that we don't waste any more memory than we need to.

The first challenge is that we want to access NaCl from Java, as well as the command-line dna utility.

The second challenge is that the crypto code should be inseperable from the dna code, so that it is impossible, for example, to use linker tricks to switch the NaCl library for something insecure.

We can't do this perfectly because it needs to be accessible from Java, which can only load native shared libraries, but it seems that we should do what we can.

So I put all of dna and NaCl into a single shared library,, using the Android NDK to build it, so that we can load it into Java and use the NaCl (and possibly dna) functions directly from in Java.
This even means that dna's main() function is in there, and so we can run dna queries and actions from in Java as though we were using the command line tool.


Then I tried a bit of good old fashion linker skullduggery, by making an empty C file, and compiling it linked against, so that it gets dna's main() from that library, by doing (in effect):

echo >null.c
gcc -o dna null.c -ldnalib

However, the linker on Android doesn't look in the host applications lib directory when linking an executable.  The Android linker also doesn't support the -Wl,-rpath= option for specifying where to find a library.

This is a real pain, and probably qualifies as a bug.  I should probably file the bug, but I wanted an immediate solution, since my past attempts at getting the Android folks to fix things have not met with much success.  Besides, I needed a solution that works with existing handsets, not future ones. So in the words of Marvin the Martian, it was back to the old drawing board.

So my solution was to create a wrapper that just dlopen()'s, and then finds and runs the main() function from in there:

#include <dlfcn.h>
#include <stdio.h>

int main(int argc,char **argv)
 void *h = dlopen("/data/data/org.servalproject/lib/",RTLD_LAZY);
 int (*dnamain)(int,char **) = dlsym(h,"main");
 if (!dnamain) return fprintf(stderr,"Could not load\n");
 return (*dnamain)(argc,argv);


The downside is that it assumes that /data/data/org.servalproject is where the application will get installed, which the Android people tell us we should not assume.

Thus I will probably be about as popular with them as I am when I tell people to root their phones to get ad-hoc mode WiFi instead of waiting for an API to turn up in some future release of Android.

But, when necessity beckons, one must do what one must do.

And so I now have a solution that works and I can move onto more pressing problems.

Thursday, October 13, 2011

Deploying a Phone Tower, Serval Style (The Day In Pictures)

So yesterday we went to beautiful Normanville with the goal of testing the maximum range between phones running our mesh telephony software.

We didn't get particularly far on that front, but we did discover some very interesting things about the unique requirements of mesh routing between mobile telephones, which will likely be the subject of a upcoming blog post.

But what we did do was deploy a mobile phone tower of a different kind.  Romana has wanted to make her wheel chair into a mobile phone tower for some time, and yesterday was her chance, as the pictures show.

Romana in her chair with Lyn beginning to attach the 7 metre long squid pole we use as the mast.

Further attaching of the mast.

Beginning to raise the mast.  The blob on the top is a Huawei IDEOS U8150 mobile phone running our software and acting as the "repeater".

 The raising of the mast:

Victory! Romana's wheel chair is now a mobile phone tower.
 But it wasn't all hard work.  The weather was glorious, and the views most excellent, and others were out and about enjoying the day ...

Or hoping for a chip.

Meanwhile, we moved to the jetty so that we could place our mobile phone tower (including blanket for the cool sea breeze).

The pole really does stand up quite high. It gets the phone about 6m above the deck of the jetty:

... which is a little higher than even the lamp post

Then we started walking along the beach towards Carrickallinga, slowly deploying a 6-hop mesh, and making multiple simultaneous voice calls amongst the nodes.

Corey and Romana did a great job standing on the jetty and talking to us via our impromptu phone network, while locals and visitors enjoyed the day:

This was also the first time that my son Caleb came along on a Serval expedition.  Little did he know that his pram was soon to become a phone tower as well.

Here I am with Lyn deploying a node some distance from the jetty:

Then it was Jeremy's turn:

Then the pram:

And all the while we were able to make calls among the various nodes.

We did encounter some serious mesh routing issues, which as I mentioned I will talk about soon, but for now I just wanted to let you all enjoy the pictures.

Wednesday, October 12, 2011

Kicking The Infrastructure Habit

Last weekend I delivered the inaugural Jim Bettison Memorial Oration at the biannual Adelaide Festival of Ideas. This was a great honour, and my presentation was well received by the audience, as was evidenced by the poignancy of their questions.

Here is the abstract of my oration:
Modern communications systems use extensive and expensive infrastructure to deliver services we could only dream of a few decades ago. This works for those who enjoy peace and sufficient wealth, but fails to reach the last billion people in poorer countries, as well as those in remote, emergency or disaster situations. Now modern mobile phones have the potential to communicate directly, to form networks without reliance on any infrastructure. The Serval Project based at Flinders University is turning this dream into a reality. It is working to make communications available to everyone, anywhere, any time especially to those who need it most.

You can listen to the oration by downloading the MP3 from the Radio Adelaide website.

You can also listen to an ABC Bush Telegraph interview about the oration that I recorded a few days prior to the oration.

Wednesday, October 5, 2011

Photos and Update from Nigeria - and introduction to Rhizome

We have a partner organisation working in Nigeria looking at citizen-sourced journalism and using our mesh network as the connectivity among a number of camera-equipped cell phones, so that citizen journalists there can make a short video or news piece and then have it distribute automatically over the mesh using our Rhizome protocol.

Rhizome is our mesh file distribution protocol that allows user-created content and software updates to spread hop-by-hop over the mesh.  This hop-by-hop approach elegantly solves many of the problems with mesh-wide file transfer, in particular bandwidth starvation and instability of long links.

We have called it rhizome because it is like the vegetable reproductive process of the same name, because we think it is a good analogy for the way that Rhizome allows data to spread by small hops, but such that over time it can spread over great distances.

We are busy working on making Rhizome do all that the team in Nigeria needs, but in the meantime, the team in Nigeria have sent us some photos showing the mesh in action.

First, they sent photographic evidence of 10 IDEOS U8150 phones forming a mesh with our software:

10 Serval BatPhones on the mesh (and a call in progress!)
They also sent us this picture which I really love, because it shows a community getting started with out technology thousands of kilometres from where we are writing the software here in Australia.  This is what it is all about, people using and benefiting from the software that we are creating and releasing.

Finally, just a week after they got 10 phones on the mesh, they sent this shot showing 15 phones on the mesh.  This was just a case of them getting 15 phones within range, as the underlying BATMAN mesh protocol should be able to handle a couple of hundred devices without any difficulty:

One of the challenges we are looking at solving with this group is auto-update of the mesh software, over the mesh.  That is, we want to be able to load a signed software update onto any one of the phones on the mesh, and have the phones on the mesh share it automatically with one another.

This is a great idea, and the Rhizome protocol will allow us to do this, however it introduces some security challenges.

In particular, we don't want someone or some party seizing or cracking our signing key, and then being able to spread a trojan update that destroys the Serval mesh.

One approach we are looking at is making the update require that the new version be signed by a majority of keys from a pool, where each key would be held by a different mesh-friendly organisation.   The pool of trusted keys could then be similarly updated by a majority vote of existing trusted keys.

This will allow us to spread the keys among countries and jurisdictions and provide the kind of distributed resilience that the Serval Project is predicated on, and even allows the software to continue to be refined and distributed if the Serval Project as an organisation ceases to exist.

If you think that your organisation might like to be one of our key custodians, let me know.

Thursday, September 22, 2011

More than we expected at the IEEE

If you have been reading this blog you will know that we sent Romana over to Okinawa to present our desire to see ad-hoc mode wifi improved, and also to consider how to use the baseband radio in mobile phones to support wifi on very low cost phones, and also to provide much longer range meshing using the ISM915/868 bands that offer very useful link budget advantages over the 2.4GHz band used by regular WiFi.

Well, we have been absolutely gobsmacked by the interest and positive response that Romana experienced at the meeting, and have the opportunity of taking up the baton to champion proposing these enhancements through the formal IEEE 802 standards process.

This is a great privilege, and also an opportunity that cannot be accepted without serious and solemn consideration, because it will basically require someone's effort, full time, for perhaps three years or more.

We are convinced that we want to, and that we should engage with this standards process, but just need to work out whether we have the resources to do it, without drawing all of our available resources away from progressing the development of the Serval mesh telephony software.

More news as we have it.

Tuesday, September 13, 2011

Why Serval is getting involved in the 802.11ah standards process

Serval is getting involved in the IEEE 802.11ah (WiFi on ISM bands below 1GHz) to try to make sure that it is well suited to infrastructure-independent ad-hoc and mesh networking at the IEEE meeting in Okinawa next week.

Here is our submission:

Basically we are asking for two things:

1. Improvements to ad-hoc mode (or provision of a new "packet radio" mode) that remove some of the problems currently faced when creating wifi-based ad-hoc mesh networks.


2. That the 802.11ah standard consider speeds below 1mbit and using cell phone baseband radios as a supported transport so that even cheap cell phones can form relatively long range mesh networks without any supporting hardware.

This second point is really important, because compared with 2.4GHz WiFi a mesh running in the ISM 915MHz band gains about +9db just from the change in frequency, which alone improves range by almost 3x.  If it supports lower bit rates, then further significant gains are possible, e.g., allowing 100kbit communications gives another +10db, for a total of 8x range versus WiFi.

There are some significant protocol challenges to be addressed, but if the standard doesn't support the use-case, then there will be no hardware and no chance.

These measures have the potential to push the indoor range up from WiFi's "about one house wide" to "about a block wide" and clear line-of-sight range up to a few km, which suddenly makes the formation of suburban ad-hoc mesh networks possible, which has profound impact for creating resilient infrastructure-independent communications solutions, for example for sustaining communications during disaster or enabling communications for rural and remote or developing populations.

We admit that we are very green to this process which we frankly find to be daunting.

But we feel compelled to try, regardless of what we perceive to be the odds of success.

Monday, September 12, 2011

More progress on the overlay mesh network

It is 5:58am and dawn is just beginning.

I have been up all night making the best of the need of an overnight sleep study for our son, which requires an adult to perform a specific function every 15 minutes throughout the night.

Programming in these little pieces feels a bit like the software development version of watching commercial television; just as you think you are about to witness something really interesting there is an interruption.

Nonetheless, I have made significant progress getting the overlay mesh to send and receive advertisements of a different kind; notices by one node that they can reach another node.

There are plenty of bugs that I know about, without even thinking about the ones that I don't know about, and also some code that is still plain old missing.  But even so, I have got the code forming a multi-hop mesh and keeping track of the routes to them.

I ran two nodes connected to two separate dummy networks, so that they couldn't see each other.  Then I connected an extra node in the middle that bridged the two networks, and after a short delay, viola, the nodes could all see each other.  The image shows the routing table from one of the nodes with only one network interface.

The 0000000* addresses in the neighbour table are clear evidence of some of those bugs I mentioned, as is the crazily large age field for one of those.

The FDB3BA1*, D64CEB4* and B167503* addresses are nodes that I had temporarily added to the mesh, and then removed again fairly recently, so the routes were (correctly) still hanging around with their link scores steadily dropping.

The important part is the route to D6850E3*, the node on the other network, for which the current route is via 30464E5*, which is the node bridging the two networks.  30464E5* was a node that I had previously bridging the networks just a short while earlier, and we can see the lingering route as it decays.

B167503* is the node currently bridging the link, and it's score is still climbing, and is about to overtake the recently removed node and become the route of choice.

So while I have a lot more work to do, the basic function of the mesh to discover multi-hop paths and switch routes based on appropriately calculated scores.

But right now, I am going to bed.

Why User-Land Mesh Networks Are Necessary For Heterogeneous WiFi Mesh Networks

The normal approach to creating a mesh network is to write a program that does some tricky stuff to work out how to get from node A to node Z via whatever nodes might be in the middle, and then tell this to the operating system so that packets get routed according to these instructions.

We are doing something different for Serval, in that we are having the program that does all that tricky stuff carry the messages between nodes, instead of asking the operating system to help out.

This has some disadvantages and some advantages.

First, the disadvantages, and why we think that they aren't such a problem for our application.

1. Such "user-land" overlay networks suffer from an inefficiency known as "double-copy", where every packet sent or received ends up getting copied not once by the kernel, but twice, once by the kernel, and once by the overlay program.  This is bad if you are processing lots of data.  But on a mesh network the actual throughput of data tends to be fairly low (it is one of the well known problems with mesh networks).  So this problem is kept under control.

2. Programs that want to know about the mesh network need to know about the overlay and have to use special calls to talk to the mesh.  This is bad if your primary goal is allowing people to run BitTorrent, SSH, for example, because you need to modify each of those programs to support the new network, and without operating system cooperation that can get a little messier than you might like.

However, for a mesh telephony platform, and where we know that bandwidth starvation is one of the greatest potential problems, there is actually merit in requiring people to go to some effort before they enable their application to consume the limited resources of the mesh.  We will be adding the support to the initial core applications we intend to run over the mesh, so this isn't really a big deal.

Now for some advantages:

1. We don't need operating system cooperation.  This makes it easier to port the mesh to all operating systems, both desktop and mobile.  It is especially important because the need to fiddle with routing tables requires root/administrative access on most platforms.

For mobile platforms, this is certainly true on Android, and I believe iOS, with together constitute the majority of smart phones.  This alone means that we have to take this path.

But it also gives us some additional upside here, in that it means we could make a USB key loaded with Serval mesh software that could be run on any desktop computer, and without administrative access or being "installed" could enable that computer to participate in the mesh.  In fact, it could probably even be an HTML5 application running in a web browser.

2. It separates us from the underlying network addressing. Many devices and networks only support IPv4, but there aren't enough IPv4 addresses to go around. But by pushing our mesh network addressing into a dedicated addressing layer we can solve this problem.

We use 256bit Elliptic Curve cryptography keys as the basis for our address

In fact, using this approach it is possible for every single Serval enabled device on a mesh to all have the same IPv4 address, provided we use only broadcast traffic, which we do for a very important reason:

Broadcast traffic is the only traffic which WiFi ad-hoc can reliably carry between devices with incompatible or buggy ad-hoc implementations.  For example, we cannot get unicast traffic to flow reliably between Mesh Potatoes and Huawei IDEOS U8150/U8180 phones.  However, we can get unicast traffic to flow between Mesh Potatoes and the original Android G1 developer phones, and between the G1 and the U8150/U8180 phones.

Unfortunately mesh routing protocols all tend to use broadcast traffic to discover available routes, and thus the routes choose the impossible Mesh Potato-U8150/U8180 links, which is very bad indeed.  But by making the mesh traffic itself broadcast we ensure that what the mesh routing protocol and what we are trying to do is the same, instead of merely similar.

Taking this a little further, by transporting the mesh topology management information in the same packets as the data we can directly measure the performance of the mesh delivering the traffic we really care about, rather than using different packets as an indirect measure -- especially since the distribution of packet sizes can otherwise be very different, and WiFi packet losses are not insensitive to packet length or differentiation between broadcast and unicast traffic.

Implementing this is of course a significant undertaking, and I have already spent a few weeks programming away.  But the effort is beginning to yield results.  Today for the first time I have the new overlay mesh detecting nodes and building the in-memory route table. You can see a screen grab here:

So here you can see some debug output from a Serval DNA node running in overlay mode using a file called dummy0 as a dummy network interface.

I realised early in the development of the overlay network that I would need some convenient simulation tools to make it easier to debug the behaviour of the mesh.  I decided the best way to do this was to make the software itself also a network simulator.  So I added support to pretend an ordinary file is a network interface.  Any instances of the overlay pointed to the same file would all append their "packets" to the end of the file, and remember where they had got up to reading from the file, and keep an eye out for any packets deposited by other nodes.

Each packet also has a header that has space for location information, transmission power, time, node velocity and heading and a whole pile of other information that can be used to drop packets that are deemed to have been sent from too far away, or suffering collision during reception and so on.

The file then also represent a log of all activity on the mesh which has already proven very helpful for debugging.  I will also create a tool that allows for the creation of visualisations of the mesh in action to help understand why things go wrong, and to explain the operation of the mesh.  I will post such animations here once I get that far.

Back to the trace, the error messages are mostly information for me for development (although there is on link-list bug that I can see I will need to fix to avoid a memory leak).

There are a few 256bit addresses that can be seen.  The all-f's is the mesh broadcast address, while the one that starts with 9d5f is the address of a node on the mesh.

These addresses are really big, and so I have already implemented a variety of mechanisms for abbreviating the addresses.

For example, the broadcast address is used so often, that it has a one-byte abbreviation (0x0f).  For other addresses, we only need to transmit as much of the address is required for the receiver to discriminate it from all other nodes.  The birthday paradox comes into play here, and means that we typically need about twice the number of bits as the base-2 log of the number of nodes on the network.  For a small network of a few hundred nodes 24 bits (3 bytes) of address is a pleasant over-kill.  Even at a global scale, we need to accomodate around 10 billion addresses = ~2^34 addresses, and so about 70 bits (7 bytes) should be enough.

We also have abbreviation strategies where neighbouring nodes that talk with each other a lot can allocate a one-byte short code for each other, however, as you might be able to guess from the trace that abbreviation method is temporarily disabled in the code to keep my life a little simpler at present -- which is a good move because of the density of new features in the code.

At the end of the trace we see a summary of the overlay routing table which shows 7-digit address prefixes for all know nodes on this network, and the link-scores to each via the various routes known to each.

The scores have two components. The first being a BATMAN-like score derived from beacon counting, although we take into account that the remote node might have different interfaces running at different beacon rates.  The second part is the number of Serval gateways that are on the path.

This gateway count is there to prevent routing loops and other nasties in meshes where the overlay has managed to infer some hierarchy, or where inter-mesh links have been purposely provisioned.

I'll talk about inferred hierarchy in another post, but it is one of several measures we are taking to ensure that a Serval mesh can scale beyond the normal bandwidth-starvation limit imposed by all nodes trying to know about all other nodes all of the time.

Our premise is that you need frequent information for nearby nodes because their "bearing" can change wildly over short time frames, but that for more distant nodes their "bearing" on the mesh should change much more slowly, and so we should be able to cope with less frequent updates about the position of such distant nodes.

But meanwhile, as you can see we are making good progress, and the Serval DNA overlay mesh is beginning to come together.

Thursday, September 8, 2011

A Thermo-Electric Saucepan?

(Please take a look at our crowd-funding campaign at

Today we received our thermo-electric saucepan from TES New Energy. You can see some pictures of it below.

This amazing device provides a full-powered USB port when placed on a fire by using thermo-electric generation.

The interesting innovation is that by incorporating the thermo-couples into a saucepan the water in the pot keeps one side of the junction cool to maximise the power output, while ensuring that the about 98% of heat that the thermo-couple cannot turn into electricity heats your water.

This also means that they can use different types of thermo-couples that work better at different temperatures.  The hot-side of the pan uses lower-efficiency thermo-couples that can tolerate the higher temperatures, and the "cold" side of that is joined to the hot side of the lower-temperature thermo-couple which is also a little more efficient, and the water in the pan helps to make sure that this lower-temperature thermo-couple does not over heat.  Together, they  avoid the complications that would normally be present if using only one of the thermo-couples, either lower yield, or the potential to be damaged by higher-temperature fires.

The saucepan approach is also an elegant way to circumvent the atrocious inefficiency that typically plagues thermo-electric generators, since in a diaster or remote location you probably need hot water, anyway, a realisation that TES NewEnergy had following the mega earthquake that struck Japan recently.

So this saucepan is about enough to charge a phone if you are in the middle of nowhere, and solar isn't an option for whatever reason -- perhaps because the power is out due to storm, volcano, or just during bad weather.

Anyway, we are looking forward to some fun testing Serval mesh phones powered by a high-tech saucepan on a campfire.