Initial testing of the new Mesh Extenders with NZ Red Cross in Dunedin

As I have hinted at previously, I went to Dunedin with NZ Red Cross last month to participate in their field exercise there, and to bring the new Mesh Extender prototypes with me for initial socialisation and testing with the NZ RC IT&Telecommunications Emergency Response Unit.

The exercise was run over a weekend, with myself and the ERU arriving a couple of days in advance to establish VHF communications and internet access in advance for them.  Here is the white board listing the task list.  Serval Mesh and RAMP+ (another project we have with NZ RC) are on the list! (I'll talk a bit more about RAMP+ later in this post).

To give a feel for the exercise, here are some shots from around the Emergency Operations Centre (EOC), which was located in a Scout camp site near Dunedin.

First, internet from two places, the satellite dish in the foreground providing VSAT internet, and the ubiquiti router (easier to see in the second image below) pulling in internet from a nearby wireless ISP.  The ISP link required activating and a fair bit of work to get going.  The ERU folks then setup an automatic fail-over between them.  The mast in the field to the right was for VHF radio. They also setup a couple of repeaters on nearby hills.

Here is the communications desk in the EOC itself, staffed by the wonderful ERU folks in their very nice new jackets that Kathmandu kindly donated:

Then here I am sitting down trying to flash and debug some teething problems on the Mesh Extenders with Steve from the ERU to the left:

Here is Andrew from the ERU taking a Mesh Extender and battery pack walk-about as part of a multi-hop over UHF test.

And another of the ERU out during the exercise collecting data using RAMP+, which is a combination of Magpi.com and the Succinct Data extreme compression and satellite uplink tool designed and built by Matthew Lloyd and ourselves, which can exceed 99% compression when uploading the XML records for Magpi, making satellite SMS practical as a transport.

The week turned out to be extremely busy for me, as the Mesh Extender prototypes were still very new, and I only managed to get them packed in time, and the firmware was very raw, and required considerable work to get to the point of having the Mesh Extenders booting quickly and reliably, including formatting their microSD cards.

This was complicated by problems we encountered with RAMP+, where the interface between the Succinct Data application on the tablets and the Garmin inReaches would break if tracking were enabled. Used separately, both work fine, but together a bug in the inReach communications library for Android gets tickled, causing an uncatchable force-quit.  This combined with a minor comedy of errors in trying to get access to the Succinct Data server back home at the University (it was now after close of business in Australia on the Friday), culminating in completely reimplementing the Succinct Data server on Amazon Web Services (for which it is much the better) from in the field, consuming a valuable 36 hour period, which I really needed to use to get the Mesh Extender firmware shaken down.

The result was that we got RAMP+ mostly operational, but the Mesh Extenders had a couple of firmware issues that prevented us from completing any multi-hop UHF tests, which was disappointing.  We also quickly discovered a critical bug in the new Serval Chat application for Android, which rendered the application unresponsive on first load in certain circumstances.  This was of course rather disappointing for myself and the ERU team, as we had hoped to be able to carry out more sophisticated and extensive tests.

The good news is that we have since isolated and corrected the faults that we encountered in both the Mesh Extenders and Serval Chat, and are now gearing up to making an updated firmware image in preparation for the next visit to Vanuatu, and, hopefully, with enough time before landing to perform more extensive testing (in the process we now have more extensive test cases for the firmware image, which means that we are able to automate regression testing of the faults we encountered in NZ).  But before I do that, I need to write about the first visit to Vanuatu...

Prototype Mesh Extender Fly-away kit with NZ Red Cross

As part of the exercise in NZ with NZ Red Cross (NZRC), I pulled together the first attempt at a Serval Mesh fly-away kit for humanitarian use by NZRC.  The contents will be sure to evolve over time, so my focus was just on getting the Mesh Extenders safely in the box for transit, and all of the cables, tools and other bits and pieces that I would be likely to need while in NZ, and later in Vanuatu.

First step was to start with the big Zargs case, which is the standard fare used by NZRC, and make sure it would be big enough:

Step 2 was to get some foam sheets of various thicknesses, so that I could keep everything safe in there, so that enthusiastic baggage handlers wouldn't be a problem.  Clark's Rubber in Australia is the convenient retailer for that sort of thing.  Of course, retail price is not ideal, and it cost about AU$400 for enough foam to fill the box completely.

From one of the 75mm thick layers, my wife and I cut the holes to fit the eight prototype Mesh Extenders that currently exist, as can be seen below, while unpacking the kit in NZ:

Thinner foam layers were above and below in the box, and had solar panels, cables and tools interleaved among them.

My feeling is that we should be able to easily fit 16 units and two solar panels and all associated cables and bits and pieces in one of these cases without great difficulty. If we forgo the solar panels, then 24 or 32 would be possible.  However, that is: (a) a lot of Mesh Extenders; and (b) you still need some way to power them.  

Thus my feeling is that 8 or 16 units per case, with a richer set of accessories will be the norm. This also allows each fly-away kit to be more affordable, and for Mesh Extenders to be more easily distributed, rather than having to split the contents of a single case.

With a 40W and 20W regular glass solar panel, the eight Mesh Extenders and other parts, the whole thing weights <23kg, making it easy to ship as an extra bag by commercial airline, which is an important design goal.

One thing I did discover when traveling with a huge metal box that says "Emergency Response Unit" in large friendly type, is that it didn't exactly have the same effects as the word "Don't Panic" on the Hitch-hiker's Guide to the Galaxy.  Basically they viewed it as being Not Personal Effects and wanted to see appropriate paperwork on reentry to Australia, to see if they could charge me import duties.

Because the kit was only out of the country briefly, it is in principle possible to export and re-import without any customs liabilities, which I managed, however, if I am going to do it confidently repeatedly in the future, there is a pile of magic paperwork that I need to explore.  Exporting to Vanuatu for the pilot similarly requires some special paperwork on the Vanuatuan side, to ensure we are able to use their customs exemption for foreign aid projects.  Another whole interesting world that I am slowly becoming acquainted with...

Injection-moulded Mesh Extender cases

It has been a bit delayed with everything going on to report it, but we have the new Mesh Extender prototypes in their shiny injection-moulded cases. Here is Mesh Extender 2.0 serial number #1 (thanks to Rachael from Second Muse for the shots):

These prototypes are almost, but not quite complete.  The main outstanding issue is that the D-SUB connectors are not the correct IP67 ones with the seals around them.  Thus, while they are ostensibly complete, with the revision 4 PCBs, they are not weather proof enough for deployment.  We will fix this with the upcoming revision 5 PCB.

Also, when those shots were taken, we didn't have the correct reverse-SMA connectors with built-in seals to seal the antenna inputs. I am still not at ease that they will seal completely.  The other niggling concern I have is that humid air will pass through the goretex seal on the bottom (the black disc in the top image), i.e., only liquid phase water will be excluded. I hadn't previously realised this limitation.  Our current approach to these problems is to apply a conformal coating to the revision 5 PCBs, so that with the housing limiting water ingress, and hopefully excluding the ants and other foreign bodies, that the conformal coating will be up to the task.  I am confident it will all work out in the end, but this is representative of the many details that come into play when trying to make hardware, and especially if it is to be used in a hostile environment, such as in the middle of a tropical jungle.

I'll also shortly post about their first excursions to New Zealand to a Red Cross exercise, and to Vanuatu, as the first stage of our pilot there.