Thursday, July 6, 2017

First visit to Vanuatu

It has been a rather busy couple of months, so I am only just catching up with sharing what we have been up to. All going well, there will be a burst of posts describing our recent activities.

But first in this post, I want to talk about our first visit to Vanuatu as part of our Pacific Humanitarian Challenge (PHC) award from the Australian Department of Foreign Affairs and Trade (DFAT).

Basically, DFAT have commissioned us to run a pilot of the Serval Mesh in Vanuatu, to be concluded and reported on by the end of 2017. I can assure you, that that timeline is feeling very tight right now.

The lead up to the first visit was busy with getting the very first prototype Mesh Extenders built.  The circuit boards were still under active development, as we tried to fix a few niggling issues around the USB port and power/radio port pinout. We were also waiting on the first injection-moulded shells for the Mesh Extenders.

Part of the busy-ness was also that we had timed our first visit to Vanuatu to coincide with the Pacific ICT Days, which gave us a very firm deadline.

Part of our motivation in being there for the ICT days, was that I would be able to present to representatives from a number of nations in the region, to raise awareness and seek input from them on what we are doing.  This broader socialisation of Serval is one of the goals of the PHC award, to help gauge the regional interest and demand for our work.

Another motivation in being there for the ICT days was to be able to see what other initiatives are underway in Vanuatu, and work out who we could partner with for the pilot project.

We also arranged to stay for about a week after, to allow plenty of time to meet with various stake-holders.  Some meetings were pre-arranged, but the majority were arranged once in country, as this is still the easiest way.  I really can't emphasise how important it is to allow time in-country when working in Pacific island nations.  You can often achieve in a week or so, what would otherwise take a month or more to try to arrange remotely. Finally, the time in country gave us the chance to better understand the local context, and begin to understand the local culture first-hand.

Now to follow the visit in pictures:

 First, it was presentations at the Pacific ICT Days.  Matthew Lloyd, who has been our key contact in NZ Red Cross for almost seven years has resigned and is now a volunteer with NZ Red Cross, and an independent consultant offering his considerable humanitarian, innovation and related experience.  Matthew accompanied me on this trip to assist with the project, and will also help out on upcoming visits to Vanuatu as well.  If you are looking for someone with considerable experience and insight into the region, I'd suggest getting in touch with him (poke me for contact details).

Matthew also presented on behalf of NZ Red Cross while there with us, talking about Succinct Data, another of our joint projects:

Then it was my turn to present about Serval and the PHC pilot in Vanuatu:

We had some of the freshly-minted new Mesh Extenders with us to pass around for the audience to look at and interact with.  Having a professionally designed and manufactured injection-moulded case makes a massive difference to first impressions: Suddenly people think of it as a product and want to know how much they cost! It might have been expensive to do the tooling, but it is already paying dividends, by removing psychological barriers.

Then it was a visit to the National Disaster Management Organisation (NDMO) as part of the ICT Days activities.  This was convenient, as the NDMO one of the organisations on our list to visit.  We are arranging with NDMO to put a Mesh Extender on their building as part of a local network in Port Vila as part of the project.  Here I am outside the door waiting to go in:

Then back at the convention centre it was time to speak as part of a panel looking at the role of ICT and telecommunications in facilitating opportunity and growth.  Two places to the right of me is Salma Farouque, who has been based in the UN World Food Programme (UNWFP) in Fiji, and has also been a huge help in connecting us in with various folks in the region:

After the ICT Days were over, it was time to start meeting with other people.  One of the first visits was to the Telecommunications & Radiocommunications Regulator (TRR), the Vanuatu equivalent of the FCC in the US, or ACMA in Australia.

I cannot overstate just how supportive they have been of us, and the practical assistance that they have given us already.  The regulator staff clearly understand the communications and social challenges facing Vanuatu, and see that we have a missing piece that can help in ways that complement the existing mobile carriers and other initiatives in Vanuatu.

In particular, the regulator sees that technologies like the Serval Mesh have the potential to provide at least basic telecommunications services in locations where conventional cellular communications will never be cost-effective to provide, because of the small, low-income isolated communities living in hilly, mountainous and/or remote locations.   So it was lovely to finally give them the opportunity to try out the Serval Mesh, and see the Serval Mesh Extenders for themselves:

Speaking of the Mesh Extenders, time was so tight before I flew out (my lovely wife even had to help me cut out the holes to pack the Mesh Extenders in the foam blocks in our shipping crate in the hours before I flew out), that I didn't have time to make up the power cables for them.  So this had to happen in Vanuatu.  Sadly my old soldering iron is no longer really up to the job, so this ended up being much more of a hassle than it should have been:

(I have since bought a really nice little AA-battery powered soldering iron, so that for future visits, including to Maewo where there is very limited mains power, I don't have a repeat of this problem.)

TRR also helped us choose a couple of Villages on the same island as Port Vila where we could conduct the pilot, and then made us the necessary introduction to the chief of the first of those villages, and came out with us and helped translate into Bislama (the local Vanuatu pidgin language that almost everyone speaks):

Speaking of Bislama, here is the crate of water we bought.  Translation: "Number #1 water / Good water, good life":

Back to the village, once they understood what we were offering, they were very interested, understanding the value of communications, and given the reality of the lack of cellular coverage in most of the village.

In this particular village, most of the buildings are still rather informal structures, and the surrounding vegetation is quite thick jungle:

Speaking of the jungle and vegetation, there are some quite impressive trees around.  The canopy typically is 10m -- 20m once you get away from the road.  This means that UHF packet radio will really only work if the Mesh Extenders are lofted high enough to clear the vegetation.  This will be one of the challenges going forward.  I expect that a lot of traffic will still deliver using store-and-forward transport in people's phones.

The main highway around Efate island has been sealed, but there are still interesting features, like this wooden bridge. One presumes that a more permanent structure doesn't exist because it would be (or has been) washed away following flooding.

Below, there are number of edible plants visible, primarily bananas and taro (I think) plants.  It seemed like days before we saw any banana trees with fruit on them, and then suddenly we saw them everywhere, so I assume it just took my eye a while to get in the zone.

I have an interest in bananas, because I grow them at home in Adelaide, where the climate really is very marginal for them.  This has come in handy before when we needed a shot showing how Mesh Extenders might be installed in the tropics, but while I was at home. Having the banana plants at home came in handy for taking the following shot:

Not in Vanuatu
But bananas aren't the only fruit on hand.  From a road-side stall we were able to buy 20 large and yummy passion for a total of A$1.20:

Then it was around to the next village, who helpfully have a nice map painted on the side of a building:

This village also has a presence from the US Peace Corp, and generally seems to have more built infrastructure than the first village we visited.  There are apparently a lot of families that are spread between these two villages, which is part of why we are looking at them for the pilot, as we will have intra-village and inter-village communications.  The villages are about 10km apart -- too far for a single Mesh Extender UHF hop, so the inter-village communications will most likely be store-and-forward by people's phones for the most part.

Disaster awareness is something that is taken seriously in Vanuatu:

Then it was back to our accommodation to do some paper work and organise more meetings, with the help of the resident cat in the accommodation.  The cat was very happy to have some extra people around

Next stop was the Office of the Government Chief Information Officer (OGCIO) to keep them informed of what we were planning, and to ask for their assistance to finalise our human ethics approval acceptance in Vanuatu.  This process is still continuing, while we wait for the final sign-off from the Vanuatuan side. This is one of many processes that would have been much harder and taken even longer if we had not been able to spend time in country to talk with the appropriate people.

Then it was time to meet with representatives from the local mobile telephone carriers with the TRR.  This was a really interesting and productive meeting. We communicated our belief that in Vanuatu it makes a lot of sense for the carriers to leverage Serval to improve customer satisfaction (and that of the TRR!) by finding low-cost ways to plug coverage holes, and having an additional capacity that can be used to support communications when towers fail following cyclone or other adverse events.

In particular, we spoke with them about the possibility of creating a two-way SMS/MeshMS gateway, that would enable them to generate revenue from traffic that is delivered between their existing networks and Serval Mesh networks.  We might not be able to achieve anything by the end of the year in the immediate pilot, however we will keep exploring this, as it has considerable potential to help remote communities in Vanuatu and elsewhere.

Speaking of telecommunications service, where we were staying was about 15km out of Port Vila.  Voice calls were easy enough by mobile phone, but getting a usable 3G data signal was not easy.  We had to resort to putting my phone out in a plastic tub to keep out the rain and slobber (see below), in a vantage point where it had clear line of sight back across the bay to Port Vila. However, even then we were at the mercy of various atmospheric effectcs.  Internet would come and go on a minute by minute basis. In the end, we had to spend a lot of time camping in the offices of the lovely people at the TRR to make use of their reliable internet.

The box could stop the slobber of wet noses, but not being knocked about.

Here is a bit more context. The fruit on the tree are pawpaw.  Most mornings we had fresh pawpaw with our breakfast, very yum.

Here is the view from the house we stayed in down to the coast.  Port Vila is out of frame to the left.

And closer down to the coast:

The private stretch of coast was rocky, with lots of interesting little critters to see in the rock pools. There are many worse places to have to work.

Back up by the house, we setup one of the Mesh Extenders with a solar panel to do some testing, and took this little shot to give an idea of just how simple an independent Mesh Extender installation can be:

Then more meetings! This time with the organisers of the Smart ICT Sistas programme. We hope to involve this group that helps young ni-Vanuatu women to learn ICT skills in the deployment of the pilot, as part of our knowledge-transfer objectives.

Then it was time to meet with some folks we met through the Pacific ICT Days, who are planning to roll out a small telecommunications network on Maewo island.  This is one of those connections that could only happen because we were in-country.  They are already planning to put in a couple of 10m high masts, so have the ideal solution for mounting a Mesh Extender clear of the jungle canopy.

Alexis (far left) has been a real gem, helping us with all sorts of further connections, including the Ministry of Health, as well as being a key driver in this Maewo part of the pilot.  Here she is with myself, and a couple of the other folks who are part of that project. The mast sections in the background will be installed on the island, and were built largely with volunteer effort by someone who builds them for one of the local mobile operators.

A quick fit-test with a Mesh Extender revealed a (totally coincidental) perfect fit!

Thursday, June 1, 2017

Initial testing of the new Mesh Extenders with NZ Red Cross in Dunedin

As I have hinted at previously, I went to Dunedin with NZ Red Cross last month to participate in their field exercise there, and to bring the new Mesh Extender prototypes with me for initial socialisation and testing with the NZ RC IT&Telecommunications Emergency Response Unit.

The exercise was run over a weekend, with myself and the ERU arriving a couple of days in advance to establish VHF communications and internet access in advance for them.  Here is the white board listing the task list.  Serval Mesh and RAMP+ (another project we have with NZ RC) are on the list! (I'll talk a bit more about RAMP+ later in this post).

To give a feel for the exercise, here are some shots from around the Emergency Operations Centre (EOC), which was located in a Scout camp site near Dunedin.

First, internet from two places, the satellite dish in the foreground providing VSAT internet, and the ubiquiti router (easier to see in the second image below) pulling in internet from a nearby wireless ISP.  The ISP link required activating and a fair bit of work to get going.  The ERU folks then setup an automatic fail-over between them.  The mast in the field to the right was for VHF radio. They also setup a couple of repeaters on nearby hills.

Here is the communications desk in the EOC itself, staffed by the wonderful ERU folks in their very nice new jackets that Kathmandu kindly donated:

Then here I am sitting down trying to flash and debug some teething problems on the Mesh Extenders with Steve from the ERU to the left:

Here is Andrew from the ERU taking a Mesh Extender and battery pack walk-about as part of a multi-hop over UHF test.

And another of the ERU out during the exercise collecting data using RAMP+, which is a combination of and the Succinct Data extreme compression and satellite uplink tool designed and built by Matthew Lloyd and ourselves, which can exceed 99% compression when uploading the XML records for Magpi, making satellite SMS practical as a transport.

The week turned out to be extremely busy for me, as the Mesh Extender prototypes were still very new, and I only managed to get them packed in time, and the firmware was very raw, and required considerable work to get to the point of having the Mesh Extenders booting quickly and reliably, including formatting their microSD cards.

This was complicated by problems we encountered with RAMP+, where the interface between the Succinct Data application on the tablets and the Garmin inReaches would break if tracking were enabled. Used separately, both work fine, but together a bug in the inReach communications library for Android gets tickled, causing an uncatchable force-quit.  This combined with a minor comedy of errors in trying to get access to the Succinct Data server back home at the University (it was now after close of business in Australia on the Friday), culminating in completely reimplementing the Succinct Data server on Amazon Web Services (for which it is much the better) from in the field, consuming a valuable 36 hour period, which I really needed to use to get the Mesh Extender firmware shaken down.

The result was that we got RAMP+ mostly operational, but the Mesh Extenders had a couple of firmware issues that prevented us from completing any multi-hop UHF tests, which was disappointing.  We also quickly discovered a critical bug in the new Serval Chat application for Android, which rendered the application unresponsive on first load in certain circumstances.  This was of course rather disappointing for myself and the ERU team, as we had hoped to be able to carry out more sophisticated and extensive tests.

The good news is that we have since isolated and corrected the faults that we encountered in both the Mesh Extenders and Serval Chat, and are now gearing up to making an updated firmware image in preparation for the next visit to Vanuatu, and, hopefully, with enough time before landing to perform more extensive testing (in the process we now have more extensive test cases for the firmware image, which means that we are able to automate regression testing of the faults we encountered in NZ).  But before I do that, I need to write about the first visit to Vanuatu...

Prototype Mesh Extender Fly-away kit with NZ Red Cross

As part of the exercise in NZ with NZ Red Cross (NZRC), I pulled together the first attempt at a Serval Mesh fly-away kit for humanitarian use by NZRC.  The contents will be sure to evolve over time, so my focus was just on getting the Mesh Extenders safely in the box for transit, and all of the cables, tools and other bits and pieces that I would be likely to need while in NZ, and later in Vanuatu.

First step was to start with the big Zargs case, which is the standard fare used by NZRC, and make sure it would be big enough:

Step 2 was to get some foam sheets of various thicknesses, so that I could keep everything safe in there, so that enthusiastic baggage handlers wouldn't be a problem.  Clark's Rubber in Australia is the convenient retailer for that sort of thing.  Of course, retail price is not ideal, and it cost about AU$400 for enough foam to fill the box completely.

From one of the 75mm thick layers, my wife and I cut the holes to fit the eight prototype Mesh Extenders that currently exist, as can be seen below, while unpacking the kit in NZ:

Thinner foam layers were above and below in the box, and had solar panels, cables and tools interleaved among them.

My feeling is that we should be able to easily fit 16 units and two solar panels and all associated cables and bits and pieces in one of these cases without great difficulty. If we forgo the solar panels, then 24 or 32 would be possible.  However, that is: (a) a lot of Mesh Extenders; and (b) you still need some way to power them.  

Thus my feeling is that 8 or 16 units per case, with a richer set of accessories will be the norm. This also allows each fly-away kit to be more affordable, and for Mesh Extenders to be more easily distributed, rather than having to split the contents of a single case.

With a 40W and 20W regular glass solar panel, the eight Mesh Extenders and other parts, the whole thing weights <23kg, making it easy to ship as an extra bag by commercial airline, which is an important design goal.

One thing I did discover when traveling with a huge metal box that says "Emergency Response Unit" in large friendly type, is that it didn't exactly have the same effects as the word "Don't Panic" on the Hitch-hiker's Guide to the Galaxy.  Basically they viewed it as being Not Personal Effects and wanted to see appropriate paperwork on reentry to Australia, to see if they could charge me import duties.

Because the kit was only out of the country briefly, it is in principle possible to export and re-import without any customs liabilities, which I managed, however, if I am going to do it confidently repeatedly in the future, there is a pile of magic paperwork that I need to explore.  Exporting to Vanuatu for the pilot similarly requires some special paperwork on the Vanuatuan side, to ensure we are able to use their customs exemption for foreign aid projects.  Another whole interesting world that I am slowly becoming acquainted with...

Injection-moulded Mesh Extender cases

It has been a bit delayed with everything going on to report it, but we have the new Mesh Extender prototypes in their shiny injection-moulded cases. Here is Mesh Extender 2.0 serial number #1 (thanks to Rachael from Second Muse for the shots):

These prototypes are almost, but not quite complete.  The main outstanding issue is that the D-SUB connectors are not the correct IP67 ones with the seals around them.  Thus, while they are ostensibly complete, with the revision 4 PCBs, they are not weather proof enough for deployment.  We will fix this with the upcoming revision 5 PCB.

Also, when those shots were taken, we didn't have the correct reverse-SMA connectors with built-in seals to seal the antenna inputs. I am still not at ease that they will seal completely.  The other niggling concern I have is that humid air will pass through the goretex seal on the bottom (the black disc in the top image), i.e., only liquid phase water will be excluded. I hadn't previously realised this limitation.  Our current approach to these problems is to apply a conformal coating to the revision 5 PCBs, so that with the housing limiting water ingress, and hopefully excluding the ants and other foreign bodies, that the conformal coating will be up to the task.  I am confident it will all work out in the end, but this is representative of the many details that come into play when trying to make hardware, and especially if it is to be used in a hostile environment, such as in the middle of a tropical jungle.

I'll also shortly post about their first excursions to New Zealand to a Red Cross exercise, and to Vanuatu, as the first stage of our pilot there.

Thursday, April 27, 2017

Bringing up the Mesh Extender 2.0 firmware image in NZ

Here in NZ with our friends from the NZ Red Cross IT & Telecommunications Emergency Response Unit (ERU), we are getting ready to use some of the prototype Mesh Extender 2.0s as part of a Red Cross South Island exercise.

While I just managed to get the 8 prototype units built up in time to fly out, I didn't have time to get the firmware image ready.  So that's what I have been doing for the past couple of days here.

One of the big problems for getting the firmware working, is that with the RFD900+ radio on the serial port, it's rather hard to tell what is happening during the boot process.  Couple that with some ~5 - 20 minute boot delays that I was trying to debug, this was not ideal.

My solution has been to enable the busybox httpd on the Mesh Extenders early in the boot process, and write some scripts that emit progress messages to a web page, similar to the Linux console boot messages.  The result is that after only 20 seconds from power-on, I can start seeing what is going on.  Here is the result after I implemented this and fixed a number of the delay problems:

Serval Mesh Extender 2.0 -- Booting

System booting for 85 seconds.
OKBOOT+20Entering S47mountstuff
OKBOOT+21Making sure bulk storage unmounted
OKBOOT+21Checking integrity of /serval
OKBOOT+23Checking integrity of /serval-var
OKBOOT+28Checking integrity of /dos
OKBOOT+36Mounted /dos
OKBOOT+36Mounted /serval
OKBOOT+36Finished mounting file systems
OKBOOT+36Reading Radio EEPROM settings
OKBOOT+43Read Radio EEPROM settings (5 lines)
OKBOOT+43Exiting S47mountstuff
OKBOOT+43Entering S49servald
OKBOOT+43Reseting /etc/inittab to default
OKBOOT+43/etc/inittab ok
OKBOOT+43Checking if RFD900 requires re-flashing
OKBOOT+75Starting servald, lbard and otaupdate
OKBOOT+76Finished S49servald
OKBOOT+78Entering S50dropbear
OKBOOT+78Locking root password
OKBOOT+78Enabling SSH login
OKBOOT+80Setting root password
OKBOOT+83Entering S94captiveportal
OKBOOT+83Starting dnsmasq
OKBOOT+84Started 5 dnsmasq processes
OKBOOT+84Exiting S94captiveportal
OKBOOT+84Entering S99mountcheck
OKBOOT+85Exiting S99mountcheck

System dmesg output

[    0.000000] Linux version 3.18.45 (serval@meshextender-imaging) (gcc version 4.8.3 (OpenWrt/Linaro GCC 4.8-2014.04 r49389) ) #15 Fri Apr 28 07:55:02 CST 2017
[    0.000000] MyLoader: sysp=d578547a, boardp=f5f9d079, parts=d5c8c478
[    0.000000] bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 00019374 (MIPS 24Kc)
[    0.000000] SoC: Atheros AR9330 rev 1
[    0.000000] Determined physical RAM map:
[    0.000000]  memory: 04000000 @ 00000000 (usable)
[    0.000000] Initrd not found or empty - disabling initrd
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x00000000-0x03ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00000000-0x03ffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000-0x03ffffff]
[    0.000000] On node 0 totalpages: 16384
[    0.000000] free_area_init_node: node 0, pgdat 8036a0f0, node_mem_map 81000000
[    0.000000]   Normal zone: 128 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 16384 pages, LIFO batch:3
[    0.000000] Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes.
[    0.000000] Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 16256
[    0.000000] Kernel command line:  board=GL-AR150 mtdparts=spi0.0:256k(u-boot)ro,64k(u-boot-env)ro,16000k(firmware),64k(art)ro console=ttyATH0,115200 rootfstype=squashfs,jffs2 noinitrd
[    0.000000] PID hash table entries: 256 (order: -2, 1024 bytes)
[    0.000000] Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)
[    0.000000] Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Writing ErrCtl register=00000000
[    0.000000] Readback ErrCtl register=00000000
[    0.000000] Memory: 60880K/65536K available (2533K kernel code, 149K rwdata, 540K rodata, 224K init, 188K bss, 4656K reserved)
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS:51
[    0.000000] Clocks: CPU:400.000MHz, DDR:400.000MHz, AHB:200.000MHz, Ref:25.000MHz
[    0.000000] Calibrating delay loop... 265.42 BogoMIPS (lpj=1327104)
[    0.080000] pid_max: default: 32768 minimum: 301
[    0.080000] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.090000] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.100000] NET: Registered protocol family 16
[    0.100000] MIPS: machine is GL AR150
[    0.590000] Switched to clocksource MIPS
[    0.590000] NET: Registered protocol family 2
[    0.600000] TCP established hash table entries: 1024 (order: 0, 4096 bytes)
[    0.600000] TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
[    0.600000] TCP: Hash tables configured (established 1024 bind 1024)
[    0.610000] TCP: reno registered
[    0.610000] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    0.620000] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    0.630000] NET: Registered protocol family 1
[    0.630000] PCI: CLS 0 bytes, default 32
[    0.630000] futex hash table entries: 256 (order: -1, 3072 bytes)
[    0.650000] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.650000] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.660000] msgmni has been set to 118
[    0.680000] io scheduler noop registered
[    0.680000] io scheduler deadline registered (default)
[    0.690000] Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
[    0.690000] ar933x-uart: ttyATH0 at MMIO 0x18020000 (irq = 11, base_baud = 1562500) is a AR933X UART
[    0.700000] console [ttyATH0] enabled
[    0.710000] bootconsole [early0] disabled
[    0.720000] m25p80 spi0.0: found w25q128, expected m25p80
[    0.720000] m25p80 spi0.0: w25q128 (16384 Kbytes)
[    0.730000] 4 cmdlinepart partitions found on MTD device spi0.0
[    0.730000] Creating 4 MTD partitions on "spi0.0":
[    0.740000] 0x000000000000-0x000000040000 : "u-boot"
[    0.740000] 0x000000040000-0x000000050000 : "u-boot-env"
[    0.750000] 0x000000050000-0x000000ff0000 : "firmware"
[    0.780000] 2 uimage-fw partitions found on MTD device firmware
[    0.780000] 0x000000050000-0x000000170000 : "kernel"
[    0.790000] 0x000000170000-0x000000ff0000 : "rootfs"
[    0.790000] mtd: device 4 (rootfs) set to be root filesystem
[    0.800000] 1 squashfs-split partitions found on MTD device rootfs
[    0.810000] 0x0000005a0000-0x000000ff0000 : "rootfs_data"
[    0.810000] 0x000000ff0000-0x000001000000 : "art"
[    0.830000] libphy: ag71xx_mdio: probed
[    1.430000] ag71xx ag71xx.0: connected to PHY at ag71xx-mdio.1:04 [uid=004dd041, driver=Generic PHY]
[    1.440000] eth0: Atheros AG71xx at 0xb9000000, irq 4, mode:MII
[    2.030000] ag71xx-mdio.1: Found an AR7240/AR9330 built-in switch
[    2.060000] eth1: Atheros AG71xx at 0xba000000, irq 5, mode:GMII
[    2.060000] TCP: cubic registered
[    2.070000] NET: Registered protocol family 17
[    2.070000] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this.
[    2.080000] 8021q: 802.1Q VLAN Support v1.8
[    2.100000] VFS: Mounted root (squashfs filesystem) readonly on device 31:4.
[    2.100000] Freeing unused kernel memory: 224K (80388000 - 803c0000)
[    3.350000] init: Console is alive
[    3.350000] init: - watchdog -
[    3.360000] random: kmodloader urandom read with 5 bits of entropy available
[    5.490000] usbcore: registered new interface driver usbfs
[    5.500000] usbcore: registered new interface driver hub
[    5.500000] usbcore: registered new device driver usb
[    5.560000] SCSI subsystem initialized
[    5.570000] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    5.580000] ehci-platform: EHCI generic platform driver
[    5.580000] ehci-platform ehci-platform: EHCI Host Controller
[    5.590000] ehci-platform ehci-platform: new USB bus registered, assigned bus number 1
[    5.600000] ehci-platform ehci-platform: irq 3, io mem 0x1b000000
[    5.620000] ehci-platform ehci-platform: USB 2.0 started, EHCI 1.00
[    5.620000] hub 1-0:1.0: USB hub found
[    5.620000] hub 1-0:1.0: 1 port detected
[    5.640000] usbcore: registered new interface driver usb-storage
[    6.470000] init: - preinit -
[    9.720000] eth0: link up (100Mbps/Full duplex)
[   10.690000] jffs2: notice: (348) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
[   10.700000] mount_root: switching to jffs2 overlay
[   10.750000] eth0: link down
[   10.770000] procd: - early -
[   10.770000] procd: - watchdog -
[   11.460000] procd: - ubus -
[   12.470000] procd: - init -
[   14.000000] NET: Registered protocol family 10
[   14.050000] ip6_tables: (C) 2000-2006 Netfilter Core Team
[   14.080000] Loading modules backported from Linux version v4.4-rc5-1913-gc8fdf68
[   14.080000] Backport generated by backports.git backports-20151218-0-g2f58d9d
[   14.100000] ip_tables: (C) 2000-2006 Netfilter Core Team
[   14.150000] mmc_spi spi0.1: SD/MMC host mmc0, no DMA, no WP, no poweroff
[   14.160000] nf_conntrack version 0.5.0 (954 buckets, 3816 max)
[   14.180000] mmc0: host does not support reading read-only switch, assuming write-enable
[   14.190000] mmc0: new SDHC card on SPI
[   14.200000] mmcblk0: mmc0:0000 SD16G 14.4 GiB 
[   14.210000]  mmcblk0: p1 p2 p3
[   14.250000] usbcore: registered new interface driver ums-alauda
[   14.280000] usbcore: registered new interface driver ums-cypress
[   14.290000] usbcore: registered new interface driver ums-datafab
[   14.290000] usbcore: registered new interface driver ums-freecom
[   14.300000] usbcore: registered new interface driver ums-isd200
[   14.320000] usbcore: registered new interface driver ums-jumpshot
[   14.330000] usbcore: registered new interface driver ums-karma
[   14.340000] usbcore: registered new interface driver ums-sddr09
[   14.350000] usbcore: registered new interface driver ums-sddr55
[   14.360000] usbcore: registered new interface driver ums-usbat
[   14.370000] usbcore: registered new interface driver usbserial
[   14.370000] usbcore: registered new interface driver usbserial_generic
[   14.380000] usbserial: USB Serial support registered for generic
[   14.420000] xt_time: kernel timezone is -0000
[   14.430000] usbcore: registered new interface driver ark3116
[   14.430000] usbserial: USB Serial support registered for ark3116
[   14.450000] usbcore: registered new interface driver belkin_sa
[   14.450000] usbserial: USB Serial support registered for Belkin / Peracom / GoHubs USB Serial Adapter
[   14.530000] usbcore: registered new interface driver ch341
[   14.530000] usbserial: USB Serial support registered for ch341-uart
[   14.540000] usbcore: registered new interface driver cp210x
[   14.550000] usbserial: USB Serial support registered for cp210x
[   14.550000] usbcore: registered new interface driver cypress_m8
[   14.560000] usbserial: USB Serial support registered for DeLorme Earthmate USB
[   14.570000] usbserial: USB Serial support registered for HID->COM RS232 Adapter
[   14.570000] usbserial: USB Serial support registered for Nokia CA-42 V2 Adapter
[   14.580000] usbcore: registered new interface driver ftdi_sio
[   14.590000] usbserial: USB Serial support registered for FTDI USB Serial Device
[   14.700000] PPP generic driver version 2.4.2
[   14.700000] NET: Registered protocol family 24
[   14.780000] ath: EEPROM regdomain: 0x0
[   14.780000] ath: EEPROM indicates default country code should be used
[   14.780000] ath: doing EEPROM country->regdmn map search
[   14.780000] ath: country maps to regdmn code: 0x3a
[   14.780000] ath: Country alpha2 being used: US
[   14.780000] ath: Regpair used: 0x3a
[   14.790000] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
[   14.790000] ieee80211 phy0: Atheros AR9330 Rev:1 mem=0xb8100000, irq=2
[   23.820000] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   23.880000] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[   26.110000] eth0: link up (100Mbps/Full duplex)
[   26.110000] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   30.070000] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[   30.280000] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[   30.380000] IPv6: ADDRCONF(NETDEV_UP): adhoc0: link is not ready
[   32.620000] adhoc0: Created IBSS using preconfigured BSSID 02:ca:ff:dd:ca:ce
[   32.620000] adhoc0: Creating new IBSS network, BSSID 02:ca:ff:dd:ca:ce
[   32.630000] IPv6: ADDRCONF(NETDEV_CHANGE): adhoc0: link becomes ready
[   36.180000] EXT4-fs (mmcblk0p2): couldn't mount as ext3 due to feature incompatibilities
[   36.190000] EXT4-fs (mmcblk0p2): couldn't mount as ext2 due to feature incompatibilities
[   36.240000] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[   36.320000] EXT4-fs (mmcblk0p3): couldn't mount as ext3 due to feature incompatibilities
[   36.330000] EXT4-fs (mmcblk0p3): couldn't mount as ext2 due to feature incompatibilities
[   36.400000] EXT4-fs (mmcblk0p3): mounted filesystem with ordered data mode. Opts: (null)
[   37.540000] random: nonblocking pool is initialized

Last modified: Fri Apr 28 09:50:53 NZST 2017

So now we can boot a Mesh Extender in under 90 seconds and know where it is up to in the meantime.

In the process, I discovered that using ext2 instead of ext4 was a really bad idea, as on the slow microSD interface we have checking the filesystem can take 8 minutes, even when empty.

Now to update the other seven, and get out into the nice New Zealand sunshine that has been taunting me all morning, and start testing!

Sunday, April 23, 2017

Preparing for exercise with NZ Red Cross

For our DFAT Pacific Pilot in Vanuatu, I am getting ready to fly out to New Zealand for a training exercise with NZ Red Cross's IT & Telecommunications Emergency Response Unit (IT&T ERU), with whom we have previously tested the Serval Mesh and earlier prototypes of the Mesh Extenders.

This time, the plan is for me to build up a fly-away kit containing Mesh Extenders, solar panels, batteries and all the necessary cables and bits and pieces for a complete functioning system.

The fun is that we don't yet have all the parts on hand, and there is still quite a bit of work to do.  So so far, the box only has me in it:

By the end of the day, it should instead have all the prototype hardware inside ready to fly out to NZ early tomorrow morning.  It is has occurred to me that this means that this Anzac Day I will be flying to a joint Australian / NZ exercise designed to protect human life.  Perhaps not on the same level as those who served in war, but it certainly reminds me of their service.

Between now and then, aside from all the paper-work, I need to:

1. Assemble 8 Mesh Extender prototypes;

2. Build 8 power/radio ID cables;

3. Program those cables;

4. Build the final firmware image to be used for the exercise and install them in the prototypes;

5. Install and prepare the microSD cards in them all;

6. Test that they work.

7. Figure out the correct way to carry LiFePO4 cells with me to NZ.

So, I'd better get to it!

Tuesday, March 28, 2017

Teaching the RFD900 about uboot, so that it doesn't interrupt the Mesh Extender boot process

The Serval Mesh Extenders logically consist of a CPU with a serial port, and an RFD900 radio.  Our RFD900 firmware provides a regular heart-beat.

This all means that the RFD900 radio, as soon as it powers on, begins sending heart-beat messages.

This ordinarily wouldn't be a problem, except that the CPU modules we are using have only one serial port, so the serial boot loader is listening on the serial port when the system first powers up, and whenever it resets.

This means that if uboot sees a heartbeat, it will interpret it as the user wishing to interrupt the boot process to interact with the boot loader.  Following Murphey's Law, this is assured to happen on every boot.

We can't get rid of uboot, as it is important to have a way to reflash the devices (especially while we are still developing the software for them).

The remaining solution is to teach the RFD900 to notice when uboot is active.

In theory, this would just take a simple little finite state machine to notice the uboot banner, and then make sure it doesn't send any characters for a while, to allow uboot to boot normally.

There are, however, two complications.

1. We have to stay silent for long enough that we don't interfere with OpenWRT's fail-safe boot mode, which just means staying silent for longer.


2. The RFD900 is set to 230400bps for talking to LBARD, while uboot uses the serial port at 115200.

It is this last problem that makes life more interesting, because we can't just watch for the characters of the uboot banner. Instead, we need to look for a character sequence at 230400bps that is what you would see if you send the uboot banner at 115200bps.

As uboot is working at 1/2 the baud rate that the RFD900 is using, every serial bit received from uboot will be interpreted as two bits. Thus each uboot serial character should be able to be reliably received as two characters by the RFD900. We're lucky it isn't the other way around, as then the bits would be only half as wide, and there would be some uncertainty about how the bits would be read.

In the process, I also had to clean up a few other spots where our RFD900 firmware was causing other characters to be emitted immediately on power-up.

So after a few frustrating hours of debugging the finer points, I can now reliably boot a uboot-based system with an RFD900 running our packet radio firmware permanently attached to the serial port.

Wednesday, March 15, 2017

The custom IP67 power cables have arrived!

So, just over ten days ago I wrote that a Chinese manufacturer said that they could manufacture and ship the cables we want within ten days, and that I would order some.  Today they arrived!  Some fun unpacking photos:

First up, I must congratulate the Chinese manufacturer for being able to manufacture and ship our order so quickly.  They really were a pleasure to work with, including when we had a minor problem with the shipping (which wasn't their fault).

Also, as the photos hopefully show, the cables are very solidly built, with much thicker wire than the little ones we were contemplating using previously.  They will be able to easily support large panels and batteries, with charge currents being limited only by the battery charge controller inside the Mesh Extender. The cables simply give the strong impression that they will do the job well, which we will test over coming months.

The only complication we encountered, was that they didn't have stock of the white cable material, so we had to opt for the black PVC cable, which will slightly increase heat loading of the Mesh Extenders.  But given the time frames that we are working to at the moment, we needed to simply be sure that we would have the cables we need, so that we can get the cable head moulding tools made and cables assembled in time.  Critical paths are everything when building hardware!