Sunday, July 30, 2017

OpenWRT firmware building/installing problem

I'm currently working on the firmware for the Mesh Extenders, and have hit an unexpected problem:

Whereas we have previously been able to easily build and flash OpenWRT firmware images onto the Domino Core modules (Atheros 71xx based, Chaos Chalmer derived firmware), it has suddenly stopped working with nasty errors from uboot, like this:

uboot> boot
Booting image at: 0x9F050000

   Image name:   MIPS OpenWrt Linux-3.18.45
   Created:      2017-07-10  23:59:50 UTC
   Image type:   MIPS Linux Kernel Image (lzma compressed)
   Data size:    5328879 Bytes = 5.1 MB
   Load address: 0x80060000
   Entry point:  0x80060000

Uncompressing kernel image... ## Error: LZMA error num: 1
uboot>

Anyone who can provide some assistance or suggestions as to how to go about tracking the problem down and fixing it would be greatly appreciated.

Some clues so far:

1. The firmware file from http://www.gl-inet.com/firmware/domino/v1/openwrt-domino-2.261.bin, and that works, so I presume that both uboot and the hardware are working.

2. We used to build and flash firmware from an Ubuntu VM in a Mac.  This problem has come about since moving to a native Ubuntu 16.04 based machine.

3. Firmware built on the Mac, which we believe used to work, doesn't seem to worrk now.

4. When this first occurred, it looked like we might have had a problem with a USB power plug, so we have ditched that, and are now running the boards from a USB port on the laptop.  It is in this configuration that we have successfully installed the firmware file mentioned in (1).

5. I *did* manage to build some working firmware images on Friday with the Linux laptop, and apart from using old versions of Serval packages, they seemed to work, before putting updated versions on stopped working.

I'll likely update this post with progress of things as I try them.

UPDATE 01AUG17: It looks like the problem is that we can't flash from uboot while a microSD card is inserted.

Works

1. gl-inet openwrt-domino-2.261.bin installed from Linux laptop via "run lf" + tftp

2. Freshly built openwrt image built from Linux laptop without Mesh Extender package, and on an older test board, that was never powered by the presumed faulty USB power plug, and with no microSD card inserted. [01AUG17]

3. Freshly built openwrt image built from Linux laptop without Mesh Extender package, on one previously powered by the presumed faulty USB power plug, and without a microSD card inserted. [01AUG17]

Doesn't Work

1. openwrt-ar71xx-generic-uImage-initramfs-lzma.bin built on Ubuntu 16.04 laptop from github.com/servalproject/openwrt

2. Freshly built openwrt image built from Linux laptop without Mesh Extender package, on one previously powered by the presumed faulty USB power plug, and with a microSD card inserted. [01AUG17]

Thursday, July 6, 2017

First visit to Vanuatu

It has been a rather busy couple of months, so I am only just catching up with sharing what we have been up to. All going well, there will be a burst of posts describing our recent activities.

But first in this post, I want to talk about our first visit to Vanuatu as part of our Pacific Humanitarian Challenge (PHC) award from the Australian Department of Foreign Affairs and Trade (DFAT).

Basically, DFAT have commissioned us to run a pilot of the Serval Mesh in Vanuatu, to be concluded and reported on by the end of 2017. I can assure you, that that timeline is feeling very tight right now.

The lead up to the first visit was busy with getting the very first prototype Mesh Extenders built.  The circuit boards were still under active development, as we tried to fix a few niggling issues around the USB port and power/radio port pinout. We were also waiting on the first injection-moulded shells for the Mesh Extenders.

Part of the busy-ness was also that we had timed our first visit to Vanuatu to coincide with the Pacific ICT Days, which gave us a very firm deadline.

Part of our motivation in being there for the ICT days, was that I would be able to present to representatives from a number of nations in the region, to raise awareness and seek input from them on what we are doing.  This broader socialisation of Serval is one of the goals of the PHC award, to help gauge the regional interest and demand for our work.

Another motivation in being there for the ICT days was to be able to see what other initiatives are underway in Vanuatu, and work out who we could partner with for the pilot project.

We also arranged to stay for about a week after, to allow plenty of time to meet with various stake-holders.  Some meetings were pre-arranged, but the majority were arranged once in country, as this is still the easiest way.  I really can't emphasise how important it is to allow time in-country when working in Pacific island nations.  You can often achieve in a week or so, what would otherwise take a month or more to try to arrange remotely. Finally, the time in country gave us the chance to better understand the local context, and begin to understand the local culture first-hand.

Now to follow the visit in pictures:


 First, it was presentations at the Pacific ICT Days.  Matthew Lloyd, who has been our key contact in NZ Red Cross for almost seven years has resigned and is now a volunteer with NZ Red Cross, and an independent consultant offering his considerable humanitarian, innovation and related experience.  Matthew accompanied me on this trip to assist with the project, and will also help out on upcoming visits to Vanuatu as well.  If you are looking for someone with considerable experience and insight into the region, I'd suggest getting in touch with him (poke me for contact details).

Matthew also presented on behalf of NZ Red Cross while there with us, talking about Succinct Data, another of our joint projects:


Then it was my turn to present about Serval and the PHC pilot in Vanuatu:


We had some of the freshly-minted new Mesh Extenders with us to pass around for the audience to look at and interact with.  Having a professionally designed and manufactured injection-moulded case makes a massive difference to first impressions: Suddenly people think of it as a product and want to know how much they cost! It might have been expensive to do the tooling, but it is already paying dividends, by removing psychological barriers.


Then it was a visit to the National Disaster Management Organisation (NDMO) as part of the ICT Days activities.  This was convenient, as the NDMO one of the organisations on our list to visit.  We are arranging with NDMO to put a Mesh Extender on their building as part of a local network in Port Vila as part of the project.  Here I am outside the door waiting to go in:



Then back at the convention centre it was time to speak as part of a panel looking at the role of ICT and telecommunications in facilitating opportunity and growth.  Two places to the right of me is Salma Farouque, who has been based in the UN World Food Programme (UNWFP) in Fiji, and has also been a huge help in connecting us in with various folks in the region:


After the ICT Days were over, it was time to start meeting with other people.  One of the first visits was to the Telecommunications & Radiocommunications Regulator (TRR), the Vanuatu equivalent of the FCC in the US, or ACMA in Australia.

I cannot overstate just how supportive they have been of us, and the practical assistance that they have given us already.  The regulator staff clearly understand the communications and social challenges facing Vanuatu, and see that we have a missing piece that can help in ways that complement the existing mobile carriers and other initiatives in Vanuatu.

In particular, the regulator sees that technologies like the Serval Mesh have the potential to provide at least basic telecommunications services in locations where conventional cellular communications will never be cost-effective to provide, because of the small, low-income isolated communities living in hilly, mountainous and/or remote locations.   So it was lovely to finally give them the opportunity to try out the Serval Mesh, and see the Serval Mesh Extenders for themselves:



Speaking of the Mesh Extenders, time was so tight before I flew out (my lovely wife even had to help me cut out the holes to pack the Mesh Extenders in the foam blocks in our shipping crate in the hours before I flew out), that I didn't have time to make up the power cables for them.  So this had to happen in Vanuatu.  Sadly my old soldering iron is no longer really up to the job, so this ended up being much more of a hassle than it should have been:


(I have since bought a really nice little AA-battery powered soldering iron, so that for future visits, including to Maewo where there is very limited mains power, I don't have a repeat of this problem.)

TRR also helped us choose a couple of Villages on the same island as Port Vila where we could conduct the pilot, and then made us the necessary introduction to the chief of the first of those villages, and came out with us and helped translate into Bislama (the local Vanuatu pidgin language that almost everyone speaks):


Speaking of Bislama, here is the crate of water we bought.  Translation: "Number #1 water / Good water, good life":



Back to the village, once they understood what we were offering, they were very interested, understanding the value of communications, and given the reality of the lack of cellular coverage in most of the village.

In this particular village, most of the buildings are still rather informal structures, and the surrounding vegetation is quite thick jungle:



Speaking of the jungle and vegetation, there are some quite impressive trees around.  The canopy typically is 10m -- 20m once you get away from the road.  This means that UHF packet radio will really only work if the Mesh Extenders are lofted high enough to clear the vegetation.  This will be one of the challenges going forward.  I expect that a lot of traffic will still deliver using store-and-forward transport in people's phones.


The main highway around Efate island has been sealed, but there are still interesting features, like this wooden bridge. One presumes that a more permanent structure doesn't exist because it would be (or has been) washed away following flooding.


Below, there are number of edible plants visible, primarily bananas and taro (I think) plants.  It seemed like days before we saw any banana trees with fruit on them, and then suddenly we saw them everywhere, so I assume it just took my eye a while to get in the zone.


I have an interest in bananas, because I grow them at home in Adelaide, where the climate really is very marginal for them.  This has come in handy before when we needed a shot showing how Mesh Extenders might be installed in the tropics, but while I was at home. Having the banana plants at home came in handy for taking the following shot:

Not in Vanuatu
But bananas aren't the only fruit on hand.  From a road-side stall we were able to buy 20 large and yummy passion for a total of A$1.20:


Then it was around to the next village, who helpfully have a nice map painted on the side of a building:


This village also has a presence from the US Peace Corp, and generally seems to have more built infrastructure than the first village we visited.  There are apparently a lot of families that are spread between these two villages, which is part of why we are looking at them for the pilot, as we will have intra-village and inter-village communications.  The villages are about 10km apart -- too far for a single Mesh Extender UHF hop, so the inter-village communications will most likely be store-and-forward by people's phones for the most part.

Disaster awareness is something that is taken seriously in Vanuatu:


Then it was back to our accommodation to do some paper work and organise more meetings, with the help of the resident cat in the accommodation.  The cat was very happy to have some extra people around


Next stop was the Office of the Government Chief Information Officer (OGCIO) to keep them informed of what we were planning, and to ask for their assistance to finalise our human ethics approval acceptance in Vanuatu.  This process is still continuing, while we wait for the final sign-off from the Vanuatuan side. This is one of many processes that would have been much harder and taken even longer if we had not been able to spend time in country to talk with the appropriate people.


Then it was time to meet with representatives from the local mobile telephone carriers with the TRR.  This was a really interesting and productive meeting. We communicated our belief that in Vanuatu it makes a lot of sense for the carriers to leverage Serval to improve customer satisfaction (and that of the TRR!) by finding low-cost ways to plug coverage holes, and having an additional capacity that can be used to support communications when towers fail following cyclone or other adverse events.




In particular, we spoke with them about the possibility of creating a two-way SMS/MeshMS gateway, that would enable them to generate revenue from traffic that is delivered between their existing networks and Serval Mesh networks.  We might not be able to achieve anything by the end of the year in the immediate pilot, however we will keep exploring this, as it has considerable potential to help remote communities in Vanuatu and elsewhere.

Speaking of telecommunications service, where we were staying was about 15km out of Port Vila.  Voice calls were easy enough by mobile phone, but getting a usable 3G data signal was not easy.  We had to resort to putting my phone out in a plastic tub to keep out the rain and slobber (see below), in a vantage point where it had clear line of sight back across the bay to Port Vila. However, even then we were at the mercy of various atmospheric effectcs.  Internet would come and go on a minute by minute basis. In the end, we had to spend a lot of time camping in the offices of the lovely people at the TRR to make use of their reliable internet.



The box could stop the slobber of wet noses, but not being knocked about.


Here is a bit more context. The fruit on the tree are pawpaw.  Most mornings we had fresh pawpaw with our breakfast, very yum.


Here is the view from the house we stayed in down to the coast.  Port Vila is out of frame to the left.


And closer down to the coast:


The private stretch of coast was rocky, with lots of interesting little critters to see in the rock pools. There are many worse places to have to work.


Back up by the house, we setup one of the Mesh Extenders with a solar panel to do some testing, and took this little shot to give an idea of just how simple an independent Mesh Extender installation can be:


Then more meetings! This time with the organisers of the Smart ICT Sistas programme. We hope to involve this group that helps young ni-Vanuatu women to learn ICT skills in the deployment of the pilot, as part of our knowledge-transfer objectives.


Then it was time to meet with some folks we met through the Pacific ICT Days, who are planning to roll out a small telecommunications network on Maewo island.  This is one of those connections that could only happen because we were in-country.  They are already planning to put in a couple of 10m high masts, so have the ideal solution for mounting a Mesh Extender clear of the jungle canopy.

Alexis (far left) has been a real gem, helping us with all sorts of further connections, including the Ministry of Health, as well as being a key driver in this Maewo part of the pilot.  Here she is with myself, and a couple of the other folks who are part of that project. The mast sections in the background will be installed on the island, and were built largely with volunteer effort by someone who builds them for one of the local mobile operators.



A quick fit-test with a Mesh Extender revealed a (totally coincidental) perfect fit!