Thursday, June 1, 2017

Initial testing of the new Mesh Extenders with NZ Red Cross in Dunedin

As I have hinted at previously, I went to Dunedin with NZ Red Cross last month to participate in their field exercise there, and to bring the new Mesh Extender prototypes with me for initial socialisation and testing with the NZ RC IT&Telecommunications Emergency Response Unit.

The exercise was run over a weekend, with myself and the ERU arriving a couple of days in advance to establish VHF communications and internet access in advance for them.  Here is the white board listing the task list.  Serval Mesh and RAMP+ (another project we have with NZ RC) are on the list! (I'll talk a bit more about RAMP+ later in this post).

To give a feel for the exercise, here are some shots from around the Emergency Operations Centre (EOC), which was located in a Scout camp site near Dunedin.

First, internet from two places, the satellite dish in the foreground providing VSAT internet, and the ubiquiti router (easier to see in the second image below) pulling in internet from a nearby wireless ISP.  The ISP link required activating and a fair bit of work to get going.  The ERU folks then setup an automatic fail-over between them.  The mast in the field to the right was for VHF radio. They also setup a couple of repeaters on nearby hills.

Here is the communications desk in the EOC itself, staffed by the wonderful ERU folks in their very nice new jackets that Kathmandu kindly donated:

Then here I am sitting down trying to flash and debug some teething problems on the Mesh Extenders with Steve from the ERU to the left:

Here is Andrew from the ERU taking a Mesh Extender and battery pack walk-about as part of a multi-hop over UHF test.

And another of the ERU out during the exercise collecting data using RAMP+, which is a combination of and the Succinct Data extreme compression and satellite uplink tool designed and built by Matthew Lloyd and ourselves, which can exceed 99% compression when uploading the XML records for Magpi, making satellite SMS practical as a transport.

The week turned out to be extremely busy for me, as the Mesh Extender prototypes were still very new, and I only managed to get them packed in time, and the firmware was very raw, and required considerable work to get to the point of having the Mesh Extenders booting quickly and reliably, including formatting their microSD cards.

This was complicated by problems we encountered with RAMP+, where the interface between the Succinct Data application on the tablets and the Garmin inReaches would break if tracking were enabled. Used separately, both work fine, but together a bug in the inReach communications library for Android gets tickled, causing an uncatchable force-quit.  This combined with a minor comedy of errors in trying to get access to the Succinct Data server back home at the University (it was now after close of business in Australia on the Friday), culminating in completely reimplementing the Succinct Data server on Amazon Web Services (for which it is much the better) from in the field, consuming a valuable 36 hour period, which I really needed to use to get the Mesh Extender firmware shaken down.

The result was that we got RAMP+ mostly operational, but the Mesh Extenders had a couple of firmware issues that prevented us from completing any multi-hop UHF tests, which was disappointing.  We also quickly discovered a critical bug in the new Serval Chat application for Android, which rendered the application unresponsive on first load in certain circumstances.  This was of course rather disappointing for myself and the ERU team, as we had hoped to be able to carry out more sophisticated and extensive tests.

The good news is that we have since isolated and corrected the faults that we encountered in both the Mesh Extenders and Serval Chat, and are now gearing up to making an updated firmware image in preparation for the next visit to Vanuatu, and, hopefully, with enough time before landing to perform more extensive testing (in the process we now have more extensive test cases for the firmware image, which means that we are able to automate regression testing of the faults we encountered in NZ).  But before I do that, I need to write about the first visit to Vanuatu...

Prototype Mesh Extender Fly-away kit with NZ Red Cross

As part of the exercise in NZ with NZ Red Cross (NZRC), I pulled together the first attempt at a Serval Mesh fly-away kit for humanitarian use by NZRC.  The contents will be sure to evolve over time, so my focus was just on getting the Mesh Extenders safely in the box for transit, and all of the cables, tools and other bits and pieces that I would be likely to need while in NZ, and later in Vanuatu.

First step was to start with the big Zargs case, which is the standard fare used by NZRC, and make sure it would be big enough:

Step 2 was to get some foam sheets of various thicknesses, so that I could keep everything safe in there, so that enthusiastic baggage handlers wouldn't be a problem.  Clark's Rubber in Australia is the convenient retailer for that sort of thing.  Of course, retail price is not ideal, and it cost about AU$400 for enough foam to fill the box completely.

From one of the 75mm thick layers, my wife and I cut the holes to fit the eight prototype Mesh Extenders that currently exist, as can be seen below, while unpacking the kit in NZ:

Thinner foam layers were above and below in the box, and had solar panels, cables and tools interleaved among them.

My feeling is that we should be able to easily fit 16 units and two solar panels and all associated cables and bits and pieces in one of these cases without great difficulty. If we forgo the solar panels, then 24 or 32 would be possible.  However, that is: (a) a lot of Mesh Extenders; and (b) you still need some way to power them.  

Thus my feeling is that 8 or 16 units per case, with a richer set of accessories will be the norm. This also allows each fly-away kit to be more affordable, and for Mesh Extenders to be more easily distributed, rather than having to split the contents of a single case.

With a 40W and 20W regular glass solar panel, the eight Mesh Extenders and other parts, the whole thing weights <23kg, making it easy to ship as an extra bag by commercial airline, which is an important design goal.

One thing I did discover when traveling with a huge metal box that says "Emergency Response Unit" in large friendly type, is that it didn't exactly have the same effects as the word "Don't Panic" on the Hitch-hiker's Guide to the Galaxy.  Basically they viewed it as being Not Personal Effects and wanted to see appropriate paperwork on reentry to Australia, to see if they could charge me import duties.

Because the kit was only out of the country briefly, it is in principle possible to export and re-import without any customs liabilities, which I managed, however, if I am going to do it confidently repeatedly in the future, there is a pile of magic paperwork that I need to explore.  Exporting to Vanuatu for the pilot similarly requires some special paperwork on the Vanuatuan side, to ensure we are able to use their customs exemption for foreign aid projects.  Another whole interesting world that I am slowly becoming acquainted with...

Injection-moulded Mesh Extender cases

It has been a bit delayed with everything going on to report it, but we have the new Mesh Extender prototypes in their shiny injection-moulded cases. Here is Mesh Extender 2.0 serial number #1 (thanks to Rachael from Second Muse for the shots):

These prototypes are almost, but not quite complete.  The main outstanding issue is that the D-SUB connectors are not the correct IP67 ones with the seals around them.  Thus, while they are ostensibly complete, with the revision 4 PCBs, they are not weather proof enough for deployment.  We will fix this with the upcoming revision 5 PCB.

Also, when those shots were taken, we didn't have the correct reverse-SMA connectors with built-in seals to seal the antenna inputs. I am still not at ease that they will seal completely.  The other niggling concern I have is that humid air will pass through the goretex seal on the bottom (the black disc in the top image), i.e., only liquid phase water will be excluded. I hadn't previously realised this limitation.  Our current approach to these problems is to apply a conformal coating to the revision 5 PCBs, so that with the housing limiting water ingress, and hopefully excluding the ants and other foreign bodies, that the conformal coating will be up to the task.  I am confident it will all work out in the end, but this is representative of the many details that come into play when trying to make hardware, and especially if it is to be used in a hostile environment, such as in the middle of a tropical jungle.

I'll also shortly post about their first excursions to New Zealand to a Red Cross exercise, and to Vanuatu, as the first stage of our pilot there.

Thursday, April 27, 2017

Bringing up the Mesh Extender 2.0 firmware image in NZ

Here in NZ with our friends from the NZ Red Cross IT & Telecommunications Emergency Response Unit (ERU), we are getting ready to use some of the prototype Mesh Extender 2.0s as part of a Red Cross South Island exercise.

While I just managed to get the 8 prototype units built up in time to fly out, I didn't have time to get the firmware image ready.  So that's what I have been doing for the past couple of days here.

One of the big problems for getting the firmware working, is that with the RFD900+ radio on the serial port, it's rather hard to tell what is happening during the boot process.  Couple that with some ~5 - 20 minute boot delays that I was trying to debug, this was not ideal.

My solution has been to enable the busybox httpd on the Mesh Extenders early in the boot process, and write some scripts that emit progress messages to a web page, similar to the Linux console boot messages.  The result is that after only 20 seconds from power-on, I can start seeing what is going on.  Here is the result after I implemented this and fixed a number of the delay problems:

Serval Mesh Extender 2.0 -- Booting

System booting for 85 seconds.
OKBOOT+20Entering S47mountstuff
OKBOOT+21Making sure bulk storage unmounted
OKBOOT+21Checking integrity of /serval
OKBOOT+23Checking integrity of /serval-var
OKBOOT+28Checking integrity of /dos
OKBOOT+36Mounted /dos
OKBOOT+36Mounted /serval
OKBOOT+36Finished mounting file systems
OKBOOT+36Reading Radio EEPROM settings
OKBOOT+43Read Radio EEPROM settings (5 lines)
OKBOOT+43Exiting S47mountstuff
OKBOOT+43Entering S49servald
OKBOOT+43Reseting /etc/inittab to default
OKBOOT+43/etc/inittab ok
OKBOOT+43Checking if RFD900 requires re-flashing
OKBOOT+75Starting servald, lbard and otaupdate
OKBOOT+76Finished S49servald
OKBOOT+78Entering S50dropbear
OKBOOT+78Locking root password
OKBOOT+78Enabling SSH login
OKBOOT+80Setting root password
OKBOOT+83Entering S94captiveportal
OKBOOT+83Starting dnsmasq
OKBOOT+84Started 5 dnsmasq processes
OKBOOT+84Exiting S94captiveportal
OKBOOT+84Entering S99mountcheck
OKBOOT+85Exiting S99mountcheck

System dmesg output

[    0.000000] Linux version 3.18.45 (serval@meshextender-imaging) (gcc version 4.8.3 (OpenWrt/Linaro GCC 4.8-2014.04 r49389) ) #15 Fri Apr 28 07:55:02 CST 2017
[    0.000000] MyLoader: sysp=d578547a, boardp=f5f9d079, parts=d5c8c478
[    0.000000] bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 00019374 (MIPS 24Kc)
[    0.000000] SoC: Atheros AR9330 rev 1
[    0.000000] Determined physical RAM map:
[    0.000000]  memory: 04000000 @ 00000000 (usable)
[    0.000000] Initrd not found or empty - disabling initrd
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x00000000-0x03ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00000000-0x03ffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000-0x03ffffff]
[    0.000000] On node 0 totalpages: 16384
[    0.000000] free_area_init_node: node 0, pgdat 8036a0f0, node_mem_map 81000000
[    0.000000]   Normal zone: 128 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 16384 pages, LIFO batch:3
[    0.000000] Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes.
[    0.000000] Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 16256
[    0.000000] Kernel command line:  board=GL-AR150 mtdparts=spi0.0:256k(u-boot)ro,64k(u-boot-env)ro,16000k(firmware),64k(art)ro console=ttyATH0,115200 rootfstype=squashfs,jffs2 noinitrd
[    0.000000] PID hash table entries: 256 (order: -2, 1024 bytes)
[    0.000000] Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)
[    0.000000] Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Writing ErrCtl register=00000000
[    0.000000] Readback ErrCtl register=00000000
[    0.000000] Memory: 60880K/65536K available (2533K kernel code, 149K rwdata, 540K rodata, 224K init, 188K bss, 4656K reserved)
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS:51
[    0.000000] Clocks: CPU:400.000MHz, DDR:400.000MHz, AHB:200.000MHz, Ref:25.000MHz
[    0.000000] Calibrating delay loop... 265.42 BogoMIPS (lpj=1327104)
[    0.080000] pid_max: default: 32768 minimum: 301
[    0.080000] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.090000] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.100000] NET: Registered protocol family 16
[    0.100000] MIPS: machine is GL AR150
[    0.590000] Switched to clocksource MIPS
[    0.590000] NET: Registered protocol family 2
[    0.600000] TCP established hash table entries: 1024 (order: 0, 4096 bytes)
[    0.600000] TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
[    0.600000] TCP: Hash tables configured (established 1024 bind 1024)
[    0.610000] TCP: reno registered
[    0.610000] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    0.620000] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    0.630000] NET: Registered protocol family 1
[    0.630000] PCI: CLS 0 bytes, default 32
[    0.630000] futex hash table entries: 256 (order: -1, 3072 bytes)
[    0.650000] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.650000] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.660000] msgmni has been set to 118
[    0.680000] io scheduler noop registered
[    0.680000] io scheduler deadline registered (default)
[    0.690000] Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
[    0.690000] ar933x-uart: ttyATH0 at MMIO 0x18020000 (irq = 11, base_baud = 1562500) is a AR933X UART
[    0.700000] console [ttyATH0] enabled
[    0.710000] bootconsole [early0] disabled
[    0.720000] m25p80 spi0.0: found w25q128, expected m25p80
[    0.720000] m25p80 spi0.0: w25q128 (16384 Kbytes)
[    0.730000] 4 cmdlinepart partitions found on MTD device spi0.0
[    0.730000] Creating 4 MTD partitions on "spi0.0":
[    0.740000] 0x000000000000-0x000000040000 : "u-boot"
[    0.740000] 0x000000040000-0x000000050000 : "u-boot-env"
[    0.750000] 0x000000050000-0x000000ff0000 : "firmware"
[    0.780000] 2 uimage-fw partitions found on MTD device firmware
[    0.780000] 0x000000050000-0x000000170000 : "kernel"
[    0.790000] 0x000000170000-0x000000ff0000 : "rootfs"
[    0.790000] mtd: device 4 (rootfs) set to be root filesystem
[    0.800000] 1 squashfs-split partitions found on MTD device rootfs
[    0.810000] 0x0000005a0000-0x000000ff0000 : "rootfs_data"
[    0.810000] 0x000000ff0000-0x000001000000 : "art"
[    0.830000] libphy: ag71xx_mdio: probed
[    1.430000] ag71xx ag71xx.0: connected to PHY at ag71xx-mdio.1:04 [uid=004dd041, driver=Generic PHY]
[    1.440000] eth0: Atheros AG71xx at 0xb9000000, irq 4, mode:MII
[    2.030000] ag71xx-mdio.1: Found an AR7240/AR9330 built-in switch
[    2.060000] eth1: Atheros AG71xx at 0xba000000, irq 5, mode:GMII
[    2.060000] TCP: cubic registered
[    2.070000] NET: Registered protocol family 17
[    2.070000] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this.
[    2.080000] 8021q: 802.1Q VLAN Support v1.8
[    2.100000] VFS: Mounted root (squashfs filesystem) readonly on device 31:4.
[    2.100000] Freeing unused kernel memory: 224K (80388000 - 803c0000)
[    3.350000] init: Console is alive
[    3.350000] init: - watchdog -
[    3.360000] random: kmodloader urandom read with 5 bits of entropy available
[    5.490000] usbcore: registered new interface driver usbfs
[    5.500000] usbcore: registered new interface driver hub
[    5.500000] usbcore: registered new device driver usb
[    5.560000] SCSI subsystem initialized
[    5.570000] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    5.580000] ehci-platform: EHCI generic platform driver
[    5.580000] ehci-platform ehci-platform: EHCI Host Controller
[    5.590000] ehci-platform ehci-platform: new USB bus registered, assigned bus number 1
[    5.600000] ehci-platform ehci-platform: irq 3, io mem 0x1b000000
[    5.620000] ehci-platform ehci-platform: USB 2.0 started, EHCI 1.00
[    5.620000] hub 1-0:1.0: USB hub found
[    5.620000] hub 1-0:1.0: 1 port detected
[    5.640000] usbcore: registered new interface driver usb-storage
[    6.470000] init: - preinit -
[    9.720000] eth0: link up (100Mbps/Full duplex)
[   10.690000] jffs2: notice: (348) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
[   10.700000] mount_root: switching to jffs2 overlay
[   10.750000] eth0: link down
[   10.770000] procd: - early -
[   10.770000] procd: - watchdog -
[   11.460000] procd: - ubus -
[   12.470000] procd: - init -
[   14.000000] NET: Registered protocol family 10
[   14.050000] ip6_tables: (C) 2000-2006 Netfilter Core Team
[   14.080000] Loading modules backported from Linux version v4.4-rc5-1913-gc8fdf68
[   14.080000] Backport generated by backports.git backports-20151218-0-g2f58d9d
[   14.100000] ip_tables: (C) 2000-2006 Netfilter Core Team
[   14.150000] mmc_spi spi0.1: SD/MMC host mmc0, no DMA, no WP, no poweroff
[   14.160000] nf_conntrack version 0.5.0 (954 buckets, 3816 max)
[   14.180000] mmc0: host does not support reading read-only switch, assuming write-enable
[   14.190000] mmc0: new SDHC card on SPI
[   14.200000] mmcblk0: mmc0:0000 SD16G 14.4 GiB 
[   14.210000]  mmcblk0: p1 p2 p3
[   14.250000] usbcore: registered new interface driver ums-alauda
[   14.280000] usbcore: registered new interface driver ums-cypress
[   14.290000] usbcore: registered new interface driver ums-datafab
[   14.290000] usbcore: registered new interface driver ums-freecom
[   14.300000] usbcore: registered new interface driver ums-isd200
[   14.320000] usbcore: registered new interface driver ums-jumpshot
[   14.330000] usbcore: registered new interface driver ums-karma
[   14.340000] usbcore: registered new interface driver ums-sddr09
[   14.350000] usbcore: registered new interface driver ums-sddr55
[   14.360000] usbcore: registered new interface driver ums-usbat
[   14.370000] usbcore: registered new interface driver usbserial
[   14.370000] usbcore: registered new interface driver usbserial_generic
[   14.380000] usbserial: USB Serial support registered for generic
[   14.420000] xt_time: kernel timezone is -0000
[   14.430000] usbcore: registered new interface driver ark3116
[   14.430000] usbserial: USB Serial support registered for ark3116
[   14.450000] usbcore: registered new interface driver belkin_sa
[   14.450000] usbserial: USB Serial support registered for Belkin / Peracom / GoHubs USB Serial Adapter
[   14.530000] usbcore: registered new interface driver ch341
[   14.530000] usbserial: USB Serial support registered for ch341-uart
[   14.540000] usbcore: registered new interface driver cp210x
[   14.550000] usbserial: USB Serial support registered for cp210x
[   14.550000] usbcore: registered new interface driver cypress_m8
[   14.560000] usbserial: USB Serial support registered for DeLorme Earthmate USB
[   14.570000] usbserial: USB Serial support registered for HID->COM RS232 Adapter
[   14.570000] usbserial: USB Serial support registered for Nokia CA-42 V2 Adapter
[   14.580000] usbcore: registered new interface driver ftdi_sio
[   14.590000] usbserial: USB Serial support registered for FTDI USB Serial Device
[   14.700000] PPP generic driver version 2.4.2
[   14.700000] NET: Registered protocol family 24
[   14.780000] ath: EEPROM regdomain: 0x0
[   14.780000] ath: EEPROM indicates default country code should be used
[   14.780000] ath: doing EEPROM country->regdmn map search
[   14.780000] ath: country maps to regdmn code: 0x3a
[   14.780000] ath: Country alpha2 being used: US
[   14.780000] ath: Regpair used: 0x3a
[   14.790000] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
[   14.790000] ieee80211 phy0: Atheros AR9330 Rev:1 mem=0xb8100000, irq=2
[   23.820000] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   23.880000] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[   26.110000] eth0: link up (100Mbps/Full duplex)
[   26.110000] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   30.070000] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[   30.280000] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[   30.380000] IPv6: ADDRCONF(NETDEV_UP): adhoc0: link is not ready
[   32.620000] adhoc0: Created IBSS using preconfigured BSSID 02:ca:ff:dd:ca:ce
[   32.620000] adhoc0: Creating new IBSS network, BSSID 02:ca:ff:dd:ca:ce
[   32.630000] IPv6: ADDRCONF(NETDEV_CHANGE): adhoc0: link becomes ready
[   36.180000] EXT4-fs (mmcblk0p2): couldn't mount as ext3 due to feature incompatibilities
[   36.190000] EXT4-fs (mmcblk0p2): couldn't mount as ext2 due to feature incompatibilities
[   36.240000] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[   36.320000] EXT4-fs (mmcblk0p3): couldn't mount as ext3 due to feature incompatibilities
[   36.330000] EXT4-fs (mmcblk0p3): couldn't mount as ext2 due to feature incompatibilities
[   36.400000] EXT4-fs (mmcblk0p3): mounted filesystem with ordered data mode. Opts: (null)
[   37.540000] random: nonblocking pool is initialized

Last modified: Fri Apr 28 09:50:53 NZST 2017

So now we can boot a Mesh Extender in under 90 seconds and know where it is up to in the meantime.

In the process, I discovered that using ext2 instead of ext4 was a really bad idea, as on the slow microSD interface we have checking the filesystem can take 8 minutes, even when empty.

Now to update the other seven, and get out into the nice New Zealand sunshine that has been taunting me all morning, and start testing!

Sunday, April 23, 2017

Preparing for exercise with NZ Red Cross

For our DFAT Pacific Pilot in Vanuatu, I am getting ready to fly out to New Zealand for a training exercise with NZ Red Cross's IT & Telecommunications Emergency Response Unit (IT&T ERU), with whom we have previously tested the Serval Mesh and earlier prototypes of the Mesh Extenders.

This time, the plan is for me to build up a fly-away kit containing Mesh Extenders, solar panels, batteries and all the necessary cables and bits and pieces for a complete functioning system.

The fun is that we don't yet have all the parts on hand, and there is still quite a bit of work to do.  So so far, the box only has me in it:

By the end of the day, it should instead have all the prototype hardware inside ready to fly out to NZ early tomorrow morning.  It is has occurred to me that this means that this Anzac Day I will be flying to a joint Australian / NZ exercise designed to protect human life.  Perhaps not on the same level as those who served in war, but it certainly reminds me of their service.

Between now and then, aside from all the paper-work, I need to:

1. Assemble 8 Mesh Extender prototypes;

2. Build 8 power/radio ID cables;

3. Program those cables;

4. Build the final firmware image to be used for the exercise and install them in the prototypes;

5. Install and prepare the microSD cards in them all;

6. Test that they work.

7. Figure out the correct way to carry LiFePO4 cells with me to NZ.

So, I'd better get to it!

Tuesday, March 28, 2017

Teaching the RFD900 about uboot, so that it doesn't interrupt the Mesh Extender boot process

The Serval Mesh Extenders logically consist of a CPU with a serial port, and an RFD900 radio.  Our RFD900 firmware provides a regular heart-beat.

This all means that the RFD900 radio, as soon as it powers on, begins sending heart-beat messages.

This ordinarily wouldn't be a problem, except that the CPU modules we are using have only one serial port, so the serial boot loader is listening on the serial port when the system first powers up, and whenever it resets.

This means that if uboot sees a heartbeat, it will interpret it as the user wishing to interrupt the boot process to interact with the boot loader.  Following Murphey's Law, this is assured to happen on every boot.

We can't get rid of uboot, as it is important to have a way to reflash the devices (especially while we are still developing the software for them).

The remaining solution is to teach the RFD900 to notice when uboot is active.

In theory, this would just take a simple little finite state machine to notice the uboot banner, and then make sure it doesn't send any characters for a while, to allow uboot to boot normally.

There are, however, two complications.

1. We have to stay silent for long enough that we don't interfere with OpenWRT's fail-safe boot mode, which just means staying silent for longer.


2. The RFD900 is set to 230400bps for talking to LBARD, while uboot uses the serial port at 115200.

It is this last problem that makes life more interesting, because we can't just watch for the characters of the uboot banner. Instead, we need to look for a character sequence at 230400bps that is what you would see if you send the uboot banner at 115200bps.

As uboot is working at 1/2 the baud rate that the RFD900 is using, every serial bit received from uboot will be interpreted as two bits. Thus each uboot serial character should be able to be reliably received as two characters by the RFD900. We're lucky it isn't the other way around, as then the bits would be only half as wide, and there would be some uncertainty about how the bits would be read.

In the process, I also had to clean up a few other spots where our RFD900 firmware was causing other characters to be emitted immediately on power-up.

So after a few frustrating hours of debugging the finer points, I can now reliably boot a uboot-based system with an RFD900 running our packet radio firmware permanently attached to the serial port.

Wednesday, March 15, 2017

The custom IP67 power cables have arrived!

So, just over ten days ago I wrote that a Chinese manufacturer said that they could manufacture and ship the cables we want within ten days, and that I would order some.  Today they arrived!  Some fun unpacking photos:

First up, I must congratulate the Chinese manufacturer for being able to manufacture and ship our order so quickly.  They really were a pleasure to work with, including when we had a minor problem with the shipping (which wasn't their fault).

Also, as the photos hopefully show, the cables are very solidly built, with much thicker wire than the little ones we were contemplating using previously.  They will be able to easily support large panels and batteries, with charge currents being limited only by the battery charge controller inside the Mesh Extender. The cables simply give the strong impression that they will do the job well, which we will test over coming months.

The only complication we encountered, was that they didn't have stock of the white cable material, so we had to opt for the black PVC cable, which will slightly increase heat loading of the Mesh Extenders.  But given the time frames that we are working to at the moment, we needed to simply be sure that we would have the cables we need, so that we can get the cable head moulding tools made and cables assembled in time.  Critical paths are everything when building hardware!

Sunday, March 5, 2017

Sourcing cables directly from a Chinese manufacturer

I'm still working on the power/radio-regulatory cable for the Mesh Extenders.  Fortunately, this process is drawing towards its natural ending.

My current challenge is to find suitable UV-stable and IP65 or better rated cables.

We have already found some on, but they leads are only ~20cm long. I'd much prefer leads approximately 0.5m - 1m long, so that there is ample length for attaching solar panels and battery packs, and general ease of connection.

A bit of hunting on yielded an enquiry from Bett Electronic Co. LTD in Shenzhen.

They have exactly the kind of thing we are looking for, and nicely, their cables are thicker (18-AWG instead of 22-AWG and 24-AWG) than the ones we found on BangGood, which means lower resistance.  A further bonus, is that they can manufacture in white, which will reduce heat loading a touch. Their cables are also IP67 certified.  Also very nice.  Here are the pictures that they sent of a sample:

Also, as the manufacturer, their prices are quite good.  Given that we are looking for cables that are 87cm long on average (compared with 20cm), and with the heavier-duty wiring, it is pleasing that the price is only about 50% higher.  They can also manufacture and ship quite quickly, apparently being able to get 100 units to us within about 10 days, including shipping by DHL.

So the only complication is that they don't have any certification of UV performance that I could use to easily determine whether they would be suitable.  However, Leah was very helpful and patient in answering my many, many questions.

What I now know is that the white cables are EPDM with added Titanium Oxide, which should have excellent UV and weathering properties.  The connectors themselves are PVC, without any special UV stabilising material.  However, according to this document, the main problem with PVC under UV, is that it will discolour (not a problem), and will have reduced impact resistance (only a minor problem).

Otherwise, the only minor issue, is that I can't figure out if the PVC is Lead-free or not. However, given the small mass of the connectors, this is not an immediate concern.  Nonetheless, I would love to be able to be sure that there was no Lead in the PVC.

So it looks like we have a good solution, and I think I will place the order with Bett Electronic in the next day or so.

Wednesday, March 1, 2017

Reading and writing the I2C EEPROM connected to an RFD900

As previously described, for the new Mesh Extender we have an I2C EEPROM for encoding regulatory information, e.g., radio permitted frequency and TX power.

Basically, I am building the tools that we will need to program Mesh Extender cables as we build them, and also in the field, to reprogram them as required.

Over the past couple of weeks I have worked on the code for the RFD900+ firmware to be able to read and write the I2C EEPROM, and then to modify our flash900 utility to be able to flash not just an RFD900 radio, but also an I2C EEPROM connected via an RFD900 radio.

I've had it mostly working for a while now, but there have been strange problems with accessing certain parts of the EEPROM, and weird intermittent write problems affecting the whole EEPROM. I tried swapping EEPROMs, to confirm that it wasn't a faulty part.

I spoke to the engineering workshop folks here, who suggested that I might need external pull-up resisters to terminate the I2C bus, which fortunately turned out not to be the case, as this simplifies the cable and PCB design process (one of them would have required some extra resisters).

They also had a nice little logic analyser that could understand I2C.  That proved to be super-helpful.  It took me less than an hour to see that the problem was when back-to-back page reads were occurring and the last bit of the last byte read was a zero.  Basically I was failing to implement the NACK required at the end of a multi-byte read sequence.  

I fixed this by inserting a dummy read with NACK, like this:

$ git show 490025c33a3e821aab05055d627d192bb0d186fc               
commit 490025c33a3e821aab05055d627d192bb0d186fc                   
Author: gardners <>                         
Date:   Thu Mar 2 14:02:22 2017 +1030                             
    properly terminate sequential read operations                 
diff --git a/Firmware/radio/i2c.c b/Firmware/radio/i2c.c          
index 76fe0be..6e9d68f 100644                                     
--- a/Firmware/radio/i2c.c                                        
+++ b/Firmware/radio/i2c.c                                        
@@ -183,6 +183,9 @@ char _eeprom_read_page(unsigned short address)
+  // Terminate the sequential read                               
+  i2c_rx(0);                                                     
   return 0;                                                      

With that in place, I could suddenly reliably access the whole EEPROM.  In fact, the correct data was always written there, it was just that I couldn't read it.

Following that, I spent a bit of time optimising the EEPROM read and write speed, and also the speed of the flash-rfd900 utility, both for flashing the radio and flashing the EEPROM.

It is now possible to flash an RFD900 radio in about 23 seconds, even when using a USB serial adapter with 16ms latency.  This is a huge improvement on the ~100 - 150 seconds it took previously.

Flashing the I2C EEPROM takes just under 20 seconds, including reading the entire EEPROM once before hand, and once after, to verify that all bytes have been correctly written.

Now I just need to tweak the format of the data that I store in the EEPROM, so that it contains all the information that I need, principally the list of countries a particular cable is suitable for, and increasing the resolution of the frequency parameter from MHz to KHz.

Monday, February 27, 2017

Attaching Serial Numbers

Today I have been researching options for permanently marking Mesh Extenders with serial numbers on the housing, cables and PCBs.  This is complicated by the tropical-maritime operating environment, and that the housings are to be made of poly-carbonate.

Poly-carbonate can't be safely laser-cut or laser-etched, because it gives off toxic and corrosive chlorine gas.

Tropical-maritime environments eat glue for breakfast, and abound in UV radiation to degrade everything.

So far, the best option I have found is to use a laser-etchable UV laminates, e.g.:

With an appropriate glue.

They do supply them with either 3M 467MP adhesive tape, which claims to be good for polycarbonate, and tolerates high heat etc, or a similar TESA adhesive.

Otherwise, you can get the laminates with no adhesive, and then apply your own, such as Dymax 3025 or 3099, which seem like they should be stronger than the 3M or TESA options.

The laminates themselves are quite cheap, less than AU$20 for 30x60cm. Assuming 4x5cm labels, this gives a material cost of only about AU$0.25 per label (plus glue) -- much less than the cost of the housing.  We will likely need one 4x5cm label and two 2x5cm labels, so that we can have safety labels on the insides, incase the one on the outside falls off some how.

For the PCB, we can just have a silk-screen white panel to write the serial number and manufacture date on using a permanent marker, so no great problems there.

For the D-SUB 25-pin cables, we are looking at a nice solution to get those professionally manufactured with a low-pressure over-mould process, for about $14 per cable, plus parts.  I need to have a chat to those folks to see what our options are there for imprinting them with serial numbers during production, or failing that, whether we can laser-etch or otherwise mark those.

Tuesday, February 21, 2017

SPI microSD card on Atheros 9k

For the new Mesh Extenders, we want to use a microSD card for bulk storage, instead of a USB memory stick.  This is in part because USB draws a lot of power just to run the bus (about 10% - 20% of the total power consumption of a Mesh Extender), but also because we have had a lot of reliability problems with USB memory sticks in Mesh Extender prototypes. Basically, they don't like being shut down suddenly.

Our preferred solution was to add a microSD slot to the Ath9k modules we are using (the Core Domino).  However, we could not find any good sources on how to do this. First, we tried using the bit-bashing driver, which we did get working, however it was horribly slow (about 10KB/sec), and would routinely freeze or reset the Ath9k (possibly due to watchdog timeouts?).

The Atheros 9k does have an SPI interface which is compatible with microSD cards, however, after searching for people having used it, I couldn't find anything sensible.  The main problem is that the 16MB FLASH that boots the Mesh Extender is also attached to the SPI bus, and so there is a need to arbitrate between that, and our microSD card.

After I had given up hope and was trying to make the bit-bash driver work, I accidentally found a page that described exactly what I wanted to do, and had some simple patches for OpenWRT CC to implement it.  You just had to assign a GPIO to be the chip-select for the microSD slot, and all would be well.  Unfortunately I had no spare GPIOs.

What I did have, however, was the chip-select line for the 16MB FLASH.  Since I had only two devices, I suspected that I could just invert that signal, and feed it as the chip-select to the microSD slot.  As hoped, this worked just fine.

The total changes were very small:

Note that while I didn't have a spare GPIO, this approach still requires that you assign some GPIO to be the chip-select line. I chose a GPIO that was not exposed on the board I was using.  For many low-cost wireless routers, like the TP-LINK WR703N and friends, there are plenty of such blind GPIOs to choose from.  Indeed, this microSD hack should be fairly easily adaptable to the 703N, MR3020, MR3040 and many other cheap Atheros 9331 (or similar Atheros parts) wireless routers, such as the very nice GL-AR150 and related family.

Update: I have spoken to the nice people who make the GL-AR150 and related devices, and they are taking a look at this for evaluation.

Mesh Extender EEPROM-equiped cable work

One of the novel features of the new Mesh Extender, is that the national radio regulatory parameters will not be stored in the main unit, but rather in the power cable, as I have described previously.

This means that we can, the theory at least, have bunches of Mesh Extenders pre-positioned regionally (or even globally), without having to configure them for each country in which they will be used. Instead, we just need to have batches of country-specific power cables for them, or similarly, program sets of power cables for them, and then send those to the relevant country/countries.

It also means that we can deal with unfortunate regulations that require firmware be locked in some countries, by having the firmware lock encoded on the power cable as well.

To implement all this, we are using 2KB I2C EEPROMs, so that we can have quite a bit of information stored, and also re-program the cables from one country to another when the need arises.

We want the RFD900/RFD868 radio to read this information directly when they power up, so that they can set their radio frequency and TX power levels appropriately.  This means it makes sense to have the RFD radio talk directly to the I2C EEPROM, rather than via the Ath9k CPU.  A second reason for doing it this way is that we don't have enough GPIOs left on the Ath9k module we are using.

All this means that I needed to implement an I2C library in the RFD900 firmware, and as mentioned in the previous post, implement the SHA-3 hash function, so that we can validate that the parameters are intact, and have not been corrupted by problems with the I2C EEPROM.

I found some nice reference information about I2C, and apart from some challenges at getting the timing and protocol right, it only took a few days work to get to the point where I can now reliably and quickly read and write the EEPROM.  This meant spending a lot of time staring at oscilloscope displays like this one.

For development, I wired up a little board with one of the EEPROMs to an RFD900, to simulate the wiring for the cable.

One nice little trick that we do to save power consumption, is that the EEPROM is powered directly from one of the GPIOs on the RFD900.  If we turn that GPIO off, then the EEPROM is physically switched off, saving a (very) little bit of power.

For the actual cables, we have the pinout planned such that the EEPROM can be soldered directly onto the relevant pins of the D-SUB 25 connector, to simplify assembly and avoid the cost of a little PCB, something like this:

Bending the pins the right amount was a bit more work than hoped, but with practice (or a little jig) could be done quite quickly. This is mainly because the chip is quite a bit fatter than the pitch between the rows of pins.  The pins are the same pitch as the pins within a row on the connector, however, so that aspect is fine.

This theme of simplifying assembly is, I am finding, a recurring one as I plan how to make something that can actually be manufactured, even in modest quantities.  All manual steps need to minimised so far as possible.  This means actually trying to do things, rather than just designing them.  So half way through writing this post, I wandered downstairs and tried to solder one of these cables together. I succeeded eventually, more or less, but in the process realised that to just fit the chip onto the back of the D-SUB connector, two pins needed to be swapped around.  For now, I have soldered it up with the temporary fix, until we can get this pin change made on the next revision of PCBs (which fortunately haven't been fabricated yet). You can see the resulting mess here:

Here is the whole thing with the nice $1.35 a piece IP65-rated connectors I found recently:

Apart from the pinout problem, it wasn't too hard to assemble, really, so I am encouraged that this could still work.  I won't stop thinking about how I can further optimise the process, however.

What I haven't covered above is how I am going to protect the cable head, so that it is sealed etc.  For that, I have been talking to the plastics manufacturing people who are helping us with the injection-moulded housing design.  They have suggested we could fairly cheaply produce a tool for low-pressure plastic encapsulation of the cable, similar to what is used to produce serial and printer cable ends.  Because it is low pressure, it is possible to use aluminium for the tool, rather than the hard tool steel that is required for the injection-moulded housing.  Because aluminium is so much softer, it is quicker (and thus cheaper) to machine.

8-bit implementation of SHA3

We have need of a hash function in the Mesh Extender for the RFD900 radio to verify that the radio regulatory parameters it is loading from the serial EEPROM in the power cable have not been corrupted.

Taking a look at the current major families of hash functions, SHA-1, 2 and 3, it looks like SHA-3 is the best.  This is not just because it is the newest (incorporating the latest advances in cryptography against known attacks, compared with SHA-1, which is showing signs of vulnerability), but because it is also relatively computationally efficient (compared with SHA-2, for example), and friendly for running on simple hardware.

I found a nice public domain implementation of SHA-3, and tried to pull that into the RFD900 firmware.  However, it upset the compiler in a variety of ways.  In short, the SDCC compiler really doesn't like 64-bit data types, and does horribly inefficient (and incorrect) things with them.

Result: I have worked through the implementation converting it to a byte-oriented implementation of the full SHA-3 hash.  It correctly calculates the hashes of various test-vectors I have fed it.  What is nice, is that the implementation is now quite a bit shorter and simpler, because I have removed some optimisations for 64-bit machines from the code.  You can see the code here:

For very short input texts, I can calculate a SHA-3 hash in under a second, which isn't too bad, given the little 8051 CPU on the RFD900 board.

Thursday, February 2, 2017

Potential Mesh Extender Use-Cases: Farms

I was talking with a colleague about our plans for the Mesh Extender in farming, and realised that I couldn't find anything written down about our thoughts in this space, so I am going to fix that right now.

While we had been thinking about it for a while, the need for something to improve the productivity and competitiveness of family farms in remote locations here in Australia was really bought home to us by Craig from the Evening Star Caravan Park in Charleville, Outback Queensland.

Craig pointed out the cycle of decline in many remote towns caused by the shift from family ownership of farms to corporatised farming.  Basically, family farms mean that there is an extra family in a community, with kids who need schooling, and more warm bodies who need health and other services.

Corporate farming tends to use less people, in order to maximise economic efficiencies. Unfortunately, this means that a town often loses a family or two in the process. Then there are less kids at the school, so the school might lose a teacher (and the teachers family).  Then the hospital (if they are lucky enough to have one) loses a nurse (and the nurses family), because the population has reduced, and so on.  At the same time, there are now less mouths to buy food from the local supermarket and stores, so they reduce the hours of their workers, which further reduces the money that they have to spend in the community and so on.  Basically, the loss of farming families causes a continual erosion of the viability of these communities.

However, it isn't easy to run a family farm, when faced by the economies of scale and other efficiencies that corporate farming is often able to harness, including the use of information technology to remotely manage their properties, which are often very large -- upto 10s of thousands of square kilometres (although in the past the largest such operation exceeded a staggering 220,000 square kilometres -- or about the size of the entire UK).

Basically, it is much cheaper, and therefore profitable, if you are able to manage your property from a network operations centre or station homestead, instead of having to drive potentially hundreds of kilometres over rough station tracks.  However, the communications solutions currently on offer to do this are either two simplistic, e.g., analog water tank level sensors, or are so complex as to require dedicated personnel, and are simply not maintainable by the family farmers, many of whom are in their 60s and 70s.  Also, the systems are simply too expensive.

The simplicity designed into Serval from the beginning has the potential to change this, by making it possible to create remote monitoring and control networks that do not require any cellular coverage (about 2/3 of Australia has absolutely no cellular coverage at all), or incur the running costs associated with it.  This is part of why we designed Mesh Extenders with "Internet of Things" like inputs and outputs -- although of course we don't need internet access, so it is really more "Mesh of Things" (MoT).  We have already prototyped a simple MeshMS control interface to support this, so that a farmer could just subscribe or unsubscribe to alerts from a sensor by sending a text message to the sensor.  Also, they would be able to do this from anywhere on their properties, instead of being tied down to a centralised control interface.  This last point is not to be underestimated in its impact.

That is, we think we can make something which is 10x cheaper than many of the existing solutions, and 100x simpler -- and that wouldn't need farmers to pay for expensive consultants to fly and drive in from big cities -- they could progressively deploy and maintain it themselves.

Of course, it isn't just Outback Australia that could benefit. Anywhere where the capital expense, availability (or affordability) of communications, or the lack of remote control and monitoring impairs the productivity of farms.

Basically, we hope that our humanitarian technology will be helpful in more than just one situation.  In return, it might just help to create additional markets that can help us to reduce the cost of Mesh Extenders for all users.

Monday, January 30, 2017

Looking for ideas for cable and connector parts

I'm currently looking at what to use to build the power cables for the Mesh Extenders.

As previously posted, one end has a D-SUB 25 socket.  The other end needs to have connectors for:

a solar panel and battery (4 pins),
USB socket (to allow charging of phones from Mesh Extender, where power budget allows),
12/24v automotive power

and it also needs to have a little serial EPROM in the cable, so that the cable can tell the Mesh Extender which radio regulatory region it is in.

All of this needs to be fairly weather proof, so that it can ideally last for at least a year in tropical maritime or outback conditions.

It also needs to be as cheap as possible.

I'm looking for ideas for all parts of the cable, to make the cables as fast and cheap to build as possible, while maintaining adequate performance.

My current thinking is leaning towards:

Using half of a for the D-SUB 25 end, then putting some currently unknown small sealed junction box on the end of that, and having the solar/battery, USB and automotive power plugs (not all may be fitted on all cables) come out the other end of the junction box.  

Those D-SUB connectors have the problem that they are not UV stabilised or IP rated, however.  Are there good simple coatings (like would spraying them with Armor-all be enough) for UV protection?  Would the connector, sitting inside a lip at at the bottom of the Mesh Extender exclude dust and water adequeately (we can at least test that once we have complete hardware units on hand)? Or does anyone know an affordable source of similar cables that are UV stabilised, and ideally, IP66 or similar rated?

For the solar/battery connector, perhaps something like, which are outrageously cheap, and already come with tails fitted, so soldering them up on a tiny PCB in a junction box would be easy.  They claim to be UV resistant and IP65 rated, although I must confess I have some suspicions given their very low cost.

I currently have no decent leads on a small junction box that can take a fat cable in one end, and some skinny ones out the other, sealing everything in the middle. Something like the things that Telstra use to join phone cables in pits would be a potential solution (not Scotch-locs, but the bigger things).  But I don't know where to source those from (yet).

Friday, January 27, 2017

Mesh Extender 2.0 Exterior and Interior drawings

Just a quick post with some images that have been generated so far during the design process, to give some idea of how the thing will fit together.  The exact dimensions and details are likely to have changed before we produce physical units, but give a clear idea on the concept.

First, from the front:

 Here  you can see the three antenna mounts: 2 for the UHF packet radio, and 1 for the WiFi. The other two bumps are screw bosses, so that you can make your own shade for hot locations to help keep the innards below 70C.  We have included this, because the thermal budget is just not quite there  for full-sun tropical locations, while sticking to only passive cooling, and with everything in an environmentally sealed case (which also traps heat).  Adding some shade fixes this problem.

We are not at this stage designing a shade, as we figure that they can be easily made by users from whatever they have laying around, for example, an old plastic bucket, waxed cardboard box, or even a drink can split open.

From behind:

The main thing to see here is the curved back to make it easier to strap to a tree or pole, and the ribbing to help provide at least a little air-flow between the unit and whatever it is attached to.

From below:
Here you can see the D-SUB 15 utility/IoT and D-SUB 25 power/radio connectors.  Also, below those you can see a couple of round markings which are drill guides in case you want to get ethernet out.  In their default state, they have gortex moisture seals to allow the unit to maintain equal pressure with the outside, without filling with condensation.

The inside of the top:

Here the main feature to see is the notch and paddle to help hold the PCB firmly in place, to minimise the effects of vibration, e.g., if a unit is mounted on a vehicle.

The PCB itself will sit inside, with fly-leads to the antennae, as shown in the remaining images.  Returning to the thermal properties, the main means for heat to exit from the unit is conductance through the antennae connectors.  Not ideal, but when you are making a sealed unit that you don't want to cost thousands of dollars per piece, your options can be rather limited.

Wednesday, January 18, 2017

Testing Mesh Extender 2.0 Build Process with Farouk from INRIA

This week Farouk from INRIA in Lille, France has been visiting, as we discuss the potential for collaboration.  It looks like what we are doing with Serval and the Mesh Extenders at the moment will be a good fit for the needs of a project that they have at the moment.  As a result, we have spent a couple of days making sure that Farouk can replicate our build process, which also gives us confidence for others building firmware images as well, and transferring as much of the necessary know-how as possible before he heads back to France, and then in a just a couple more days, that I head back to Australia.

In this process, we are also trying to document all of the outstanding software and hardware issues for the Mesh Extenders at, so that we can better coordinate our activities, and also hopefully make it easier for others to contribute or replicate our work.

Tuesday, January 17, 2017

Testing the Super-Capacitor

We have included a 1F 5.5V super-capacitor on the new Mesh Extender board.

The idea of this is so that when power is lost, the unit has a couple of seconds to stop writing to the microSD card, and perhaps tidy things up to ensure a clean boot next time.

However, we haven't managed to get it to work properly yet.

The boards have a 20 Ohm resister in-line with the super-cap, so that there is no unmanaged in-rush current when charging the capacitor from empty.

The capacitor takes a few minutes to fully charge to about 4.8V (5V - the voltage of the zener diode that forms part of the power loss detection circuit).

At this point, it should be possible to then remove the power source, and have the unit operate powered solely by the super-capacitor, until its voltage level drops below about 3.3V.

GPIO24 is the line that reads our input-power-available signal (effectively the input power before passing through a zener diode, after which it charges both the super-capacitor, as well as powering the rest of the board.  To test the system, I configured that GPIO line for input, and had a little shell script that sits in a loop reading from it.

root@OpenWrt:/# cd /sys/class/gpio
root@OpenWrt:/sys/class/gpio# echo 24 > export
root@OpenWrt:/sys/class/gpio# cd gpio24
root@OpenWrt:/sys/devices/virtual/gpio/gpio24# while [ true ]; do cat value; don


It reports 1 when there is an input power supply (other than USB, which is another little problem we need to solve), and 0 when there is no input power supply, but is running on the super-capacitor.  At least that is the theory.

However, with the 20 Ohm resister, the voltage on the low-side is only about 2.2V, much too low to be useful.  Thus I tried putting a 10 Ohm resister in parallel, dropping the combined resistance to about 6.7 Ohms.  That still wasn't enough.

Then I tried bridging the resister by using a multi-meter in current measuring mode after having already charged the capacitor.  This gives effectively 0 Ohms, and provides an upper bound of what is possible with this super-capacitor.  With that configuration, things were more promising, as I saw u-boot try to boot, but as can be seen, as soon as it tried using the DDR RAM, it would reset, as presumably the DDR RAM draws too much power for the super-cap to sustain the supply above 3.3V:


*        U-Boot 1.1.4  (Jun  6 2015)        *

AP121 (AR9331) U-Boot for DOMINO v1


*        U-Boot 1.1.4  (Jun  6 2015)        *

AP121 (AR9331) U-Boot for DOMINO v1


*        U-Boot 1.1.4  (Jun  6 2015)        *

AP121 (AR9331) U-Boot for DOMINO v1


*        U-Boot 1.1.4  (Jun  6 2015)        *

AP121 (AR9331) U-Boot for DOMINO v1

*        U-Boot 1.1.4  (Jun  6 2015)        *

AP121 (AR9331) U-Boot for DOMINO v1

*        U-Boot 1.1.4  (Jun  6 2015)        *

AP121 (AR9331) U-Boot for DOMINO v1


As can be seen above, there is no line with 0, before it hits the u-boot reboot loop, indicating that the board does not operate on the super-capacitor,  even for just a few milliseconds.

We'll have to think of an appropriate solution to this problem, but first we need to understand the situation more completely.  

Is the capacitor unable to provide enough current for the DDR RAM?

Is the capacitor too small?

Also, only one of three super-capacitors seemed to work.  The other two might have been damaged at some point in the process, but having 2/3 faulty makes me think that we shouldn't be relying on them.