After the successful bug hunting last week, this week I have turned my attention to setting up to easily verify whether the bugs have been fixed. This means setting up a multi-hop Mesh Extender network in a convenient location.
The building I am based in was purpose-built for the College of Science & Engineering, and we were able to have a number of specific provisions included during the design phase to make this kind of test network easier to deploy, and we are now finally able to start making fuller use of that. More specifically, we have 13 points around the building where we have copper and fibre directly to our lab, and also have some funky mounting brackets, on which we can fit radio hardware, all nicely protected from the outside weather.
So the obvious approach was to fit several Mesh Extenders to some of those points, and form a multi-hop UHF network. Here are some photos from various angles showing the units:
First up, we have Mesh Extender serial #23 with SID BCA306B0*, which is at point NE1. Note the extra long 6dB antenna we fitted to this one to make sure it could talk to the one at N4 (1A15ED32*). Even with that antenna, the link is quite poor, often getting only one packet every 5 to 10 seconds. While on the one hand annoying, this is actually quite helpful for testing the behaviour of Mesh Extenders in marginal link conditions.
Mesh Extender serial #34 is at point N4, and has SID 1A15ED32*:
Note that we have removed the Wi-Fi antenna from this one. This was necessary to make sure that it can't communicate via Wi-Fi with any of the other units, which it was otherwise able to do intermittently, due to being relatively near to the one in the lab, which is on the same floor, and the one on level 5, to which it almost has line of sight up to via the stair case.
Mesh Extender serial #50 is at NW5 and has SID 5F39F182*:
And finally, Mesh Extender serial #17 (6ABE0326*) is sitting on the rack in our lab:
Mesh Extender #34 (1A15ED32*) is the only node that can see all the others, which we confirmed by using the debug display on LBARD:
I was still fiddling around with things at this point, but you can see already that the RSSI for BCA306B* is quite a lot lower than for the others. Although here it looks like the link is healthy, it really does come and go a lot, and goes seconds at a time without a single packet arriving.
The last of the four rows, the one showing 39F1821A* is actually a known-bug manifesting. This is a ghost connection, caused by receiving a packet from 5F39F1821*, with the first byte having gone walk about for some reason. We are still to work out what the cause of this problem is. It's on our bug list to find and fix.
Now, to give a better idea of how the network is laid out, here are a couple of shots from the outside of the building. The first is looking along the northern facade to give a sense of the length of the building. It is from memory about 50m long.
And here is the whole facade:
The glass treatment means that very little signal can actually get out of the building, bounce of something, and find its way back in. We estimate the attenuation to be about 15dB at 923MHz, so a total of 30dB loss to pass out of the windows, and back in again, not counting any free space losses etc. Thus we suspect that the links between the units are due to internal reflections in the building, and perhaps a bit of common mode radiation along metal parts of the building interior. It is almost certainly not due to transmission through the ~30 - 50 cm thick reinforced concrete floor decks of the building.
Given that we are in a building with such poor RF propagation properties, it is rather pleasing that we are able to get any link at all between the first and fourth floors, given the multiple concrete decks and other obstacles that the link has to face.
So, with the network in place, it was time to do a little initial testing with it. For this, I had a phone running Serval Chat in the lab, and I went down to level 1 by BCA306B0 and sent text messages, using the automatic delivery confirmation mechanism to know that a message has got to the other end, and indeed that a bundle has made its way back containing the acknowledgement.
First, we tested using the old version of LBARD, and were able to see the failure of messages to deliver that we had seen in the field. Then it was a visit to all the Mesh Extenders to update LBARD (we will in the near future supply them all with ethernet connections back to the lab, so that we can remotely upgrade them). The result after upgrading LBARD was messages were reliably delivering, taking between about 1 and 4 minutes to get the delivery confirmation. This was of course a very pleasing and comforting result, as the inability to reliably deliver messages was the most critical issue we were seeing in Vanuatu.
The large variation in time we expect to be simply due to the very high rates of packet loss that we see on the link between BCA306B0 and 1A15ED32 -- something that we will confirm was we continue testing. The main thing is that we now have a test environment that is both convenient, and realistic enough, for us to reproduce bugs, and confirm that they have been fixed.