Crossing the Charles River by Mesh Extender

I am just about to head off to the airport to head home to Australia, but had the chance last night to do one last range test with the Mesh Extenders here in Boston.

Again I put one Mesh Extender on the wardrobe here in room 704 at the Kendall Hotel: 

I then caught up with Jon from the Awesome Foundation, and we took advantage of the warm evening to walk across the bridge to the far side of the Charles River to see if we could get signal over there.

Once we were on the far side, we were able to obtain an adequate signal over a stretch of around a kilometre along the far bank.  Here I am with the Mesh Extender, connected in a single hop to the Mesh Extender in my hotel room.

To give an idea of where we were, and the distance of the link, see the image below. We also had link at most points along the Charles River Reservation IV.

While we were obviously down at ground level, I was conscious of the many apartment buildings along the foreshore of Back Bay East that would have even better for linking across the Charles River.

There are probably hundreds of apartments along Storrow Drive that would have been able to get a link to the Kendall Hotel with a Mesh Extender, again confirming my view that the Mesh Extenders make mesh telephony much more practical, by making it easy to connect relatively distant locations without aiming antennae, or needing lots of hops.

Range Testing Serval Mesh Extender on the National Mall in Washington with the New America Foundation

Please take a look at our crowd-funding campaign at igg.me/at/speakfreely.

The following is a joint blog post with the New America Foundation's OTI/Commotion Wireless project:

On an uncharacteristically chilly May morning, members of the Commotion and Serval projects set out for the National Mall in Washington, DC to test Serval's latest piece of hardware: the Mesh Extender. Commotion Wireless is an open-source toolkit of software, documentation, and training materials that strengthens communities by allowing them to build their own local communication infrastructure. Serval is a mesh networking software designed to act as an ad hoc communications network where other infrastructure is either absent or unavailable -- such as in remote areas or disaster scenarios.

Some of us stood at the Washington Monument while others were at the Lincoln Memorial -- a distance of nearly a mile. Using two Mesh Extenders, we successfully sent text messages and shared files between our phones (running Serval) -- entirely independent of the cellular infrastructure.

On the way back to the office, we hopped on the Metro -- DC's subway system -- to run another impromptu field test. From opposite ends of the train, we were able to send and receive messages through six subway cars (and their passengers) while the train was moving. That meant we could do something Metro riders usually can't -- send and receive messages while in the subway tunnels.

These results represent a significant breakthrough, since until now Serval and Commotion have been limited by the relatively short range and low power of Wi-Fi. In addition to increasing range and power, the Mesh Extender  removes a major obstacle to widespread adoption of mesh for mobile phones: rooting. Normally a prerequisite for Androids to connect to a mesh network, this technically challenging process for installing a new operating system can cause problems down the road for the rooted phone. In this case, the Mesh Extender routes the messages, not the users' phones, eliminating the need for rooting.

However, the Mesh Extender is still a prototype. Software issues make voice calls possible but indecipherable. Further development, including better error correction and noise cancellation, will allow for not only voice calls but potentially even longer-distance connections. Despite these remaining challenges, the Mesh Extender is a huge step towards reliable decentralized infrastructure.

A device with multiple radios and a small processor, the Mesh Extender essentially acts as a relay between phones running Serval software. It is lightweight, portable, and relatively cheap and easy to build. The parts can be purchased and assembled for as little as $99. Impressively, the battery can support three to five days of continuous use. The Mesh Extender uses omni-directional antennas (as opposed to point-to-point links), which make for much easier set up and
configuration, and allow for truly mobile networks.

The Mesh Extender operates simultaneously on the 2.4 GHz and 900 MHz bands, both of which are unlicensed. This allows phones running Serval to tether to the closest Mesh Extender over Wi-Fi (2.4 GHz), while the Mesh Extenders themselves communicate over the 900 MHz band, which is both less congested and has better propagation characteristics.

Range Testing Mesh Extenders in Boston & at the MIT Media Lab

While here in Boston with the Shuttleworth Foundation, I decided to take the opportunity to see what kind of coverage we can obtain with a Mesh Extender in my hotel room.  

My room is on the 7th floor of the Kendall Hotel in Cambridge, which provides a good vantage point in amongst buildings that are often 10 - 20 floors.  I put the Mesh Extender on top of the wardrobe in my room, and set about walking around the local area to see how far the signal would reach.

You can get an idea of the coverage in the map below, generated using the coverage mapping tool written by Loïc, one of our students.  This tool reads the signal strength directly from Mesh Extender via the ServalD HTTP interface, and merging it with a GPS fix from the phone.  The resulting trace is processed with some scripts to produce an HTML page that overlays the data points on Google Maps.

We expect to release the mapping tool in the next few weeks. 

You can see that coverage in this case extends one to two blocks. Not surprisingly, range is shortest when faced with the tallest buildings, e.g., through the Boston Marriot Hotel.  Nonetheless, the signal almost always appears to penetrate through an entire building, and is usable on the other side.  

Basically, whereas Wi-Fi can punch through about one wall before fading out, the UHF packet radio link can punch through something like 10 walls before fading out.

Points with confirmed coverage back to my hotel room. Actual coverage is somewhat better, but the collection tool needs tweaking to show spots with good link below 10dB.

The keen eye will notice that this area is right next to MIT, and indeed we have coverage right around the MIT Media Lab where we are having our Shuttleworth Foundation meetings.  So one morning I decided to see if we could establish a mesh between my hotel room and the room we were in the Media Lab building.  

That room faces the away from the hotel, so the signal would have to punch through the Media Lab building, as well as the MIT Health Services building which is in between the lab and the hotel.  In the image below the room in the media lab where we were meeting is roughly indicated by the arrow near the bottom of the image.  

As the link was from the top floor of one building to the top floor of another building it is likely that some signal followed a free space path outside of the buildings, but there is certainly no line-of-sight path.

The link frustratingly dropped out once we were inside the building with the Mesh Extender. However, we did find that by placing the Mesh Extender out on the deck outside that we had a steady link back to the hotel room.  So while we couldn't have the Mesh Extender inside the room, it was possible to have a mesh phone inside the room, talking to the Mesh Extender outside on the deck, and from there back to the hotel. In the image below you can see that the phone could see both Mesh Extenders, as well as the other phone in my pocket.  The Mesh Extender is visible out on the deck in the background.

In the MIT Media Lab with link back to the hotel via the Mesh Extender on the balcony.

So while it would have been nice if link could be sustained inside the building, it was still pretty easy to establish a link between two buildings a couple of hundred metres apart in a built-up urban area, with one of the Mesh Extenders indoors.

Tools for mapping Mesh Extender coverage

In previous tests of the Mesh Extenders we were manually noting the signal strength at various locations, and then drawing maps by hand to get an idea of the coverage.

Thanks to the great work by one of our students, Loïc, we now have a nice tool that lets us collect GPS traces annotated with Mesh Extender signal strength.

Now we can collect continuous traces of signal strength very easily.  Here is one where Loïc put one Mesh Extender on a hills hoist in a suburban back-yard, and set about riding around the neighbourhood with the other Mesh Extender in his backpack to see how it performed:

The ability to quickly deploy Mesh Extenders without great care or aiming is well illustrated in this situation.

A map of the signal strength is visible below.  Red and brown means no link, while orange through blue is progressively stronger signal.

As in previous tests, we see coverage of a couple of hundred metres in typical suburban conditions, reaching to a radius of around ten houses, and thus covering approximately 25x - 100x as many houses as Wi-Fi.

DiskWarrior Error (-36 2747) "hardware failure" unless you wait for hours: solution

I have a DROBO which I am trying to recover with DiskWarrior.

When I connect the DROBO and try to run DiskWarrior, I get an error -36 "hardware failure".  

I can hear the DROBO doing "stuff".

If I wait several hours until the DROBO is quiet, then DiskWarrior doesn't get the error, and I can try to recover it.

Scratched my head over this for a while, so figured I would post the solution in case anyone has the same problem.

All the forum posts are about dead disks.  

My disk isn't dead, it just has a sick filesystem, as proven by the fact that if I wait long enough, DiskWarrior can try to do something with it.

The disk activity was a clue.  

I suspected that TimeMachine or something else on the mac is trying to fsck the file system in preparation for mounting.  Unfortunately that takes HOURS on this large volume full of time machine backups.

So opened Terminal and typed: ps -ef | grep fsck_hfs and there was the culprit:

    0 67332    18   0  6:26am ??         1:02.06 /System/Library/Filesystems/hfs.fs/Contents/Resources/../../../../../../sbin/fsck_hfs -y /dev/disk5s2

To kill it, type kill followed by the second number from the left, in this case: kill 67332

The disk activity stops, and you can then run DiskWarrior.

(If you find this post helpful, please consider making a donation to servalproject.org so that we can help people communicate during disasters). 

Multi-Hop MeshMS (Mesh SMS) and Ringing Phones via Mesh Extender

Today we did some more testing with the prototype Mesh Extenders, and finally got around to filming the Mesh Extenders actually relaying mesh data, rather than just reporting link quality figures.

As a side effect, we also ended up testing some of Jeremy's new mesh routing code.

The topology we used was two un-rooted Android phones and two Mesh Extenders. Each phone was connected to one of the Mesh Extenders as a Wi-Fi client.  The Mesh Extenders used their Wi-Fi and UHF packet radios to connect to each other.  Thus there were three hops from one phone to the other, as shown in the peer list screen grab below:

Jeremy added the nice routing information to the peer list recently, so that you can see the number of hops to get to a peer, including the route taken.  This makes it much easier for us to debug multi-hop paths.

We then put one of the Mesh Extenders on a bench by a window in the lab, and left Jeremy next to it with one of the phones:

I took the other Mesh Extender walk-about and in the RV Bakfiets to see how far we could go, sending and receiving text messages and making the phones ring at each location.  An actual voice call over the limited bandwidth of the Mesh Extenders will not be possible until we add support for CODEC2 into batphone. What was encouraging was we did hear pops of audio when we placed calls, indicating that it was only the bandwidth starvation that was stopping us from having voice calls.

The idea was to test reachability, rather than probe the exact limits of range.  We are planning a little phone app that will log signal strength and GPS location to help map the limits of range more exactly, but that will be another day. A couple of locations were of the menu today:

First, in the bush behind the University.  There the challenges are vegetation and undulating land.  The near edge of the bush land where I went was visible through the lab window where the Mesh Extender was sitting, with the building not getting in the way.  Here I am in the bush with the Mest Extender and phone after doing the test there:

 Here is a map showing approximately where we were.  The lab is the green spot, and we were about where the red spot is. We had about +13dB link margin, which is ample.

Here you can see me send and receive text messages, asking Jeremy to call my phone, which he does:

Total distance was about 315m, which was pretty nice, given all the trees in the way, and the there are some low rises in the path as well.  Also, the car park was full of cars, so the path wasn't as clear as the satellite image suggests.

Then later in the afternoon I needed to head down to the plaza in the central part of the campus.  Discussion in the lab was not very positive about getting to the plaza in one hop.

I rode my bakfiets down, with the Mesh Extender in the box. The first view is roughly in the direction of the lab -- through a pile of trees, and up the hill, and through the entire Engineering building -- there was no good line of sight for this one, just a strong elevated position.

 This is the view across the plaza, with a number of people enjoying the unseasonably warm weather this week (28c just three weeks out from winter):

Here I call Jeremy's phone and receive a message from Jeremy while on the plaza:

The distance was about 415m, with +12dB link margin.  Riding around the plaza the signal was fine unless I ventured too far East (right on the map) towards the buildings, in which case the obstructions were just too much. 

As with the other recent tests of the Mesh Extenders, we continue to see that the distance between each hop can be something like 10x that of Wi-Fi, and hence cover 100x the area, whether in urban areas, open country, vegetated areas or otherwise.  The following list of posts is a summary of some of the tests we have performed so far, all lending support to this premise:

http://servalpaul.blogspot.com/2013/05/urban-testing-of-mesh-extender-part-1.html
http://servalpaul.blogspot.com/2013/05/urban-testing-of-mesh-extender-part-2.html
http://servalpaul.blogspot.com/2013/03/testing-serval-mesh-helper-prototypes.html
http://servalpaul.blogspot.com/2013/02/3km-mesh-link-using-mesh-helper.html
http://servalpaul.blogspot.com/2013/02/mesh-link-using-our-prototype-mesh.html

This is quite important, because the human voice has roughly the same propagation characteristics as Wi-Fi, that is, you can typically shout at someone if they are in Wi-Fi range, which means that prior to the Mesh Extender, single-hop mesh communications were basically no further than you could shout at someone.

But now are consistently facilitating communications an order of magnitude further, to a distance that is more like that of a hand-held CB radio in a single hop.  But unlike a CB radio, we have rich digital services, and once we complete the firmware rewrite we will be able to carry services over many Mesh Extender hops, thus also eclipsing the range of ordinary CB radio.

So perhaps we should be marketing Serval as something like "Digital CB" rather than just as "mobile mesh telephony"...

Urban testing of Mesh Extender - Part 2

Following the previous test where we placed a mesh extender on a bench in my house, we decided to test again with the mesh extender located in a better, higher location to simulate a purposeful installation.

The first step was to mod our existing prototypes to have the antennae sticking out of the top of their tubs. This reflects our view that the antenna need to stick out to obtain the best performance.  They can be seen here ready to be tested:

One was then promptly installed on our roof:

One thing you will notice is that we have lots of trees in our area that are higher than where we put the Mesh Extender.  So we would continue to suffer from a lack of line-of-sight. Nonetheless, we were confident that we would get substantially better range than in the previous test, but only had a short time window to test in.  This was a bit of an issue, because we would need to cover a few kilo-metres of roads in about twenty minutes.

We thought about driving around, and getting out periodically to measure signal strength, but that didn't sound like fun, and wouldn't give a good idea of real "on the street" coverage, since cars are pretty good shields.  So while it is a situation we should examine, it wasn't the goal of today's test.

Fortunately, I own a Dutch Bakfiets (cargo bike), and one of our developers, was willing to sit in the bike while we rode around.  This worked really well, as we could easily cruise around at about 10km/hour - 15km/hour, and Andrew could read signal strengths off as we went around. You can see him in the bike here, holding his phone to view the signal information, and with the Mesh Extender sitting between his feet.

So how well did it work?

Well, first up, I tested performance around at the local super-market, which we could almost but not quite reach with the Mesh Extender inside my house.  But this time it was possible to get signal about 10m or so into the super-market building.  Here is the Mesh Extender in the fruit-and-veg department showing a solid link of about 10dB:

Then it was time to ride around the neighbourhood and see roughly how far we could get.  Last time we had reliable links to around 200m, and some links to about 260m.  This time we had reliable links to around 500m, spanning about three blocks:

Several hundred dwellings would be within range of this unit. Again, this is the per-hop range, and once we improve the radio firmware to mesh, it will be quite feasible to cover significant distances with multiple hops. 

Part of the rush in this test was that I had to collect Caleb from Childcare. I left the Extender running on the roof, and took the other with me, and came back a different route to see whether we could pick up any signal along Marion Road (the main road running top to bottom in the image).  We did indeed pickup a link along portions of Marion Road at a distance of up to about 850m, including the point shown here:

Not entirely surprisingly the link could be picked up on the further side of Marion Road, but not the nearer side, presumably due to RF shadowing.  

The Mesh Extender in the Bakfiets was less than 1m off the ground, and raising it higher would most likely have improved things, and suggests that distances of a few kilometres might be possible roof-to-roof with good line of sight.

So overall, the range was at least 2.5x better than with the indoor placement.  This is not surprising. It was, however, a pleasant surprise to see urban distances of almost 1km possible -- despite not using advanced error correction and detection schemes in the radio. 

Such a roof-top deployment has the potential to provide coverage over hundreds of dwellings, and it is easy to envisage even a relatively modest mesh of perhaps a dozen or so nodes being able cover an entire suburb, and allow people to make use of mesh services by carrying a small Mesh Extender with them to connect.

Urban testing of Mesh Extender - Part 1

Yesterday we finally got around to performing an urban test of the Mesh Extender prototypes.

To make this a realistic test, we have one Mesh Extender just sitting inside my house on a cupboard.  There was no antenna aiming. It was not up high. It was inside a double-brick house, placed no better than you could expect an uninformed user to place it.

We know that in similar circumstances that Wi-Fi has a range of about one house, i.e., you can often see your immediate neighbours Wi-Fi, but not any further.

We wanted to see how much further we could cover with the Mesh Extender's 915MHz packet radio.

The first part of the test consisted of walking around the block where we live.  This block has 11 houses in a row along its length, with us on one end (the green marker on the map).  From where we placed the Extender to the far corner on the footpath is about 192 metres as the photon flies. Transmit power was +24dBm (250mW) and air bit rate was 128kbit.

We were able to maintain link all the way around the block, without it dropping even once.  This is very encouraging, as it suggests that anyone on our block would be able to put a Mesh Extender in their house, and we would have digital communications between them. Again, contrast this with Wi-Fi where if you are lucky you can see your next-door neighbours Wi-Fi.

Encouraged by the success of this, we decided to see if we could get a link from my house to the local super-market.  The Mesh Extender remained where it was, and we walked around to the super-market. The idea was to push as far as we could until link was lost.

We took the route along Minchinbury Terrace, and then across to the bottom-right corner of the shopping complex, and then walked towards the red mark on the map.  The link became intermittent once we were on Chambers Street, and was only reestablished intermittently at the shopping centre.  The maximum distance where we had link, in the foyer of Coles, was about 262m.  

Remember that these units were only metre or so off the ground, and one was inside a double-walled brick house, and that there were a number of similarly constructed dwellings and large trees in the way. This is really quite a hostile environment, and is certainly not unreasonably optimistic compared with expected real-world deployment in urban areas.

As mentioned above, Wi-Fi range of about 1 house means that forming a suburban Wi-Fi mesh requires almost 100% of homes to participate.  In contrast, if we draw a circle around the 192m range where we had solid link, and the 192m-262m range where a link might be possible, the story is very different:

The solid range area includes about 90 separate homes, and that ignores that a few of those are blocks of units.  This means that an adoption rate of 2/90 = about 2% would be sufficient to almost guarantee the formation of a continuous mesh, with people just placing the Mesh Extenders in a convenient location inside their home.

If we assume that half the homes in the 192m - 262m range can establish a link, then that adds another 25 houses (not even counting the ones that didn't fit in the image).  In fact, in most realistic urban or suburban deployments then dwellings are likely to be packed more closely.

But most importantly, if even a few Mesh Extenders are placed in better vantage points, perhaps on a roof, or even on the top of a book case, then it is possible that the range will be extended even further. Radio range is typically proportional to the square root of the height of the transmitter above ground level, so doubling the height of the radio from ~1m to ~2m could increase the range by about 71%, which would double the number of houses in range. Double the height at both ends, and the effect is combined, and the range would be doubled.

Placing units 4m or so off the ground, e.g., on the roof of a house would likely extend the range much more, as in addition to the 2x range due to increased height, the obstruction of the house and other objects near ground level would be greatly reduced, and distances in the kilo-metres should be achievable in favourable conditions.  It seems quite feasible that well placed units could reach potentially thousands to tens of thousands of other homes.  This is something that we may try to test in the next few days.

This is the kind of capability that makes mobile mesh telephony able to cover very large areas, and connect whole communities, and without requiring wide-spread adoption to begin forming the network.

And I again emphasise that this is without any aiming or special skills on the part of the people "installing" the Mesh Extenders.

To achieve this we have some hardware design work to do, as well as some interesting work on improving the packet radio firmware.  If anyone would like to pitch in and help, we'd love to hear about it.