Detecting Taps on Microsoft Band with RX and C#

This is a follow on to my original post on using RX with the Band to stream sensor information.

Using the RX streams and some maths to detect (roughly) when a user taps on the band. This allows you to send a notification to the band, for example, saying “I think your home, tap band to turn on your heating”.

Below is the result, I can really easily “await” tap or shake events! You can grab the code here and see how it all works below.

var stream = await band.GetShakeOrTapStream();

await stream.FirstAsync().ToTask();

So I started off by grabbed all the accelerometer data as CSV through the output window while tapping on the band.

accelSub = accStream

.Where(x=>x.SensorReading.AccelerationX != 0)

.Subscribe(x=>{

Debug.WriteLine("{0},{1},{2}", Math.Round(x.SensorReading.AccelerationX, rounding), Math.Round(x.SensorReading.AccelerationY, rounding), Math.Round(x.SensorReading.AccelerationZ, rounding));

});

Naively I thought the taps would be obvious to the human eye, they weren’t!

-0.093994,-0.691406,0.763672 -0.100098,-0.691895,0.760254 -0.099365,-0.677002,0.756348 -0.101807,-0.653564,0.746094 -0.105957,-0.640381,0.731934 -0.114014,-0.647949,0.733154 -0.106445,-0.656982,0.730225 -0.078857,-0.67627,0.730713 -0.063232,-0.686768,0.743652 -0.055176,-0.682129,0.765381 -0.040527,-0.676025,0.760986 -0.01709,-0.66626,0.751709 -0.001709,-0.64624,0.735352

So this is where I switched over to Excel to experiment with the data. The graph shed some light on what was going on.

With this basic data you could see changes in the accelerometer data on all three Axis when a tap took place, looked good to me.

So what about normal motion follow by a tap, what does that look like? In the above I’m standing still and tapping the band but that’s not realistic for a user. So I simulated some normal motion, like walking and drinking water, then had a go at tapping the band. (I also did one hard tap while moving around).

What you can see is that during normal motion there are a range of changes on all axis, they’re smoother and not strongly correlated. What I can see during taps is a high change on at least 2 axis.

To make these multi axis peaks easier to spot I summed all the motion on each axis and subtracted this value from the previous reading, to show the aggregate change between readings.

This appeared to clearly show the peaks where the taps occurred and successfully differentiate them from normal motion.

With the data in hand I started to make it into an algorithm which could run over the RX stream.

The scan operation gives me the ability to take input from the current and last event receive, it’s usually used to create aggregate values on the fly, however, I’m using it to compare the two readings and output the change between them. This gives me a RX stream with the output of the graph above.

Where the difference between the current and last reading was large I’d output that a tap event had occurred.

accelSub = accStream

    .Where(x=>x.SensorReading.AccelerationX != 0)

    .Select(x =>

        double[] array = { x.SensorReading.AccelerationX, x.SensorReading.AccelerationY, x.SensorReading.AccelerationZ };

        //We're looking for taps so could come from any axis. Aggregate the reading to see!

        return array;

    .Scan<double[], Tuple<double[], double>>(null, (last, current) =>

            if (last == null)

                return new Tuple<double[], double>(current, 0);

            var variation = (last.Item1[0] - current[0]) + (last.Item1[1] - current[1]) + (last.Item1[1] - current[1]);

            return new Tuple<double[], double>(current, variation);

})

    .Where(x=>x.Item2 > 4)

    .Subscribe(x =>

        Debug.WriteLine("Tap Detected" + x.Item1);

});

So this approach did the job, when stationary, tapping created a great result. Below you can see 5 distinct taps.

The problem was that, when moving around, we get some large aggregate motion readings with big changes between them due to sustained motion, like swinging your arm.

I need to look for short, sharp spikes for these to be taps or shakes. Rather than large gradual changes.

At this point I had an experiment with implementing a high pass filter over the data to detect the taps and shakes, unfortunately it didn’t work well during my tests. This is something I hope to come back to at some point.

Adding in a time based buffer for 400ms and then calculating the variance of the items capture within the buffer gave me a much cleaner reading. Row 27 shows a tap, the other peaks are me picking up water bottles, opening doors and generally trying to simulate normal movement.

So with that added in we end up with this:

	using Microsoft.Band;
	using Microsoft.Band.Sensors;
	using Microsoft.Band.Tiles;
	using System;
	using System.Collections.Generic;
	using System.Linq;
	using System.Reactive.Linq;
	using System.Text;
	using System.Threading.Tasks;
	using Windows.UI.Popups;

	namespace TestBand
	{
	class BandAccessWrapper
	{
	public IBandClient Client;
	private string TileName;
	public BandAccessWrapper(string appName)
	{
	TileName = appName;
	}
	public async Task Connect()
	{
	try
	{
	// This method will throw an exception upon failure for a veriety of reasons,
	// such as Band out of range or turned off.
	var bands = await BandClientManager.Instance.GetBandsAsync();
	Client = await BandClientManager.Instance.ConnectAsync(bands[0]);

	}
	catch (Exception ex)
	{
	var t = new MessageDialog(ex.Message, "Failed to Connect").ShowAsync();
	}
	}

	public async Task<BandTile> AddTile(BandTile tile, bool overwrite)
	{
	IEnumerable<BandTile> tiles = await GetTiles();

	if (tiles.Any(x => x.Name == tile.Name))
	{
	var currentTile = tiles.Where(x => x.Name == tile.Name).Single();
	if (!overwrite)
	{
	return currentTile;
	}
	await Client.TileManager.RemoveTileAsync(currentTile.TileId);
	}

	await Client.TileManager.AddTileAsync(tile);

	tiles = await GetTiles();

	tile = tiles.Where(x => x.Name == TileName).Single();

	return tile;
	}

	public async Task<IObservable<BandSensorReadingEventArgs<IBandHeartRateReading>>> GetHeartRateStream()
	{
	return await GetSensorStream<IBandHeartRateReading>(Client.SensorManager.HeartRate);
	}

	public async Task<IObservable<BandSensorReadingEventArgs<IBandAccelerometerReading>>> GetAccelerometerStream()
	{
	return await GetSensorStream<IBandAccelerometerReading>(Client.SensorManager.Accelerometer);
	}

	public async Task<IObservable<TapEvent>> GetShakeOrTapStream(double threshold = 4, int timeSampleMs = 300)
	{
	var accStream = await GetAccelerometerStream();
	return accStream
	//Filter out any empty readings, band seems to report every other reading as all 0's
	.Where(x => x.SensorReading.AccelerationX != 0)
	//Get our readings, don't need the event args
	.Select(x => x.SensorReading )
	//Scan over the readings creating an aggregate reading for motion on all axis.
	//Output all axis and the aggregate motion
	.Scan<IBandAccelerometerReading, ChangeInMotion>(null, (last, current) =>
	{
	//If we're the first the change is 0
	if (last == null)
	{
	return new ChangeInMotion(0, current);
	}

	//Get difference in motion on all axis vs last reading as positive # then sum to get aggregate change in motion.
	var aggregateChangeInMotion = (last.Reading.AccelerationX - current.AccelerationX) * -1 + (last.Reading.AccelerationY - current.AccelerationY) * -1 + (last.Reading.AccelerationZ - current.AccelerationZ) * -1;

	return new ChangeInMotion(aggregateChangeInMotion, current);
	})
	//Collect a set of results over a timespan, around 400ms worked for me to detect taps and shakes
	.Buffer(new TimeSpan(0, 0, 0, 0, timeSampleMs))
	//Caculate the variance of the aggregate motion reading
	.Select(x =>
	{
	var listOfAggregateMotion = x.Select(y => y.ChangeVsLastReading);
	return new TapEvent(Variance(listOfAggregateMotion), x.Select(y=>y.Reading));
	})
	.Where(x => x.Variance > threshold);
	}
	private double Variance(IEnumerable<double> nums)
	{
	if (nums.Count() > 1)
	{
	// Get the average of the values
	double avg = nums.Average();

	// Now figure out how far each point is from the mean
	// So we subtract from the number the average
	// Then raise it to the power of 2
	double sumOfSquares = 0.0;

	foreach (int num in nums)
	{
	sumOfSquares += Math.Pow((num - avg), 2.0);
	}

	// Finally divide it by n - 1 (for standard deviation variance)
	// Or use length without subtracting one ( for population standard deviation variance)
	return sumOfSquares / (double)(nums.Count() - 1);
	}
	else { return 0.0; }
	}

	public async Task<IObservable<BandSensorReadingEventArgs<T>>> GetSensorStream<T>(IBandSensor<T> manager) where T : IBandSensorReading
	{
	var consent = manager.GetCurrentUserConsent();
	if (consent != UserConsent.Granted)
	{
	await manager.RequestUserConsentAsync();
	}

	var supportedIntervals = manager.SupportedReportingIntervals;
	manager.ReportingInterval = supportedIntervals.First();

	var stream = CreateObservableFromSensorEvent<T>(manager);

	return stream;
	}



	private IObservable<BandSensorReadingEventArgs<T>> CreateObservableFromSensorEvent<T>(IBandSensor<T> manager) where T : IBandSensorReading
	{

	var obs = Observable.FromEvent<
	EventHandler<BandSensorReadingEventArgs<T>>,
	BandSensorReadingEventArgs<T>>
	(
	handler =>
	{
	EventHandler<BandSensorReadingEventArgs<T>> kpeHandler = (sender, e) => handler(e);
	return kpeHandler;
	},
	async x =>
	{
	manager.ReadingChanged += x;
	await manager.StartReadingsAsync();
	},
	async x =>
	{
	manager.ReadingChanged -= x;
	await manager.StopReadingsAsync();
	}
	);
	return obs;
	}

	private async Task<IEnumerable<BandTile>> GetTiles()
	{
	IEnumerable<BandTile> tiles = await Client.TileManager.GetTilesAsync();
	return tiles;
	}

	public class ChangeInMotion
	{
	public ChangeInMotion(double ChangeVsLastReading, IBandAccelerometerReading Reading)
	{
	this.ChangeVsLastReading = ChangeVsLastReading;
	this.Reading = Reading;
	}
	public double ChangeVsLastReading { get; set; }
	public IBandAccelerometerReading Reading { get; set; }
	}

	public class TapEvent
	{
	public TapEvent(double Variance, IEnumerable<IBandAccelerometerReading> Readings)
	{
	this.Variance = Variance;
	this.Readings = Readings;
	}
	public double Variance { get; set; }
	public IEnumerable<IBandAccelerometerReading> Readings { get; set; }
	}
	}
	}

view raw BandAccessWrapper.cs hosted with ❤ by GitHub