Stephen Visser

Tech, Life, Etc
Sep 16, 2013

Setting up your own push notification server for Windows 8 can be a challenge. Novices can easily become overwhelmed trying to work within a paradigm they aren't familiar with. Many developers would rather use Azure's platform than set up and manage their own solution.

Azure's comprehensive suite of services and compatibility with other mobile vendor's is appealing. It is also recognized for being easy to configure, powerful, highly available, and performant. There is little incentive for many organizations to choose to put their own system in place.

However, once developers are familiar with how they work, push notifications aren't so intimidating. Companies can avoid paying expensive cloud fees by rolling their own in a matter of hours. The following is a high-level discussion of how this can be done.

image

In the above picture, the purple Windows box represents the client. This could be your Surface Pro or Nokia 920. The Notification Client Platform is MS-speak for the API that Windows exposes to you for initiating the notification process. The green Cloud Service box is your server. WNS is a Microsoft-hosted service that keeps track of all Windows clients.

To understand why the process is convoluted, you need to first understand what problem push notifications are trying to solve. Up-to-date apps are only possible by either polling servers for changes at high frequencies or through server-initiated messaging. In order for a server to initiate a connection over traditional networks (HTTP / TCP / IP), there needs to be a persistent, long-term HTTP socket. All mobile platforms have a WNS equivalent: a server that manages a single long connection with a device in order to initiate message-sending instantaneously. Any app on the device can use this same communication channel since it is shared and communication is multiplexed to the correct recipient on the client.

Windows Push Notification Service (WNS): https://login.live.com

This is Microsoft's bridge that brokers connections between your server and any one of your clients. Clients initiate 'channels' with the WNS which are referenced in the form of URI's. The server authenticates and connects to the WNS in order to facilitate communication with any of the clients.

Server

This MSDN article describes the process of sending a push notification in detail. Before transmitting any information, it is necessary to authenticate. To do this, the server needs to request and receive an access token. This access token will be used to identify the server when sending specific messages.

Using parameters that define your app (client_id and client_secret), you send an HTTPS request to https://login.live.com/accesstoken.srf. The response from this request includes the necessary credentials (access_token).

After this access token has been received, the server can begin sending messages to clients using their Channel URI's.

The most complex part of this whole process is managing the Channel URI's for every client. Each client must know to save (or update) their own URI on the server every time it changes. Thus, the server typically has some sort of persistent store (SQL Server) that keeps track of an identifying characteristic of the client and it's corresponding URI.

Once the server needs to make a push notification, it can perform a lookup in its database and send an HTTPS POST to the URI with a bunch of parameters set. There are many HTTP header values that should be set, but the most important is the access_token that was negotiated earlier.

The body of this POST request contains the XML message according to the tiles, toast or badge schemas documentation.

Your server should also be set up to support SSL since each client will inform the server of its URI. This URI should be treated as a secret so that attackers can't send rogue notifications.

Client

Every time the app is opened, it needs to send the client's channel URI to ensure the server can reference it. This is similar to an unlisted phone number. The only way to make the call is to have the recipient give you their phone number. This is done using the Notification Client Platform API, specifically, the CreatePushNotificationChannelForApplicationAsync method.

As mentioned before, the URI that results from this call should be sent to your server so it can dial up the client.

From there, each client needs to just wait until a message comes down the pipe. This is done using the PushNotificationChannel class. You can find more details about how to do this online.

Conclusion

I went over how push notifications work in detail. You should understand why sending and receiving notifications can get complicated. Hopefully this helps you get most of the way there. And if not, thankfully there's Azure.

Mar 4, 2013

A Weekend of PI:

Last weekend I participated in a Hackathon at Startup Edmonton. I wanted to play around with the Raspberry PI which I had purchased a couple months ago, but hadn't yet gotten around to exploring.

The goal was to hook the PI up to my Garage Door. My idea was to run a web server on the PI, opening the garage door from an app on my local network.

Garage Doors

I did some preliminary research to find out how a Garage Door operates. Turns out its pretty simple:

The buttons on the inside of your garage just complete a sensing circuit. Depending on which button is pressed (Light, Open, Lock), a different resistance is applied to the circuit. Controlling the opening and closing of the door is the simplest scenario: no resistance at all. I used an Allen wrench to test my theory; on the motor unit, I shorted the two poles connected to the twisted copper wire from the button. The garage began opening. After shorting them again, opening stopped. Shorting a third time caused the garage door to move in the opposite direction, closing.

There is a third pole on the motor unit which is used for protecting the door from closing on top of you. As far as I know, this acts in very much the same way. One of the infrared light sensors will change the voltage characteristic of the control circuit when the line-of-sight is broken. I never got around to incorporating this protection circuit, but it's there for the doing.

After the research, all I needed was a electrical way of shorting the aforementioned circuit.

PiFace majik

As I mentioned in another blog post, the PiFace is a powerful interface to the real world. After snapping the PiFace board on top of the Pi, we can use one of its relays to control the circuit.

Start by making sure your Pi has the Python libraries installed and is configured to run the SPI interface required by the PiFace. Instructions can be found here

Web

After installation, we'll write a small script which runs a python server for hosting our open/close code. Copy the following code to the pi:

#!/usr/bin/python

import os
import sys
import time
import piface.pfio
import SimpleHTTPServer

try:
    path = sys.argv[1]
except IndexError:
    path = '.'

os.chdir(path)

piface.pfio.init()
relay = piface.pfio.Relay(1)

class PiServer(SimpleHTTPServer.SimpleHTTPRequestHandler):
  def do_POST(self):
    relay.turn_on()
    time.sleep(0.5)
    relay.turn_off()
    self.send_response(200)

httpd = SimpleHTTPServer.BaseHTTPServer.HTTPServer(('', 80), PiServer)

print 'Started server. Hosting contents of "{cwd}"'.format(cwd=os.getcwd())
httpd.serve_forever()

... and run it:

nohup sudo ./server.py &

Then on any computer computer, run curl to toggle relay for half a second:

curl -d "" http://raspberrypi.local

One side-effect of running the code is that the contents of the current directory will also be hosted. This may be very insecure if you run in your home directory. What we probably want to do instead is give it an index.html to use:

mkdir web
cd web
vi index.html

Edit the index.html to your satisfaction then run the server hosting just this file.

nohup sudo ./server.py web &

Here's an extremely simple page that opens and closes the door: (Obviously optimized for webkit devices)

<!DOCTYPE html>
<html>
    <head>
        <title>Open Sesame</title>
        <meta name="viewport" content="width=device-width">
        <link href="bootstrap.min.css" rel="stylesheet">
    </head>
    <body style="height:320px;display:-webkit-box;-webkit-box-pack:center;-webkit-box-align:center;">
        <button class="btn btn-large btn-primary" onclick="$.post('/');">TOGGLE</button>
        <script src="//ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>
    </body>
</html>

Best of luck!

Mar 3, 2013

In a recent project, I worked with the PiFace shield for the Raspberry Pi. I highly recommend getting one of these if you have a wide variety of projects for your Pi, but don't know where to start.

I learned a lot of this stuff in University, but since then haven't seen anything hardware-related. Throughout this project I was continually reminding myself what a collector was, how serial circuit timing worked, and event simple circuit sight-reading. It was fun, but a lot of work. I thought I would detail some of the specifics so that others can learn about it too!

PiFace

The PiFace is a PCB with some output and input rails, two relays, and some on-board LEDs and pushbuttons. Here's the data sheet, and a simpler version.

To make this all work, the PiFace does not just extend the existing GPIO pins; it acts as an SPI slave, creating 8 dedicated outputs and 8 dedicated inputs through the simple Pi Serial pins. The PiFace also has the potential to be expanded with several other PiFace boards for more complicated projects through its addressing scheme.

Make Pi SPI-friendly

The Raspberry Pi does not have the SPI interface enabled by default; to enable, run the following command:

sudo sed -i -r 's/^(blacklist spi-bcm2708)/#\1/' /etc/modprobe.d/raspi-blacklist.conf

To set up the pi for SPI programming in Python, run the following script:

wget -O - http://pi.cs.man.ac.uk/download/install.txt | bash

Reboot!

sudo reboot

The manual instructions for the above process can be found here.

Note: When connecting the PiFace, make sure the PI is turned off so nothing surprising happens. When I rewired jumpers on the PiFace while the PI was on the voltage spike caused the PI to reboot.

SPI Interfaces

The Pi Face documentation doesn't describe what's happening on a physical level, so here's a (semi-)simple breakdown:

A MCP23017 SPI → GPIO chip connects to the serial pins of the PI. Using the PI's SPI capabilities, we can read and write to these GPIO pins. The chip has 16 GPIO inputs; the Pi Face uses one bank of 8 pins for input, and the other 8 pins for output.

SPI is a simple communications protocol that transmits information one bit at a time. I dug into what's going on by looking at Pi Face's python source code and the 23017 data sheet:

Basically, the 23017 behaves like an addressed array of 22 1-byte (8 bit) registers. A register can be thought of as a mailbox; unless it is explicitly changed, it keeps its state until the chip is reset. Each of these register mailboxes either configures the chip to behave in a specific mode, or allows us to read or write to the GPIO pins.

Each time we send a message to the 23017 we need to follow a special protocol. Each message starts with an OPCODE byte. The first four bits will always be 0,1,0,0 in binary, an arbitrary code that the 23017 uses to kick off the communication. The next 3 bits in the OPCODE are an address for the board that's being used. Think of these bits as being aphone number for the target chip called a 'hardware address'. This is used to augment the SPI protocol to allow for more devices on the bus. Using this scheme, we can have up to 8 devices on the bus without requiring more wiring.

We tell the Pi Face what it's phone number is by changing jumper pins JP1 & JP2. The default is where both jumpers are set to the 0 connection; this means our Pi Face is the only chip on the bus, and will be selected by default. In most simple cases, we can just use 0,0,0 to address our chip.

The last bit in the OPCODE byte is a 1 for reading and 0 for writing the contents of the target register.

An example OPCODE

01000001
││││││││
│││││││└─read/write
││││└┴┴─address bytes
└┴┴┴─default initialization

The next byte sent to the 23017 is the address of the register we are writing to. In every case except one, each 1-byte register is paired for a total of 16 bits; each bit of the register represents a control state for a single GPIO pin. One of the registers controls the 8 'A' pins, the other, the 8 'B' pins. The 23017 data sheet has all the options, but here are a few examples:

  • IODIR – I/O DIRECTION REGISTER: This is used to set the default direction of each of the GPIO pins (0 - 15): either input or output.
  • IOCON – I/O EXPANDER CONFIGURATION REGISTER: This is the general configuration register, and is the only non-paired register. Each bit is used to configure the chip as a whole, not an individual pin. For our use-case, the only bit of interest is the HAEN bit which is set by default when the PiFace software initializes the chip so that we can use hardware addressing.
  • GPPU – GPIO PULL-UP RESISTOR REGISTER: This enables or disables a pull-up resistor on inputs so we don't draw too much current.
  • GPIO – GENERAL PURPOSE I/O PORT REGISTER: Gets or sets the actual value of the GPIO pins (depending on whether they are input or output).

Each of these registers have an address to read or write to. After the OPCODE, the byte representing the desired address is sent. A data byte is sent third; the byte will be empty when reading. Here are two complete examples:

01000000 00000000 11111111
       │ ││││││││     └ 0xFF, direction to be 'in' for all pins 
       │ └┴┴┴┼┴┴┘            
       │     └0x00, Register for IO Direction for Port A
       └write


01000001 00010011 00000000
       │ ││││││││     └ 0x00 (read, data is irrelevant)
       │ └┴┴┴┼┴┴┘            
       │     └0x13, GPIO Results Port B
       └read

Darlington Transistors

There is another chip on the Pi Face, an array of Darlington Transistors: ULN2803A. The rationale for this chip is that it is more flexible in terms of possible current/voltage characteristics. The 23017 operates on a voltage between 1.8 and 5.5V and the GPIO pins can't drive more than 25 mA. These are quite limiting in the real world, so the 2803a gives us some additional flexibility.

Each of the Pin Outs on the screw-rails of the PiFace can be driven by a voltage of up to 50V and current of 500 mA because of the Darlington Transistors. Electrically, this is done by giving attaching each Darlington transistor pair emitter to ground, and exposing the collector to the output pin. When the base of the transistor is switched on, the darlington pair allows current from the output pin to be grounded through the array.

An alternative form of output pin has the chip itself driving the voltage, but there is a risk inherent in this type of setup since the attached circuit may draw too much current, voltage may spike or any number of other things could happen to damage the chip. The output voltage is also limited to a reasonable number, typically 3.3V or 5V. In the case of a Darlington array, we could drive a complicated, high voltage circuit with no issues; it would be very robust.

Relays

The relay control circuit is connected through jumpers 5 and 6 to the circuit, allowing us to open and close these relays using output pins 0 and 1. Relays can be purposed in much the same way as Darlington transistor pairs or MOSFETs, but also support AC circuits. Relays are nothing more than electrically-controlled mechanical switches. As a result, without complex current voltage characteristics inherent in semiconductors, relays tend to be more robust and less technically demanding for a hobbyist's purpose.

They work by driving current through the wire coil 'control' part of the circuit which creates a magnetic field. This magnetic field causes a pin to physically move in the relay, completing a circuit. When there is no current applied to the control circuit, the second circuit is closed by an internal spring.

In the case of the PiFace's relays, they are rated for up to 250V AC at 10A. As way of comparison, the max current that PI will drive is 16mA, but it will depend on other things (the default is 8mA, and min is 2mA). These relays up for nearly any task we can throw at them.

Conclusion

The PiFace chip works great as part of any hobby project. Check it out and happy building!

Further Reading:

Some other cool peripherals:

Mar 3, 2013

These are instructions for getting started with the Raspberry Pi on a Mac. Windows users may want to refer to a different set of steps.

My goal is to set up the Pi without attaching a monitor or keyboard; later distros of Raspbian run ssh in init.d, that is, on startup. Thus, it shouldn't be a requirement to use a monitor or TV at all, otherwise known as 'headless'.

Download latest image

Latest Raspian images will be available here.

Before opening/extracting zip, run

shasum /path/to/2013-pi-image.zip

Compare with the sum on the downloads page. Good? Let's get it on...

Detailed instructions here on how to get the card loaded.

One comment on the above instructions: If you are using a shell other than bash (ie fish in my case), you may have to expand the .img location; I couldn't use the ~ character in my if option.

Got your card magicked up? Pray continue.

Bootstraping is the new hotness

Once I attached my SD card to the Pi, I used an ethernet cable to connect the Pi to my local network. After firing up the Pi by connecting power and waiting a minute or so, we should be able to find it on the network.

Depending on your local network / router configuration, this may be easier for you than it was for me. One way to discover devices on your network is to use nmap. You may have to download it first.

We will use it to look for SSH clients that are open, printing out information to differentiate from other potential clients. Make sure you set the subnet to match the one you're currently on:

nmap -p 22 --open -A -T5 192.168.1.0/24

Once you've found the ip address for our lost little Pi, we want to set it up for wireless action.

Use the wpa_passphrase command to generate the PSK. Add the result to the /etc/wpa_supplicant/wpa_supplicant.conf file:

wpa_passphrase 'REAL_SSID' 'PUT_ACTUAL_PASSWORD_HERE'

The resulting file will look something like this:

trl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
network={
    ssid="REAL_SSID"
    psk=1234…abcd
}
network={
        key_mgmt=NONE
}

You won't need the commented-out plaintext password.

One last thing to do is set up the pi so it broadcasts its hostname using bonjour:

sudo apt-get update
sudo apt-get install libnss-mdns

If you purchased a supported wireless key, you shouldn't need to do much in the way of driver installation; it's just plug'n'play.

Just

sudo poweroff

When the Pi is powered off, remove the ethernet wire and attach the USB dongle.

Power up the Pi.

You should be able to access it using Bonjour.

ssh pi@raspberrypi.local

Adding sshweetness

If you're going to ssh to your pi quite often, the following will make it effortless:

On your local machine, do the following to copy your public ssh key to the clipboard:

cat ~/.ssh/id_rsa.pub | pbcopy

On the Pi, create a new file called ~/.ssh/authorized_keys containing the copied text. This will allow you to ssh without a password.

You can also create a ~/.ssh/config file on your local machine which contains the following:

Host pi
    HostName raspberrypi.local
    User pi

This will allow you to ssh with the trivial shortcut:

ssh pi

Also, there is some general setup that you will want to do if you haven't already:

sudo raspi-config
  • expand_rootfs is important; you may run out of space without it.
  • change_timezone if you want
Jan 27, 2013
The Magic

The other day I discovered the wild mysteries of

Array.apply(null, {length: 15})

This is pure magic, creating an arbitrarily long array without holes. In fact, the above code is essentially the same as doing:

[undefined, undefined, undefined, undefined... undefined]

The 'without holes' part is important since ES5's array methods will only iterate through elements that have explicit values. For instance,

new Array(15).forEach(function(item){ console.log(item); });

won't iterate at all; the callback is never called.

Also,

var a = [];
a[15] = 'hello';
a.forEach(function(item) { console.log(item); });

will only be called once, on the last element.

So, how does this work?

The Function.prototype.apply function does most of the hard lifting. ES5 allows apply to be called with array-like objects, and will respect each element of the array even if there are holes. At each hole in the array, it will infer undefined as the value.

The other key here is that the Array constructor doesn't need to have the new keyword. Just calling the constructor function as a regular function will create a new array with the given elements.

Lastly, {length: 15} is the most terse way to define an array-like object.

Should we actually use this?

One issue is that using the Array constructor is normally discouraged since it is ambigous:

new Array(5);    // [..., ..., ..., ..., ...] (A holey array with length:5)
new Array();     // []
new Array(5, 10) // [5, 10]

There is no way to create an array with a single element using the constructor function. Array literals are preferred for their succinctness and performance.

There are performance issues since you are looping through the list to create it then looping a second time to perform an action. See this jsPerf for more info.

Also, unless you've seen this form before, the intention isn't very clear.

However, I gotta say, I really like it!

Nov 12, 2012

Art

After a visit to the Art Institute of Chicago yesterday, I've thought about what makes art. A perennial question for a sometimes-art-afficionado, it is an important question in a creative space. Software requires an intent, an obsession with meaning, creative flair and intuition in addition to its logic and pragmatism. At the Software Craftsmanship North America conference, we were admonished to avoid shipping shit, let our code speak, express ourselves, and think about our users. All of these things share something with an artist with a brush, clay or wax.

Jun 19, 2012

Anybody who spends lots of time on the command-line will appreciate suggestions for man pages, make files, history, colorful syntax, etc. Enter fish. Anybody who hates excessive configuration will love fish. You will probably love fish.

Get it

Get it! The installer should just work™. When I first tried, the installer didn't update my settings (commented on at the very bottom of this edited post), so I had to do some extra stuff. That shouldn't be a problem now.

Shortcuts

There are some great shortcuts which are part of fish. However, some of them can't be used without first enabling:

Terminal.app > Preferences > Settings Tab > Keyboard Section > "User option as meta key" checkbox

Now try Alt-w on a command or Alt-l on a directory. Also of interest is Alt-p which appends | less; to the end of the line.

CDPATH Variable

Why has no one ever told me about the CDPATH variable?

If you have a root directory that you're always going back to (for me, ~/Development), you need to run:

set -U CDPATH ~/Development .

Now, no matter where your working directory is (PWD), you can:

cd path/inside/development/directory
Jun 13, 2012

It wasn't so long ago that I started hearing people say that Javascript is the assembly language of the web. Just look at the cornucopia of languages that compile to JS; developers are not content with JS standardization efforts. Such fragmentation is unprecedented with no end in sight.

It doesn't stop there either. Dozens of template frameworks and style preprocessors exist alongside other static assets.

Development thrives on choice. I think Coffeescript does a nice of returning Javascript to its functional roots, adding support for dependencies, adding magic like list comprehensions, and avoiding pitfalls like falsy values. But I also recognize that developers need the right tools for the right job. Sometimes this is personal preference, and other times it is mandated by the task at hand or the tools available. It shouldn't matter.

Compilers are not only for bearded men and boring university lectures. Compilers ensure sanity in your source code. Compilers make sure that assets are glued together properly. Compilers allow you to customize the output for efficiency, readability, etc.

The Clang compiler is taking off. Licensing, speed and modularity are important factors in its adoption. Even if we are not talking about C, perhaps we can learn something from this effort. In fact, JS is already an output of LLVM. We are in desperate need of a consistent, modular toolset that will give developers their choice of asset types compliled in a normalized way.

There are some ongoing efforts, but each one is not ambitious enough. Ender.js is a wicked library for the modern web. Prophesying that the days of monolithic libraries are over, it gives developers access to only the tools that they need at the time. Hem and Builder are not agnostic or modular enough to allow developers to write however they like. What the web needs is a tool that will make the wacky world of web apps sane.

One of the primary reasons that compile-to-javascript hasn't taken off is that there is no standard way to express dependencies. Common JS is an excellent effort, but hasn't yet reached critical adoption for key libraries.

New frameworks, languages, and preprocessors crop up every day with the goal of reducing a point of friction in development. Yet there is no LLVM-esque point of inter-operability for the web. Turn on the bat signal, Web City needs its hero.

May 23, 2012

I'm getting more familiar with Spine.js and I'm loving it. Spine is far more consistent and elegant than Backbone as evidenced with the default index.html file. This is automatically generated by Spine.app ( >>spine app my_proj ) when you create a new project:

Break it down

Here's what's happening line-by-line:


var jQuery  = require("jqueryify");

jqueryify is a CommonJS module for jQuery. According to the Hem.js docs, the rationale for using Node.js modules everywhere is that:

you don't have lots of libraries floating around your application. This also has the advantage of explicit versioning; you'll have much more control over external libraries

The name jqueryify is because of the crowded NPM namespace; it is just something unique to call the module.

The require function will call the jQueryify module which will appropriately initialize the caller's namespace. jQuery is a bit disobedient since it also adds its names to the window (global) namespace. Most modules will behave better with scope. This means that we could get away with just calling require("jquerify") and ignoring the return value.


var exports = this;

This follows patterns of var that = this which just affects variable binding.


jQuery(function() {
    //Do something
});

This is jQuery doing its thing. When the page is loaded, the code inside the function will be called.


var App = require("index");

This finds the index module (originally ./app/index.coffee) and stores the variable. In the background, Hem and Stitch are doing their thing creating a module by compiling our CoffeeScript.


exports.app = new App({el: $("body")});

This instantiates our Spine.js Controller class and tells it to infect the entire body HTML element.

It also exports our app instance to the exports variable. (Which happens to be the global window object).

Conclusion

Before I started using Spine.js, I wasn't very familiar with CommonJS. CommonJS is an attempt at standardizing the interaction between JavaScript modules. As stated in the Spine.js documentation, CommonJS has several tangible benefits:

  • Module code design
  • Dependency management
  • Scope isolation, and
  • Namespacing

In general, I've found Spine.js incredibly opinionated. Luckily, most of these opinions are founded in good practices, due in part to Alex McCaw's wealth of experience.

So far, just from this one file, we can see how Spine.js has given us a powerful mechanism to enforce total client-side rendering. It enforces MVC architecture, and provides handy tools to get there. So far, my travels through Spine.js and CoffeeScript have been exciting.

May 20, 2012

One of the toughest decisions to make in any project is what to abstract. In the fragmented landscape of JavaScript, you can do anything.

Abstract

On one end of the spectrum, we can conjure up an app where the project files don't at all resemble the result. A compelling example of this would be to use:

Dependency injection

A descriptive approach would be to use a framework like Dijon. I am also intrigued by the potential power of this approach.

Backbone.js

Backbone is the sweetheart technology of many developers, but there are several parts that make me cringe. The Routes seems hacked together, controllers are labeled View (huge pet peeve), and the Collection is a vestige of DocumentCloud's initial application and shouldn't be part of the framework proper.

However, there is a glut of frameworks being built on top of Backbone. Chaplin, Thorax, and Marionette layer various architecture patterns on top of backbone to make it more specific. I like them, but some of them (like backbone.js itself) seem overly specific to certain use-cases.

Batman.js

An interesting framework for Coffeescript that is full-featured. I haven't yet had much opportunity to inspect it.

Ember.js

The way that Ember automatically renders templates, is controller-agnostic and has the notion of container views is appealing, but not quite as smart as Spine.js

What I really want is a declaritive view model. One that supports view composition and requires less typing. One that allows controllers to be composed (Spine has the lovely concept of Stacks). I should maybe try a variety of options before making up my mind.