Scheduling for the internet

While trying to figure out the best way to synchronise scheduled event start times across different users in different time zones, I managed to work my way around in a bit of a circle yesterday.

I have an app that allows users to schedule events and obviously specify start times. In my initial hacking I was specifying those start times and storing them in the database as strings (not even validated as dates, really just a place holder to mock up the site). So yesterday I thought I would tackle this to make it more functional.

I added a date picker widget and a time picker widget and fixed things so that only dates and times could be specified which I then stored in my database.

But wait, I thought, how do i know what timezone the user intends. After all, when i publish this event to other users I will want to give the start time in their timezone. Hmm... So I started my research on timezones.

I started out trying to put a timezone picker on my event scheduler page which would default to the current timezone of the clients browser. This actually isn't so simple.

A major complication is that I don't really want the timezone, I actually want the locale of which the timezone is only a feature. The other feature is daylight savings time (DST). There are only so many time zones (which is quite manageable) but there are lots of variations in the treatment of DST (not so manageable). Unfortunately for me I need to consider DST if I am to know what real time an event organiser is actually aiming for (they will always be working in local time I presume and would not care to specify start times in UTC).

Here are a few of the interesting libraries that I looked at to get get a handle on this.

• On the client to detect and select a timezone
• Josh Fraser provided the best hope for something simple with his client timezone and DST detection algorithm in Javascript. But he does mention that instead folks should use...
• Jon Nylander's jsTimezoneDetect solution. This is apparently much more advanced and works off the complete list of time locales from the Olson zoneinfo database. Unfortunately this would be tricky to integrate in my web page and would provide a huge number of options for users. I've seen these before on the internet and they are annoying.
• Then on the server to get a nice unix time in my database
• node-time looked interesting
• moment.js seemed to talk the talk but on further analysis I wasn't sure if it knew about DST or if I would have to tell it
• timezone-js may have been the most promising

But then came my small eureka moment... Why am I doing all this work? Well, pretty much because my client side controls give me strings and not date objects. However the browser does know what time locale it's in and how to present dates. So this is where I returned almost to my starting point.

I ripped out all the timezone selector stuff from my page and instead I used the client side date functions to generate a Date object there and transmit a nice simple unix time back to the server for storage. For those that don't know, unix times are the number of milliseconds since 00:00:00.000 01/01/1970 (UTC). They don't care about time locales. So now I do all the locale specific formatting and parsing in the browser. Seemed obvious after I'd done it :)

I may add a widget to my pages to let the user know which zone the times are being displayed in but I'm not sure even that is worth the effort. It would only catch a few issues with people working on devices with the wrong timezone set.

Grunt watch and the Node.js require cache revisited

Still inspired by James Shore's series "Let's Code: Test-Driven Javascript",  I've been continuing with my endeavors to get the Grunt watch stuff working flawlessly(?).

In my last post I mentioned some niggles that were remaining from my previous workaround.

• The workaround only addresses the Mongoose issue
• The workaround assumes intimate knowledge of Mongoose
• Grunt watch still explodes silently when unhandled errors are encountered in tests
• undefined references
• nonexistent requires
• etc.

The good news is that I think I have addressed all of these. In addition to that I've figured out some stuff about how to extend grunt and how to manipulate the Node.js require cache.

First off I thought I'd take a look at Mocha to see if it handled things better. After all Mocha also has a watch function.

• Mocha watch does not explode on undefined references (which is nice)
• Mocha watch does still explode on nonexistent requires (actually I didn't find this out till much later on when integrating with grunt)
• Mocha watch still failed to handle my Mongoose issue
• Unfortunately Mocha watch doesn't integrate with JSHint and actually I'd quite like to lint my code on file changes too

So despite only having a small advantage in not falling over so much I thought Mocha showed more promise than NodeUnit and as James noted it is much more active on GitHub. In fact it's under the same banner as Express and Jade which are definitely very popular and well maintained frameworks for Node.js.

Next thing was to integrate Mocha with Grunt so that i can use the Grunt watch function to both lint and run tests on file changes.

The nice thing about writing my own task to run Mocha instead of NodeUnit is that it was then quite easy to fix the issue of exploding on nonexistent requires... It just needed a try/catch around the mocha.run call. In retrospect I could probably have added this to the existing NodeUnit task but by the time I got to this point, I'd already ported all my tests to Mocha.

[A short interlude on Mocha and Should...]

James noted in his videos that Mocha is targeted as a BDD test framework and as such he is not so keen on it's verbosity. I can see what he means but, to be honest, I don't find it much of an issue and in fact quite like it, so for a while at least, I think i'll stick with it.

I also tried the should.js assert library that provides an interesting take on asserts by making them a bit more natural language like. Things like: thing.should.have.property(things).with.length(5);

On first take I thought cool and went full steam ahead in making all my asserts like this. Currently though I'm not sure I like it.

For one, I keep thinking that I should be able to write something in a natural way but find that it's not really supported - it kinda feels like I'm being teased. This will lessen I guess as I really learn the idioms.

A more annoying problem though is related to the way Javascript handles types and comparisons. I keep finding comparisons that i think should work and don't and then comparisons that I think shouldn't work and do! I think this is made worse by hiding the comparisons inside assert functions. As a result I'm starting to come to the opinion that not only is the should framework more trouble than it's worth but in fact any assert framework that hides comparison logic is not such a good idea to use in tests in Javascript. This includes very standard things like: assert.equal(object1, object2);

I may revert to just a single check function that will better reflect how comparisons would actually be written in production code. Ie: assert(conditionalCodeThatResolvesToTrueOrFalse);

[...interlude over]

So there I have it, I can now run my tests as files change and rely on the watch task to keep going no matter what happens (so far!). Just the mongoose problems to resolve then, and actually I added another.

• If a unit test beforeEach function falls over then the after functions are not run
• This means that as I open a database connection in before and close it in after, when I get such an error I then continue to get failures when files change due to not being able to open the database anymore (it's already open)
• Not as serious as the silent failures as at least the watch process keeps pinging me and I can restart it. But still a little annoying

This new issue got me thinking again about the require cache. My previous investigations here had proven fruitless but then, perhaps I had been led astray by some dubious comments on StackOverflow. Beware, this code does not work:

for (var key in Object.keys(require.cache)) {delete require.cache[key];}

So now I was thinking about the Mongoose module.

• The problem isn't that the changed module is still in cache
• The problem is that the Mongoose module is still in cache
• In fact the problem is that any modules are still in cache
• I must clear the cache completely before running my tests!
• Actually I had tried this and it didn't seem to work
• However I had tried it in my tests themselves, now I could try it in my new grunt task :)
• I had already needed to add code that dropped all my own files from cache to make things work. It made sense to drop the rest too when I come to think about it.

So i fixed the code above:

for (var key in require.cache) {delete require.cache[key];}

Tidied up my mocha task adding support for options and this is what I have in a new module...

To use this I dropped it in a grunt tasks directory and updated my grunt.js file...

Note that the call to loadTasks takes the directory name. Also note that I overrode the built in NodeUnit test task and that the options to pass into mocha are given in the mocha config property.

So that's it I no longer have to use my Mongoose workaround as the Mongoose module is cleaned up along with everything else before I run the tests :)

I hope this will save me from similar gotchas in other modules too, but I guess I'll just have to code and find out :D

NodeUnit, Mongoose and Grunt watch

Edit: Although interesting to me as a history to my Node.js testing issues, this article is now pretty much superseded by this one which better addresses all of the below problems

Now that James Shore's "Test Driven Javascript" series has kicked off I've been integrating unit tests into the 5Live hangout project. This has, for the most part, been simpler than I expected. NodeUnit is pretty easy to use and from one of the comment threads I have been introduced to Grunt which allows me to tie all my lint tasks and unit tests into a single automated script (James has been doing this himself in Jake but I figured I would give Grunt a try as it does some of the 'grunt' work for me :)) .

Like I said, for the most part this has all been going swimmingly. One of the nice features of Grunt that I discovered is the watch task. This allows me to watch a list of files and when they change, automagically kick off my lint tasks and unit tests - very nice :D

There are some problems though. My application uses Mongoose to interact with a MongoDB database. As such I follow the standard mongoose pattern of using a singleton and defining my model schemas like this...



As I'm doing TDD I actually start off with something like this in a separate test file...



That's all hunkydory. I can run the tests and they pass. I can kick off grunt watch and leave it running while I start editing my files. Let's see what happens when I change my test, thusly...



As expected grunt watch pings me to let me know that my test has failed :)

So I go back to my model and update the greeting function...



Gah, grunt watch pings me again to say that my test still fails. Puzzlement abounds!

If I stop grunt watch and run the tests manually they pass! So what's going on?

Well I wasted a lot of time messing around with the require.cache object, as I figured it was something to do with Node.js module caching, but that wasn't it at all. Either NodeUnit or Grunt is smart enough to remove the changed files from the module cache (I think it must be Grunt that does this but I didn't check).

Eventually I realised that it was the mongoose singleton that was causing the problem. After all this only happened with my mongoose model tests. As the mongoose singleton persists between test runs it doesn't matter that I change the methods on my models, the old versions also persist.

Again I tried a number of workarounds but so far the best seems to be the following.

First I created a wrapper for the mongoose singleton which allows me to reset the schemas...



Next I integrated this wrapper into my tests (only the tests, i still use the the mongoose singleton directly in the model implementations)...



So why do I prefer this solution and what else did I try?

Well I also had another workaround which fixed the problems with method updates.

• Instead of using Schema.methods to assign methods I used Model.prototype
• Instead of using Schema.statics to assign static methods I just assigned them to the Model directly

Why didn't I like this?

• This solution meant a little rejigging of the code to what seemed like a non standard pattern in the actual implementations
• This did not fix a similar problem with updating the actual Schema - ie. adding or removing fields

I still don't much like my eventual workaround as...

• it depends on knowledge of the the internals of Mongoose (which might change)

But at least it's contained in my tests and seems to work for all changes in the model.

However, even with this workaround in place I'm still  not fully happy with the way grunt watch works.

• Annoyingly, it exits on a number of test failures particularly when things are not yet defined. This happens a lot when doing TDD, it's how we write failing tests.
• When it does exit it doesn't actually ping me. As such I have to keep looking to see if it stopped (If i have to do this all the time, it occurs to me that I may as well not use it and instead run my tests manually)
• I'm now just waiting for the next gotcha as I only worked around a problem with Mongoose.
• It seems to me quite likely that there will be other libraries that follow similar internal patterns and they are likely to trip me up in the same way

I have a solution to suggest though...

• Spawn a new Node.js process at least for every grunt watch event if not for every NodeUnit test
• Wouldn't this fix the problem once and for all?

Amazon EC2 learnings

Yesterday I discovered that Amazon AWS offer a free tier which basically means I can have a free server (possibly more than one) in their cloud for a year!

Awesome, I'll have some of that :)

I decided to see if could get our 5Live hangout project up and running there. Figured it would be useful as our free heroku plan limits us to a 16MB database!!!! On AWS I can have up to 30GB of storage for free :)

Of course I'll have to be my own sys admin to get it though. So that's where the adventure begins. This is what I needed to setup...

• A server
• Some storage
• Install Node.js
• Install MongoDB
• Install Git
• Open ports to allow access from the interwebs

In figuring this stuff out I probably created and terminated about 20 EC2 instances. There's the first 2 things I learned...

• Amazon refers to it's virtual machines as EC2 instances
• Amazon calls deleting an instance "terminating it"
• When you terminate an instance it does not go away immediately (takes about 20 minutes) but it is not recoverable
• There is an option for termination protection which I haven't tried but might be a good idea :)

Only a limited number of the virtual machine types are covered by the free tier but that's ok I only actually tried 2 of them. Didn't think I'd be interested in running my stuff on windows so I only tried the Amazon Linux and Ubuntu 12.04 images. Both of which are free in the micro configuration (1 core, 613MB RAM). After switching between the 2 a few times I settled on Ubuntu mainly because it is more familiar to me. However my research suggests that the Amazon Linux images might be better optimized for EC2.

Now for the real purpose of this blog post, which is mainly for my own notes, these are the steps to setting up the above list of requirements.

Create an Amazon AWS account

First we need an AWS account

1. From  https://aws.amazon.com/free/ sign up for a new account if you don't have one and verify with the fancy phone call verification
2. Wait for email confirmation of the new account

Create an EC2 instance

We need a virtual machine

Choose the free tier eligible machine type

Keep the default machine options

Create a new security group

1. Head over to http://aws.amazon.com/console/ and sign in with your new account
2. Select the EC2 link
3. Select the Instances/Instances link on the left hand side
4. Click the Launch Instance button
5. Choose the Classic Wizard option and click Continue 
6. Choose Ubuntu Server 12.04 LTS 64bit and click Select 
7. Keep the default options for the machine type as pictured above and click Continue 
8. Keep the default options for for the machine features as pictured above and click Continue 
9. Enter a name for the instance (this is only used for display in the AWS console and is not the machine name) and click Continue 
10. Next you will have to create a key pair - this is used instead of passwords to log on to the virtual machine using SSH (If this is not the first instance on the account then you can reuse an existing key pair). Enter a name for the key pair and click Create & Download your Key Pair - keep this somewhere safe but accessible. Then click Continue 
11. Create a new security group with at least port 22 open so that you can SSH to the instance as pictured above. I have decided that it is best to create a new security group for each EC2 instance as it is not possible to change to a different security group after the instance has been created. However it is possible to change the rules in a security group, so if you want different instances to have different rules then you need to create different security groups for each instance. Then click Continue 
12. You will then be presented with a page to review so just click Launch and on the next dialog click Close 

Create an EBS volume

We need an Elastic Block Store volume so we can separate our MongoDB data from the OS volume

1. Select the Elastic Block Store/Volumes link on the left hand side. Notice that there is already an 8GB volume for the EC2 instance OS. Make a note of the zone for this existing volume (eg. us-east-1d), we will want to create our new volume in the same zone so the EC2 instance can be attached to it
2. Click Create Volume 
3. Select the size of the volume (eg. 10GB) and the same zone as noted in the last step. Don't select a snapshot. Click Yes, Create 
4. Right click the newly created volume and select Attach Volume 
5. Select the newly created Ubuntu instance and leave the Device field to the default. Click Yes, Attach. This will actually attach the volume to /dev/xvdf and not /dev/sdf on this version of Ubuntu, as noted on the dialog

Start the instance and log on using SSH

We're going to need our key pair file in the next step. On OSX and linux it can be supplied to the ssh command using the -i option but on windows I use Putty. Putty does not accept *.pem files as generated by amazon so it's necessary to convert it to a *.ppk file using PuttyGen. Anyway follow these steps to logon...

1. In the AWS console go back to Instances/Instances on the left hand side
2. Select the instance and on the Description tab scroll down until you find the Public DNS entry. This is the public host name of your server. As an aside it also contains the static IP address in case you want to know what that is - eg. ec2-.compute-1.amazonaws.com
3. Launch Putty and paste the Public DNS host name into the host name field
4. Prepend the host name with ubuntu@ so that you don't need to specify the user name when connecting (the default user is called ubuntu)
5. On the left hand side select Connection/SSH/Auth.
6. Under Private key file for authentication browse for the *.ppk file generated by PuttyGen from the *.pem file created and downloaded from Amazon
7. Go back to the Session section at the top on the left hand side and save the session with a sensible name
8. Click Open and you should just be logged in as the ubuntu user (after accepting the public key)

Format the EBS volume and mount it permanently

We want a nice efficient file system and it seems that it's de rigueur to use XFS. XFS is supported by the Ubuntu 12.04 kernel but the tools to format volumes are not there by default. Anyway here are the steps to follow at the command line...

1. sudo apt-get install xfsprogs
2. sudo mkfs -t xfs /dev/xvdf
3. sudo mkdir /mnt/data
4. sudo nano /etc/fstab

The last step will start nano so that we can edit the /etc/fstab file to ensure that our volume is mounted whenever the machine reboots. Add the following line...

• /dev/xvdf /mnt/data xfs noatime,noexec,nodiratime 0 0

Write out the file with ctrl-o and exit with ctrl-x.

Now we need to mount the data volume. At the command line...

• sudo mount -a

Install the latest stable Node.js

At the time of writing the default Node.js package available in Ubuntu is 0.6.12 and the latest stable is 0.8.2. In order to get the latest stable release do the following at the command line...

1. sudo apt-get install python-software-properties
2. sudo apt-add-repository ppa:chris-lea/node.js
3. sudo apt-get update
4. sudo apt-get install nodejs npm

Install and start the latest stable MongoDB

At the time of writing the latest MongoDB was 2.0.6 and that is what we download in the following steps. Check with http://www.mongodb.org/downloads to see if there is a newer version. At the command line...

1. cd ~
2. curl -O http://downloads.mongodb.org/linux/mongodb-linux-x86_64-2.0.6.tgz
3. tar -xzf mongodb-linux-x86_64-2.0.6.tgz
4. cd mongodb-linux-x86_64-2.0.6/bin
5. sudo mkdir /mnt/data/db
6. sudo chown ubuntu /mnt/data/db
7. ./mongod --fork --logpath ~/mongod.log --dbpath /mnt/data/db/
8. cd ~
9. tail -f mongod.log

This will start the MongoDB daemon in the background and output the logging to ~/mongod.log. The last command allows you to check that the daemon starts up ok. Once it has completed the startup sequence then it is safe to ctrl-c out of the tail and mongod will continue running. To stop mongod, the safest way is from the mongo client. At the command line...

1. cd ~/mongodb-linux-x86_64-2.0.6/bin
2. ./mongo
3. use admin
4. db.shutdownServer()

The last command shutdown the server and prints out lots of stuff that looks like errors but it should be fine and it should be possible to start the server again as before.

Install Git

I use GitHub and all my code is up there so I need git to put it on my new server. At the command line...

• sudo apt-get install git

Opening more ports

While developing Node.js applications I usually use the default Express port of 3000. You will remember that when we created the server instance we only opened port 22 in the security group. In order to hit the server on port 3000 we have to add that to our security group too...

1. In the AWS console select Network & Security/Security Groups on the left hand side
2. Select the the security group created specifically for the server instance
3. Select the Inbound tab
4. For Create a new rule select Custom TCP rule 
5. For Port range enter 3000
6. For source enter 0.0.0.0/0 
7. Click Add Rule 
8. Click Apply Rule Changes 

It should now be possible to connect to services running on port 3000 from the internet. Remember that the host name is the Public DNS entry under the EC2 instance description.

Startup Weekend Amsterdam - The 5Live Crew

Had a fantastic time at Startup Weekend Amsterdam this weekend. Big up to the organisers :)

It was hard work and I think it will take a while for my girlfriend to forgive me (I found out about it and signed up Thursday with much too little discussion) but I'm so glad I did it.

It kicked of at 6pm Friday with 66 pitches! Some good, some bad, some completely indecipherable but all of them given and received in great spirit, in 60 seconds or under (with change overs I guess it would have been at least 2 hours of pitches, phew).

There was a shameless pitch to clone Kickstarter in Europe (did you know Europeans couldn't raise funds on Kickstarter?)... This actually got one of my votes.

There were a lot of pitches to provide match up services for sports enthusiasts, I counted at least 4. These were interesting to me as I know a French guy (hey, Alex, if you're reading!) who's had some success with this in Canada for ice hockey. Check out Hockey Community! Like I say interesting but i din't vote for any of them as I didn't want to encourage competition for my friend (who I think should branch out to other sports ;))

A cool sounding app called OhHeyWorld got my second vote. With one click it lets you notify friends and family when you arrive at a destination by checking in by email, facebook, twitter, etc all at once. I really liked this but it didn't get so many other votes so happily the guy pitching it ended up on our team :)

My final vote though went to a Googler who wanted to build an app on top of Google Hangouts to match coaches, music teachers, etc to people online for live lessons via video. 2 things peaked my interest. I've had this conversation before, particularly with musicians. And second it would be nice to learn something about integrating with Google Hangouts.

I didn't pitch any of my own ideas, although I kinda wish I had. Next time I will... and there will be a next time :)

So with all voting completed the pitches were narrowed down to 20. By which time the Google Hangout idea had merged with another to provide business mentoring over video conferencing, inspired by Quora (of which I'm largely unaware, but have just signed up). This is what I ended up working on with 7 other cool guys. We were 5 business guys, 2 developers and 1 designer (I think). Heavy on the business side and it showed after the first day. I'm not sure how many visions there were or how many times we pivoted (being 1 of the 2 techies I was pretty much head down coding the whole time - hoping that it was in the right direction) but at the end of the night on Saturday the cracks were starting to show. It was time for rest.

The next morning, although we were down one business guy who I think was unimpressed by the lack of focus and constant direction changing, there was definitely more commitment to get things done and to try and ship, after all we only had till 4pm.

We seemed to have a single vision now and we set about building it up and applying copy, etc. We needed a presentation and we needed some kind of validation. To be honest we already had some validation. All the pivoting the day before had come from surveying conference participants and collecting feedback. Ok, we didn't get out of the building physically but we did sell the expertise of one of our team members something like 7 times for $5 (Now he has to do some video conferences with the people who stumped up the cash). This wasn't even with conference folks, these were real people on the interwebs. We sold him through fiverr at an admittedly knock down price for his real estate know how and you may say that we only validated that people will buy gold if we sell for the price of coal. I disagree though, we validated that people are willing to pay for live interactive video sessions in which they can learn something... and they actually paid, not just promise to pay or expressed an interest.

This was our eventual proposition (I almost typed final but there may be more pivots to come) and 5Live was born. We will provide a market place to match up those with skills/knowledge to those who want to have those skills/knowledge. Sessions can be scheduled by the skilled for free and spots can be reserved by the young padowans for 5 euros/dollars or less. Then at the designated time each will click on some button and enter into a live interactive video conference through which knowledge will be imparted. The subsequent revenues will then be distributed back to the skilled.

I see it as a place where everyone is equal and sharing their wisdom with each other. Perhaps in the evening instead of watching TV. I see it as a natural progression from broadcast TV through social video like YouTube to the next wave of social knowledge sharing where you can ask your questions live. It's Etsy meets YouTube meets fiverr meets Quora (if I understand Quora correctly). This vision may not be the same as the other founders and may change once it's tested (will change!) but right now it looks like it has legs.

So at the end of the weekend what did we have?

Third place!! Yay :)

We were pushed out by a very worthy non-profit, charity/donation app called Easygiving who took first place for allowing people to easily manage and adjust monthly charity plans and donation distribution. And a service for helping you transfer your apps between mobile phone platforms when you switch to Android and can't find that app you loved so much on your iPhone (or vice versa) called Aloha. Kudos to them :)

But we still believe in our thing! And to prove it here are the slides, a video and a link so you can see more (if the link is dead then it probably means we changed the name and hit the big time - you should have been here last week ;))

• Slides
• 5Live site (minimum viable product?)