York Solutions, LLC

unpkg - A CDN for npm packages

Wed, 30 Nov 2016 00:00:00 +0000

Now we can stop the deadly sin of committing our assets to git for Bower, CDNJS, and jsDelivr. Instead let’s use unpkg, a CDN for npm.

We typically think of npm as a package manager for node modules written in plain JS that will run without any compilation, but npm is actually built to support the publishing of compiled and packaged assets. This is useful for compiling CoffeeScript or transpiling ES2015. In fact, npm allows us to publish anything. So lets publish our client-side assets there too.

unpkg is a CDN that runs in front of npm. Any package published on npm is available through unpkg. Assets are accessed through URLs using this simple format: https://unpkg.com/package@version/file. For example, https://unpkg.com/lodash@4.17.2/lodash.min.js.

Generate a UMD package using your favorite build tool.
Add your packaged assets to .gitignore so it’s not commited to git.
Add your packaged assets to the files array in package.json so they’re part of the package published to npm.

It’s simple and beautiful. unpkg is now my preferred CDN.

Element.querySelector() Evaluates the Entire DOM

Tue, 01 Nov 2016 00:00:00 +0000

I used to think that Element.querySelector() and Element.querySelectorAll() worked like jQuery.find() in that they would evaluate a CSS selector against an element’s descendants. But today I learned that they are not the same.

Instead of evaluating selectors against an element’s descendants, querySelector() and querySelectorAll() evaluate selectors against the entire DOM then filter the results to those that are descendants of the base element.

Consider the following example, inspired by MDN.

<div>
  <p>Some <span>paragraph</span> text.</p>
</div>

console.log( $('p').find('div span').length ); // prints 0

console.log( document.querySelector('p').querySelectorAll('div span').length ); // prints 1

Using Redis to Persist PHP Sessions on Heroku

Fri, 22 Jul 2016 00:00:00 +0000

This article focuses on the specific task of using Redis to persist PHP sessions when running your app on Heroku. If you’re not in that exact same situation then this article will not apply to you.

I assume that you have a pre-existing PHP app deployed to Heroku that uses sessions and that, like me, you didn’t realize until after the app was deployed that you lose your sessions every time you deploy a new version of your app.

1. Provision the Heroku Redis Add-on

Provisioning the Heroku Redis addon is straight forward. I prefer provisioning add-ons via the dashboard.

2. Add the Redis Platform Package to Composer

Add this line to your composer.json require list:

"ext-redis": "*"

That’s a special composer syntax for platform packages. This informs Heroku that it should enable the PHPRedis package.

You will know if Heroku recognized the platform package by seeing an entry in your build log that looks like this:

-----> Installing platform packages...
       - php (5.6.24)
       - ext-redis (2.2.7)

Locally, you’ll have to install Redis yourself or use regular PHP sessions. If you choose not to install Redis locally then you can use the --ignore-platform-reqs flag when running composer update and composer install, otherwise Composer will throw an error when it’s unable to find the extension.

You might review the full list of other PHP extension that you can enable on Heroku via composer.json. I noticed I was compiling a 3rd party extension that Heroku already had available for me. Using Heroku’s built-in version lowered my deploy time from minutes to seconds and lowered my slug size by about 200 MB.

3. Configure PHP Sessions to use Redis

PHP session storage is configured via the php.ini variables session.save_handler and session.save_path. Normally with Heroku we would recommend setting those in a .user.ini file. However, the default conenction string that Heroku uses for Redis is not the same format that we need to use in PHP. Therefore we must set this values in our PHP app code so that we can parse Heroku’s format and construct the format that PHP expects.

<?php

  // Assume we don't want to use Redis if the environment variable isn't set.
  // This prevents us from needing to install PHPRedis locally. We're okay with
  // using the default file storage for sessions in our development environments.
  if($_ENV['REDIS_URL']) {
    $redisUrlParts = parse_url($_ENV['REDIS_URL']);
    ini_set('session.save_handler','redis');
    ini_set('session.save_path',"tcp://$redisUrlParts[host]:$redisUrlParts[port]?auth=$redisUrlParts[pass]");
  }

4. Validate

Load any page in your app that starts a session then connect to your Heroku Redis instance and check for the existing of a session key.

heroku redis:cli -a APP-NAME -c APP-NAME

That opens a connection to the Redis CLI in the remote Heroku instance.

ec2-54-221-206-137.compute-1.amazonaws.com:6379> keys *

That tells Redis to list all keys. You should see something like this:

1) PHPREDIS_SESSION:vm0790r1veot5dhkkpdotjek71

There we see that our PHP app successfully saved a session to Redis.

Conclusion

Herouk is an amazing platform. It made this a painless experience.

A mighty thanks to DigitalOcean for a guide on how to setup Redis for PHP session handling the hard way.

If you prefer to use Memcached, Heroku has written a guide for you.

Using JSON Schema to Validate JSON

Wed, 04 May 2016 00:00:00 +0000

When creating a new REST API I wanted to validate the return format of the JSON response. While sarching for a solution, I discovered JSON Schema.

JSON Schema defines a human and machine readable description of a JSON structure and format. It looks like this:

{
	"title": "Example Schema",
	"type": "object",
	"properties": {
		"firstName": {
			"type": "string"
		},
		"lastName": {
			"type": "string"
		},
		"age": {
			"description": "Age in years",
			"type": "integer",
			"minimum": 0
		}
	},
	"required": ["firstName", "lastName"]
}

It’s a little verbose, but it’s flexible. Most importantly it enables automatic validation of JSON responses. With many JSON schema validators available, you can quickly integrate them into your tests. Here’s an example using tv4.

assert(tv4.validate(data, schema));

The first weakness you might run into is that the spec allows undefined properties by default. In most situations, you want unexpected properties to be an error. Most validators account for this by having a “ban unknown properties” mode.

If your schema is advanced you might start wanting some form of inheritance, but JSON schema doesn’t support it. GEDCOM X JSON Schema gets around this by merging the property objects itself as it assembles the schema.

Reset Migrations With Knex

Wed, 04 May 2016 00:00:00 +0000

Knex is a Node.js SQL query builder that supports multiple database implementations. It also supports migrations which are essential to any serious project using a relational db.

During local development, when running tests, and particuarly when testing migrations, I often wanted to reset the db and rerun all migrations. I was shocked to find that Knex migrations doesn’t have a reset method.

I first tried to follow some advice and write a bash script. After a coule of hours trying different methods of running postgres commands from the CLI and battling database user permissions (both for authentication and db access rights) I ended up with this:

#!/bin/bash

# Delete and recreate the database
psql -U postgres -h 127.0.0.1 -c 'DROP DATABASE IF EXISTS my_database;'
psql -U postgres -h 127.0.0.1 -c 'CREATE DATABASE my_database;'
psql -U postgres -h 127.0.0.1 -c 'GRANT CONNECT ON DATABASE my_database TO my_db_user;'

# Run migrations to generate the schema
knex migrate:latest

It did finally work, but I wasn’t happy with it. It was very specific to my environment. Plus I don’t like including bash scripts in Node projects unless there’s absolutely no other sane method.

After reading more about Knex migrations and realizing that the long promised arctic migrations API wasn’t coming, I looked at the current Migrate source to find that the API was very neat (perhaps it’s improved in the past year). I quickly found a very simple solution.

var migrate = knex.migrate;

// Force unlock in case of a bad state
migrate.forceFreeMigrationsLock()

// Get completed migrations
.then(function(){
  return migrate._listCompleted();
})

// Rollback migrations
.then(function(completedMigrations){
  return migrate._waterfallBatch(0, completedMigrations, 'down');
})

// Migrate to the latest
.then(function(){
  return migrate.latest();
});

Affordable Email Forwarding For Small Projects

Wed, 23 Sep 2015 00:00:00 +0000

The biggest frustration I’ve had recently as a developer is finding cheap email forwarding. When working on a small project, such as RootsSearch, I want to have an email listed somewhere for anyone to contact me. I want that email to be forwarded to my personal inbox so that I don’t have to check multiple accounts with multiple hosts.

Domain registrars often offer free email forwarding, such as Namecheap or Google. But I like to use CloudFlare for my DNS and CloudFlare requires you to use their nameservers which means your registrar can’t control your MX records and therefore you need a separate solution for email forwarding.

When I first ran into this problem I figured the solution would be easy. A previous employer had used Dyn and later DuoCircle for email forwarding so there was at least one company out there. But DuoCircle charges $59.95 a year per domain. That quickly adds up for small projects, especially if you have multiple.

33Mail looked promising, especially at only $1 a month to use custom domains. But I quickly ran into a feature I couldn’t tolerate nor prevent: to every email I received, 33Mail added a banner that said “This email was sent to the alias… and 33Mail forwarded it to you.” I didn’t mind so much seeing the banner in the emails I received, but I didn’t want them to appear in the quoted section of an email thread when I responded to customers. I could easily remove them when replying (which I did) but it meant I was again on the hunt for a good email forwarding solution.

Pobox was another promising candidate. It has been around a while and was cheap at $20 a month. Though their volume limits seemed low and I didn’t want to be paying for services that I wouldn’t be using.

After weeks of searching on and off, I finally found Mailgun by Rackspace. I was skeptical at first because it looked just like Mandrill which doesn’t support inbound email forwarding out of the box. But it turns out that Mailgun does support email forwarding. Their pricing is also perfect: 10,000 emails every month for free. Sold.

Drawing Family Pedigrees With D3

Thu, 11 Jun 2015 00:00:00 +0000

I recently set out to learn how to draw family pedigrees with D3. Their website is full of examples including an actual pedigree chart. That example was much too simple for my requirements. I wanted it to be styled like most pedigrees in software with a box around the person. I also wanted it to expand, collapse, pan, and zoom. As I perused other D3 examples, I had difficulty mixing and matching techniques and separating the basics from the fancy stylistic additions. I decided to document and share what I learned in a incremental set of examples so that you wouldn’t have to become an expert in D3 just to draw a pedigree.

The D3 pedigree examples are hosted on GitHub. At the time of writing, they include 5 examples:

Basic: A basic static pedigree.
Expand and Collapse: Click on persons to expand and collapse the tree.
Smooth Transitions: Changes are animated when the tree is expanded or collapsed.
Ancestors and Descendants: Show both ancestors and descendants.
Ancestors and Descendants - OOP: A more sane example of ancestors and descendants using classes.

Visit the repo to learn more about the implementation details. Here, I want to share my thoughts about whether D3 is a good tool for drawing pedigrees.

Pros

D3 has a built-in [tree layout]. It’s defaults need to be tweaked a little for rendering pedigrees, but it still works great.
Drawing the connecting lines is trivial in SVG.
Startup time for a basic pedigree is very small.

Cons

D3 is too powerful for most people to tame. Without being able to copy and paste from the enormous library of examples, most users would not be able to get anything done¹. The moment you need to modify anything, you have to become an expert in both SVG and D3.
SVG is great for shapes with a little bit of text. But pedigrees are lots of text with a few shapes. SVG does not support text wrapping nor inline styles (the ability to style one word, character, or phrase different than the other text around it). You can mimic those feature by adding different text nodes and calculating their position, but it’s not ideal. Some browsers allow you to embed HTML in a foreignObject so that you can use HTML and CSS to style text, which has all the tools we need, but support is partial for those browsers that allow it while no version of IE supports it at all. That renders the foreignObject useless for the majority of production projects.
The complexity increases by orders of magnitude if you want to make the pedigree dynamic and interactive (in our case to expand and collapse).
D3 does not lend itself well to an OOP approach. This is why the examples are generally one large wall of code.

Conclusion

If you just need a basic pedigree display than D3 is great. You’ll end up spending most of your time getting your data into the proper format than you will configuring the tree. However, if you need anything more advanced then I would recommend just using HTML. Ancestry, FamilySearch, and findmypast all use HTML for their pedigrees. MyHeritage is the only large tree website I know of that uses SVG and they didn’t use D3 to do it.

I don’t think I could’ve done anything either without those examples. ↩

Automated Testing and Code Coverage in Node.js

Wed, 20 May 2015 00:00:00 +0000

Code coverage is a valuable metric for understanding the quality of your test suite. In this post we will show how to quickly generate code coverage statistics for your JavaScript and Node.js libraries using Mocha, Travis CI, Istanbul, and Coveralls.

Test Suite with Mocha

We need a test suite before we can generate code coverage. Our favorite testing framework is Mocha with Chai for assertions. A test file might look like this:

var expect = require('chai').expect;

describe('Array', function(){
  describe('#indexOf()', function(){
    it('should return -1 when the value is not present', function(){
      var array = [1,2,3];
      expect(array.indexOf(5)).to.equal(-1);
      expect(array.indexOf(0)).to.equal(-1);
    })
  })
})

Then in our package.json file we would add a test script to run with npm test.

"scripts": {
  "test": "mocha"
}

Automated Testing with Travis CI

We can automatically run our test suite for each commit using Travis CI. Sign up, add your GitHub repository, then add a .travis.yml file to the repo.

language: node_js
node_js:
- '0.10'
script:
- 'npm test' # tell Travis how to run your test suite

Every time a commit is pushed to your repository on any branch, Travis will automatically queue up a test runner that clones your repository, installs all dependencies, runs your test suite, and reports the status. You can even add a status badge to your repos README that shows the current build status and acts as a handy link for Travis. The best part is it’s 100% free for public GitHub repos.

Code Coverage with Istanbul

Istanbul is the only viable JavaScript code coverage tool we are aware of. It’s fast, easy to use, can be installed via npm, and doesn’t need to be installed globally. To run, just add another entry in the package.json scripts field.

  "coverage": "istanbul cover ./node_modules/mocha/bin/_mocha"

You’ll notice that we have to call the mocha bin file directly. This is because the main mocha file forks into a child process as explained here.

When you run npm run coverage it will generate coverage statistics in the coverage directory including a browsable HTML report in coverage/lcov-report/index.html. We suggest adding coverage to your .gitignore file so that it’s not committed to your repo.

Automated Code Coverage with Coveralls

Similar to how we used Travis CI to automate our testing, we can use Coveralls to automate our code coverage. The first step is to sign up and enable your repository. This will generate a token which you need to add to a .coveralls.yml file.

repo_token: mycoverallsrepoymltoken

Second, you need to install the coveralls node package. This allows us to send our node coverage stats to Coveralls so that they can store and process it.

Third, we need a new command in our scripts field:

  "coveralls": "npm run coverage && cat ./coverage/lcov.info | ./node_modules/coveralls/bin/coveralls.js"

This command will run our test suite to gather coverage stats then pipe them into the coveralls node package which sends the stats to coveralls.

The last thing to do is update .travis.yml so that our new coveralls command is run automatically by Travis each time a commit is pushed to our repository. This will still run tests and track our build status while also allowing coveralls to track changes to our code coverage.

script:
- 'npm run coveralls' # tell Travis how to run your test suite

An option step, though one we highly recommend, is to add a coverage badge to your README that shows off your coverage percentage and adds a quick link to Coveralls browsable coverage stats.

Conclusion

Our full scripts field in package.json now looks like this:

"scripts": {
  "test": "mocha",
  "coverage": "istanbul cover ./node_modules/mocha/bin/_mocha",
  "coveralls": "npm run coverage && cat ./coverage/lcov.info | ./node_modules/coveralls/bin/coveralls.js"
}

Of course, you can add additional scripts. We encourage you to browse the documentation for Mocha, Travis CI, Istanbul, and Coveralls to see what else is possible.

See FS-Traversal for a real example of the setup we described here.

Using HTML Templates with JavaScript on the Client

Thu, 07 May 2015 00:00:00 +0000

Generating HTML in JavaScript can be painful. It quickly gets out of control when building any advanced UI.

Let’s say we want to display a contact list using a list of contacts with this schema:

{
  name: 'Joe',
  img: '/pics/Joe.png',
  phone: '987-654-3210'
}

We want the resulting HTML to look like this:

<div class="contacts">
    <div class="contact">
        <img class="thumb" src="/pics/Joe.png" />
        <div class="name">Joe</div>
        <div class="phone">987-654-3210</div>
    </div>
    <!-- more contacts... -->
</div>

Vanilla JavaScript

Traditionally, with vanilla JS, we generated elements directly. jsFiddle

var list = document.createElement('div');
list.className = 'contacts';
for(var i = 0; i < contacts.length; i++){
    var contact = document.createElement('div');
    contact.className = 'contact';
    
    var img = document.createElement('img');
    img.src = 'http://lorempixel.com/50/50/people/' + contacts[i].name;
    img.className = 'thumb';
    contact.appendChild(img);
    
    var name = document.createElement('div');
    name.className = 'name';
    name.innerHTML = contacts[i].name;
    contact.appendChild(name);
    
    var phone = document.createElement('div');
    phone.className = 'phone';
    phone.innerHTML = contacts[i].phone;
    contact.appendChild(phone);
    
    list.appendChild(contact);
}
document.body.appendChild(list);

Using vanilla JS is a lot of work; 23 lines of code to generate our simple contact list. It’s also error prone because of cross-browser compatibility issues.

jQuery

jQuery, released in 2006, solved our browser compatibility issues and made DOM manipulation easier. jsFiddle

var list = $('<div>').addClass('contacts');
for(var i = 0; i < contacts.length; i++){
    var contact = $('<div>').addClass('contact');
    
    var img = $('<img>')
        .attr('src', 'http://lorempixel.com/50/50/people/' + contacts[i].name)
        .addClass('thumb')
        .appendTo(contact);
    
    var name = $('<div>')
        .addClass('name')
        .text(contacts[i].name)
        .appendTo(contact);
    
    var phone = $('<div>')
        .addClass('phone')
        .text(contacts[i].phone)
        .appendTo(contact);
    
    list.append(contact);
}
$(document.body).append(list);

This is about the same number of lines of code as vanilla JS, though there are many less characters which I say leads to enhanced readability. We also don’t have to worry about browser compatibility.

Handlebars

We can do better. Instead of building the HTML in JavaScript, let’s separate the HTML from the JavaScript in the same way we separate CSS from HTML. We’ll use Handlebars to create HTML templates which are later compiled and populated.

First, our HTML template is embedded in a script tag in our HTML page.

<script id="contacts-template" type="text/x-handlebars-template">
    <div class="contacts">
        {{#each contacts}}
        <div class="contact">
            <img class="thumb" src="http://lorempixel.com/50/50/people/{{name}}" />
            <div class="name">{{name}}</div>
            <div class="phone">{{phone}}</div>
        </div>
        {{/each}}
    </div>
</script>

Then in JavaScript we retrieve, compile, and evaluate our template. jsFiddle

var source = document.getElementById('contacts-template').innerHTML;
var template = Handlebars.compile(source);
document.body.innerHTML = template(data);

That is much better. Our HTML is in a template which makes refactoring the HTML much easier (imagine trying to refactor the HTML construction in the first two examples). We also get syntax highlighting for the HTML when in our IDE. And now our JavaScript has been reduced from 23 lines to 3 lines for this simple example (the savings can be much larger for more complex projects).

Checkout the jsFiddles for complete and working examples of the three usecases we presented here: vanilla JS, jQuery, and Handlebars.

Asset Bundling in Express

Wed, 29 Apr 2015 00:00:00 +0000

Find-A-Record is built on Express, a web framework for Node.js. Remarkably, Find-A-Record has no tests, no build system, and no asset bundling system. That’s okay for the static pages such as the home page, but the main web app relies on 15 JavaScript files and 3 CSS files in addition to the 3 JavaScript files and 3 CSS files included on every page in the site. It’s unwieldy.

Including so many static assets has undesirable consequences. First, it increases page load times because each asset needs to be downloaded separately.¹ Each request incurs the overhead of HTTP, TCP, and the limitations of the speed of light as it travels over the wire.

A cascading consequence of having so many assets is that we don’t want to add more. Therefore, the majority of the custom JavaScript is kept in one 1,500 line file. We don’t want to split it up because it just adds more asset files for downloading.

We need a way to bundle our assets. If the assets are bundled then we reduce the overhead of 24 total requests to 4: global CSS file, global JS file, page-specific CSS file, and page-specific JS file.

The main reason why we’ve been avoiding a build or bundle system is that it’s a big commitment; you have to buy into a specific workflow and hope it scales well into the future with your code base. Does it support minifying of JS and CSS? Does it have a development mode where you can include unbundled and unprocessed versions of your assets (very helpful for debugging)? Does it require bundled and processed files to be committed to your repository? Does bundling occur when the app is started? Does the app need to be restarted when source files are changed?

Since we’re using Express, middleware was our first thought but all we could find were BundleUp and Pound (which is built on BundleUp). We have successfully integrated bundle-up but we know it’s not a long-term solution since it’s not being maintained and its dependencies are way out of date.

What we like about BundleUp is that bundles are configured inside of the app itself. This makes it easier to figure which assets are included in which views and makes the development mode possible where assets are included directly instead of bundled. If you use an external bundling system and you want a development mode then bundles need to be configured both internally and externally which can be a mess.

One downside of BundleUp is that it runs when the app starts so it slows startup time. The documentation also leaves a lot to be desired. We noticed it was touching our source files and causing Nodemon to consistently restart the app. It turns out that it’s written to “compile” all assets, even if they’re not being processed (such as SASS or CoffeeScript) or minified. The default compile behavior for vanilla JS and CSS is to copy but our paths were configured such that it was writing over the originals with an exact copy.

We don’t like Bower because of it’s reliance on manifest files and how you can only exclude files in the manifest as opposed to marking only a few for inclusion. Besides, Bower is designed to handle and bundle external dependencies. All of the assets we want to bundle are already installed locally and committed to the repo. We just want to process local files and bundle them all together.

Browserify’s strong point is for bundling CommonJS modules in a way that enables them to be used in the browser. This allows you to right JavaScript code in Node (leveraging Node’s handy require syntax for modules) and then package it up for the front end. We use Browserify for other projects but that’s not our use case here.

Duo is a new kid on the block and fits our development style a little better, but again it’s likely much more than we need.

In the future we hope to create our own middleware for Express. All it needs to do is allow for bundles to be configured and then process and minify them. It can’t be too hard, and we can’t believe BundleUp is the only thing out there like it.

I can’t wait until HTTP2 is more ubiquitous. ↩