It is a thought experiment to help you determine what are the most valuable pieces of code that you will write, and perhaps achieve some enlightenment on the nature of software. The experiment can be done in your head or with a pen & paper:

Review the existing application, and find all of the parts that you think are re-usable in your current effort. Perhaps there will be just a few, perhaps there will be none, perhaps there will be a bunch.

Consider each of the re-usable parts and answer the questions:

• why is this re-usable?
• what would have to change in the current effort to make this not re-usable?

Now, skip forward in time and imagine you have successfully completed re-writing the parts that you decide to re-write. It was a wonderful success, and the application has grown and expanded in wonderful ways.

Technologies have changed, it is 10 years in the future, and it is time for the next re-write. Perform the experiment again, and write down the answers that you will want to have this time.

The point of the experiment is to highlight that there is very little value intrinsically stored in source code. The only things that have long-term value are abstractions and standards.

Standards (html, css) are valuable only as long as the technology survives. They are valuable because everyone is dependent on them, and it simply costs too much to abandon them completely.

A sub-category of standards are vendor-specific technologies. Think VB6 forms, .NET forms, ASP or PHP or XAML. They still have value, but they come with the cost of a technology tie-in – they are only valuable as long as the technology lives.

Many standards are a form of debt. You accept that you will get a lower cost of development today, but that the application will only be able to be maintained and grow as long as the technology or standard is viable. It is often a very good type of debt, because there is a balloon payment at the end that you will never have to make (because the application will die before it is necessary).

Abstractions are longer-lived, because they can represent some fundamental domain knowledge. They are valuable as long as the domain and your assumptions do not change.

Dogfooding

http://en.wikipedia.org/wiki/Eating_your_own_dog_food.  The week was very valuable to me in that I discovered several bugs in the node-perfectapi package.  I also discovered some new features that would be helpful, so that was great because it will help me make the product better.

Design

My design had a couple of flaws.  That’s not so bad really – in my experience at least, a design seldom survives coding without some changes.  Luckily they were not major flaws –  I needed one more component to better manage the load on user-facing endpoints, and I needed to add something to support configuration of the deployed services.

Usability

Although the pieces are all working together, it is not yet a usable solution.  For one thing, there is only a publish function, no way to update or remove packages.  For another, you really need to be able to configure services for them to work.

Learning new things

This was one of the major gains for me – I learned all about Redis.  It was not pretty.  The node-redis module I used works really great, but it is at a very low level.  Coding against Redis feels a but like coding in assembly language.  Concatenating strings to form keys feels wrong, as does the process of having to create my own indexes, and do my own serializing and deserializing of objects.  There is definitely a market for modules that take the Redis experience and add a layer of abstraction in order to lower the workload.

I also discovered the async module.  Without it, coding against Redis in Node.js is next to impossible.  Really, it is ridiculously hard.  Painful like hard exercise (but without the health  benefits).

Another discovery was testing with mocha and should.  This was also a little frustrating at times, because one page (albeit a long page) of mocha documentation really does not cut it.  However once I got the hang of it, it was nice to have some tests running.

Next Steps

I’ve invested enough in this that I think I will continue to improve it until it is usable for my purposes.  Not in one week though

I’m also working on creating an AMI image for Nodester.  While I rejected Nodester for my own purposes (it does not have the features I wanted), I still would like to contribute a little to that project.

Well Friday is done, and it was one of those days where a stupid bug eats hours of the day.  Pushed through it though (went to lunch – my favorite technique)

I failed to get everything done that I wanted to in 5 days, but at the same time I have met all of my original goals, except for an Amazon AMI image for hosting the front-end proxy.

The Good

All of the components are now functioning together.  The code is split into the following:

• Service registry – a single service which is responsible for maintaining all of the information about the installed services
• Machine proxy – a self configuring reverse proxy running on each host machine
• Machine host – a self configuring service host running on each host machine.  When it finds an eligible service, it installs it.  Also doubles as a command line interface to register applications (when run on a client machine)
• Front-end proxy – a pre-configured pound reverse proxy running in front of the machines hosts, directing traffic and terminating the SSL endpoint (https).

I was able to use the CLI to register multiple apps, and the machine host and proxy services picked them up, installed them and exposed them correctly via the reverse proxy.  That is awesome!  Its just the happy path, but it makes me happy

I have an Amazon AMI ready for the host machine.  It is ami-114b9b78.  Login as user `ubuntu`.  There is a readme.txt in the home folder with further instructions to configure the services.  But its not much use without a server running the machine registry

The Bad

I realized that the frontend proxy needs to be self-configuring too, in order to direct the traffic to the correct machine(s).  So I’ll have to write another component.  The good part is that it can re-use a lot of the logic of the machine proxy code.

I also realized that I am not handling configuration for the services, which is an essential piece.  For example, if I install a service that depends on another, then I need to configure the endpoint for that other.  Perfectapi allows for service configuration through environment variables, but the service registry and the machine host do not yet have a way to make use of that technique.  I’ll have to add some code for that.

The Ugly

The code is not production ready.   It handles the happy path ok, but needs more work to be able to recover from problems (e.g. when a service fails to install).

Thursday is over, the output of which is at https://github.com/perfectapi/node-paas-machine-host.  The purpose of the machine-host is to wait around until it sees that something needs hosting, and then host it.  It also takes care of the main command-line interface, providing the ability to publish an app from the command line.

The biggest challenge yesterday was that I had originally designed the upload of a packages files as a “push”, i.e. zip up you package and publish it.  Unfortunately that causes a number of issues, the main one being that Node.js doesn’t yet have a nice unzip component.  I looked at a lot of possible packages that could handle it, but their APIs were very low level and I don’t have time to deal with that complexity.   Also, the zip files turn out to be quite big, which slows things down.  Finally, its not that friendly to make people zip it up.

So…I switched to a “pull” model, where the host will pull the files out of Git.  That limits us currently to publishing publicly accessible packages stored on github or similar host, but hat can change later.

Another challenge was something that I ave struggled with every time I try to use it – Nodes support for spawning processes.  There are 3-4 different ways to spawn a process, but the problem is that for any given scenario, only one of them works properly.  So I try `spawn`, then I debug it because it is not working, then finally I try `exec` and it works right away.  Aargh.  Not  good.

Anyway, enough writing this blog; I still have a lot to do if I want to get something working by the end of the day…

• Writing a PaaS (using Node.js) in 1 week – Monday
• Writing a PaaS (using Node.js) in 1 week – Tuesday

Well Wednesday is done.  The reverse proxy is written, code at https://github.com/perfectapi/node-paas-machine-proxy.

As a reminder, the purpose of the reverse proxy was to provide a single IP + host endpoint to each machine (host) on which multiple services are running.  We can run multiple services on that host, or multiple instances of the same service.  The reverse proxy provides routing and round-robin load balancing between the services.  Each service runs on a unique path, e.g. http://host/service1, http://host/service2.

A higher level proxy will provide routing between machines and SSL termination, providing endpoints like https://mydomain.com/service1.

The reverse proxy must also automatically update its configuration when it detects a change in the Service Registry, and expose load information for the host (so that we can know when it is over capacity).

How did it go?

I had some conflict on whether to use one of the node.js reverse proxies (bouncy, node-http-proxy) for the routing, or to use a more mature Linux package.  In the end I decided to go with haproxy, mainly because it has detection of when a particular endpoint is not responding, and can switch that endpoint off.  This should prove useful for when problems occur, or when services are being upgraded or scaled down.

It also supports hot-switching of the proxy configuration, which is awesome.

Once the decision was made, configuring haproxy was not very hard.  I had to take the time to read its configuration manual (a gigantic text file, argh).  Ok, I just skimmed it

Testing

I did less testing on this component than I did on the service registry.  I think its one of those things where you can only find the issues once you start loading up services and seeing what happens.

I did do enough testing to know that the code does what I want – its just a matter of figuring out if what I want will work in practice.  So I’ll do that on Thursday/Friday, and hopefully it will work itself out.

Determining Load

Determining the load was fairly easy.  I use a combination of the node-provided

os.loadAvg()

method, and calling

ps -A u

to find the process using the most CPU percentage.

Well I’m going to call Tuesday done. The Service Registry is written and installed on my server. Source code is at https://github.com/perfectapi/node-paas-registry.  As with all perfectapi APIs, there is an automatic test app page.  I’ve exposed this one to the Internet, at https://services.perfectapi.com/paas/registry/testapp/

So I’m on target, but it was a rough day.  My first challenge was Redis.  I’ve not used it before, and there was a learning curve involved before I could effectively use it.  I used the node-redis package to access it, and that package worked well.  It was just hard, trying to learn all of the commands, and when to use sets or hashes or whatever.  That ate a lot of time.

My next challenge was the async nature of the node-redis client.  Its fine if you’re just doing one thing, but I had a scenario that went like this:

• open a connection to the database
• get the set of instance ids by service name (via callback)
• for each instance id, get the full instance object (via callback), and if it matches some additional criteria then do something with it and stop processing
• close the connection to the database

The problem was that this is callback hell.  I couldn’t even figure out where to close the connection.  A little research found me the async package, which was a lifesaver.  Once I started using that, it became very natural to solve the problem.

Finally, I had some problems with perfectapi  Required parameters are supposed to validate, but they do not seem to be doing so.  I’ll have to take a look at that separately.

Anyway, lets hope tomorrow goes smoother…

• The week is over.  There is a retrospective post here.
• I have posted Tuesdays notes here, Wednesday’s notes here, Thursday’s notes here, and Friday’s notes here.

I have decided to write a simple PaaS (Platform as a Service), in order to demonstrate the capabilities of my perfectapi API framework for Node.js.   I don’t want to compete with other PaaS offerings – this is just a need that my company has – to demonstrate how to simply and easily self-host and scale services developed with perfectapi.

Doing it in one week is a challenge I put to myself, because I have a limited amount of time to devote to this, and because doing it in one week (5 days) can drive a little PR for perfectapi.

So what exactly will I be building?  The platform will have the following qualities:

• Ability to deploy multiple Node.js perfectapi-based services, and load balance multiple instances of those services
• Each PaaS platform supports a single domain endpoint – in my case it will be services.perfectapi.com
• PaaS service endpoints accessible via JavaScript, Node.js, C#, REST and command line, built using perfectapi
• Hosted service endpoints accessible via JavaScript, Node.js, C#, REST and command line, built using perfectapi
• Provide the groundwork to automatically scale out as load increases (actual feature of automatically scaling out is not in scope for this week, but manually scaling out will be possible)
• Services all run behind SSL
• Complete code and Amazon AMI images (Ubuntu 11.10)  to be provided, so that others can reproduce the PaaS environment easily
• Written in Node.js and using freely available Linux tools where necessary
• No security to speak of (limit access using firewalls or shared secrets)

Day 1 (Monday) – Design

I spent most of Monday in Caribou coffee, researching and designing on my iPad.  Oh, the life I lead

The design I have consists of several components.  At the core is the Service Registry, which stores a list of the services installed in my domain.

Day 2 (Tuesday) - Service Registry

The service registry maintains a record of detail for each service instance, including the unique service name, and the path on which it will host.  So for example, the following:

• “Payment Portal”, /payments, attributes,  files
• “Payment Portal”, /payments, attributes, files
• “Mailer”, /email, files

…represents 3 service instances.  The first two are instances of the same service, and host on the same “/payments” path.  The 3rd is another service.  Each instance has its own copy of the files, and has a set of attributes.

In my “services.perfectapi.com” domain, the records above would represent 2 endpoints:

• https://services.perfectapi.com/payments – load balanced across 2 instances
• https://services.perfectapi.com/email – pointing at a single instance

The Service Registry does not create these services, or manage their endpoints, or anything really other than record their existence.   It will have the following API commands and queries:

• RegisterInstance(service name, path, attributes, files) – creates a new record of a service instance
• UnregisterInstance(service name, path, [host], [port]) – removes a single matching existing record of a service instance
• GetServiceInfo(service name) – returns array of host, path, port, attributes, files
• ListServices – lists service names
• ListUnclaimedServices – lists service names which do not yet have a port and host specified.  May list the same service name multiple times if their are multiple matches.
• ClaimService(service name, path, host, port, attributes) – updates an unclaimed service instance with new details.  If there are multiple unclaimed instances, then only one will be updated.  If there are no matches, returns an error.

In version 1, the Service Registry will probably store its data in Redis.  In later versions I would like if it could use DNS-SD.

Day 3 (Wednesday) - Reverse Proxy

The Service Registry was a single service for a domain.  The Reverse Proxy is a single service per machine.  The Reverse Proxy monitors the Service Registry, and ensures that all instances on the current machine have a route from outside.  So for example, on the current host we may have the following in the service registry:

• “Payment Portal”, /payments, port 4001, host ABC
• “Payment Portal”, /payments, port 4002, host ABC
• “Mailer”, /email, port 4003, host ABC

The Reverse Proxy ensures that all services are accessible on port 80, i.e.

• http://abc/payments – load balanced across 2 instances
• http://abc/email – pointing at a single instance

It also monitors the Service Registry for changes, and when it finds that there is a discrepancy, it updates the proxy configuration to create or remove a route.

In addition to routing traffic, the reverse proxy exposes an endpoint with load information for the machine it is installed on.  It exposes two methods:

• GetLoad – returns a number indicating current 5 minute load average.  1.0 or higher indicates that the server load is at or above capacity
• GetCulprit – returns the name of the service instance that is most likely the culprit, i.e. the instance that is using the most CPU.  (I/O ignored because Node.js processes should not be I/O intense, and network ignored because it is unlikely to be a limitation in a cloud environment).

Day 4 & 5 (Thursday & Friday) - Hosting Server

Like the Reverse Proxy, the Hosting Server runs a single instance per machine.  It contains the bulk of the PaaS functionality.  Specifically, it is responsible for:

• installing new app instances, on request (copying files, launching the process, ensuring it remains running).
• removing app instances, on request
• updating App registry when apps are added or removed

The expected API methods are:

• PublishApp – publishes a new app
• DeleteApp – deletes an existing app
• UpdateApp – combination of a delete and a publish (publish new, then delete old)
• GetAppDetails – returns the attributes of an app
• ListApps – lists all apps
• ListAppInstances – lists the instances for an app

The Publish workflow goes something like this:

• Write new instance(s) to app registry (with empty host and port)
• Poll App Registry every 1 minute
• If an unclaimed service instance is found, and this server meets to service instance requirements, and this server’s load is low enough, then claim it (write host and port to Service Registry).

The definition of “ this server’s load is low enough” is that a call to the Unix “uptime” command returns a 5 minute load average number less than 1.0 per CPU of the system.  This criteria should work on both Amazon EC2 Instances and Rackspace Cloud Servers.

Day 5 (Friday) – SSL Termination & Load Balancing

Another Reverse Proxy.  This one is a Linux package called “pound”, which can do SSL termination (host the https) as well as load balancing amongst our servers.  It works off of a static configuration file.  The file will not need to change as long as the list of servers remains static.

Auto Scaling Server (Future)

The part I will not build this week.  In order to scale out, we need to

• detect the need to scale – using GetLoad API call on the Reverse Proxies.
• determine which service needs scaling – using GetCulprit API call on the overloaded Reverse Proxy
• update the Service Registry with new instances

That takes care of scaling out within the existing machines in our cloud.  Once we exhaust the capacity of the existing machines, we have to start using 3rd party services, such as SCALR or Amazon’s Auto Scaling.  In that case, it probably makes sense to move the SSL termination and Load Balancing function off of our simple “pound” package.

The demo page showcases the following PerfectAPI features:

Self-configuring endpoint

Simply add a script reference to the provided javascript file in your HTML and then you can directly call the exposed methods of that API without further knowledge of urls

Simple rpc-style calls

Calls to the api are simple asynchronous calls in the rpc style (see example code on the demo page). There is no Ajax, JSON, JSONP, REST to worry about – its just simple Javascript code

Long-running API calls

Generating an image can take a while. Normally, this would timeout after 15-30 seconds, however this does not happen with PerfectAPI.

Test Page

The demo page is a custom page, but there is also an automatically generated “test” page for the API, where you can explore the API and see actual code in the language of your choice.

My business model is the common open source one, i.e. make a product that is free & open source and charge for additional services.

The page has a signup for the “public beta”, which is really just a chance for someone that is interested to get in on the ground floor with free support and services.

PerfectAPI is both the name of the business and the name of the product, which can be a little confusing.  The product is a set of tools that facilitate the development of APIs, and the meshing of different APIs together in the cloud in such a way that they form a complete service offering.

Some of my competitors are apigee.com, hook.io (by nodejitsu).  Both of these companies offer a way of building a service offering using APIs, but they have a different focus than each other and myself.

My own focus is based on a vision of interconnected, discoverable services on the Internet, done very simply.  I know what my next steps are and I’m moving forward with that.

One of those next steps is to try to drive involvement and usage in the Node.js community.  I think my Node.js offering is still a beta product, but nevertheless is very compelling for someone wanting to create a service API on the Internet.

Another piece I have to get working soon is authorization, i.e. OAUTH, OpenID, etc.

Anyway, if you have any questions, please comment on the website itself or leave a comment below.

The Costs

First off, on a 3 year hosting plan at Godaddy, I was paying about $3.20 per month, which is very very cheap.   You cannot get that price directly, but I had some coupons.  A more normal cost for Godaddy is about $5.00 per month.   My final costs for my new hosting solution on EC2 are:

• First year: $0 per month.
• Next 3 years: $ 6.43 per month, plus bandwidth (mostly free, unless I hit the front page of Reddit).

This is not as good as Godaddy, but it is a very acceptable rate to me (given I have complete root access to the machine and can do anything I want with it).

Below is the analysis of the EC2 costs.  For reference, here is a link to the EC2 pricing sheet.  My costs are for the us-east Amazon region – costs in other regions are different.

• First year:  Amazon gives away one t1.micro instance plus bandwidth to new AWS customers for the first year.  This is why the first year is free.
• Next 3 years:  $100 for 3 years of “Heavy Utilization Reserved Instance” (t1.micro), plus $0.005 per hour ($43.80 per year), do the math and that comes out to $6.43 per month.

Actually doing the move – the easy way

[UPDATE: I have posted a short YouTube video on how to do this]

The first step was to go into my existing WordPress blog and do a complete export of my posts and pages (Tools/Export/All Content).  With this downloaded to my local machine, I felt ready to get going.

WordPress boasts a “famous 5 minute install”, but don’t be misled by that – it can be done, but that depends highly on your Linux skills.  The PerfectAPI vision is to make things simpler, so I have for you the following, with all my experience of doing the move baked in.  Skip to “the hard way” further down in this post if you like it that way.

• ami–e5e6328c – a us-east Amazon EC2 Ubuntu 11.10 EBS image that is set to go with what you need.  Assuming you are already signed up and have some experiencing using ssh with Linux machines on EC2, this should give you something close to the 5 minute install.

To start with the image above, first launch it into a t1.micro instance.  Be sure to use a security group that allows port 80 (http) access, and one that allows SSH access (port 22).  Once the image is running,  ssh into it using a tool like Putty.  A reasonable & quick guide to using Putty for this purpose can be found here.  Note that the login name for the instance above is “ubuntu“, not “root”.

Once logged in, install WordPress by executing the command:

sudo ./install.sh

(This is my custom install script).  Installing in this way ensures your instance has its own unique passwords for mysql and wordpress (not the same as everyone else using the above image).

Once the install completes (10 seconds), go the the AWS console and set yourself up an “Elastic IP”.  (An elastic IP is just a public IP address that you can point your domain at).  Associate your new Elastic IP with your new instance.    Then, go to your blog at:

http://your-elastic-ip/

…and complete the WordPress setup.  Use Tools/Import/Wordpress to re-import all of your posts and pages.  Setup your theme, play with your widgets, install some plugins.   Do not change the url of the blog in your General Settings until you are ready to switch over.  Failing to heed this advice will make the new blog redirect to the old whenever you login.  Very annoying and difficult to change back.

Final step is to switch your domain records – use whatever tools your DNS provider has to point your domain at your elastic IP address.  After that, you can set the correct URL in the General Settings of the new blog.   It can take a while before DNS changes kick in – if you’re impatient, you can temporarily edit your hosts file to see the changes early.

Put a note on your calendar for 1 year after your Amazon AWS signup date, to purchase yourself a reserved instance, so that the lower pricing kicks in.

Doing the move – the hard way

The first trick for getting any new instance up on Amazon is to find a base AMI image that you like.  I like the Ubuntu images  at Alestic, so that is where I started.  After creating an image with an instance, you have a way to go before you can even get started on WordPress, for example you need to install a LAMP stack (apache, mysql, php), you want to ensure the instance stays up-to-date with linux security patches, etc.  Anyway, I created scripts to be able to do all of this in a repeatable way, and they can be found on my amigenerator project on Github.

If you need, here is a direct link to the scripts.  The scripts only work when run from the ami-generator tool, so looking at them on Github is mostly for education.  The ami-generator tool itself is still in alpha stage, so I am not going to include instructions on how to install/use it here.

My advice – just do it the easy way instead.  (But if you do have suggestions on how to improve the scripts, please do let me know).