From zero to cloud - guide to setting up a dev process with the Amazon AWS Cloud pt. 1

Hi all,

once again I managed to get some time off to write.

There have been interesting things happening in my career as I had the opportunity to plan and execute an entire companies move from Rackspace to Amazon AWS with a very diverse app base with PHP Symfony 1, 1.4, 2.X apps, Perl (mod perl) apps and Java (Red5) apps, Magento stores, vendor specific tools like Active Collab.

It has taken me quite some time to figure out some problems as there is no real comprehensive thing out there to start easy with the Amazon AWS adventure and the Amazon guides are a bit dry.

To move you closer to the problem domain faced here, I will point out the steps that you need to take to have a complete application development cycle. Later I will hopefully manage to write a post about every step of the way, so it is much easier to understand the big picture, how things work together in an AWS based application.

I will not write about all the problems, that we had to overcome, because of the nature of the applications themselves, as the capacity for an app to be clustered is beyond the scope of this article. The most important rule however is - do not write anything on the server that is important to anyone, either upload it to Amazon S3, some attached webdav or the database (some db's have the option to save files like MongoDB's GridFS).

So the first usual step in the apps development is the actual development no surprises here.

1. Version control and simple collaboration (e.g. code reviews).

GIT and github.com to keep the code, collaborate and make it easy to pull from anywhere. I have actually pushed the company to move away from CVS, which wasn't very difficult as it is frustrating to work with it. Github has a fantastic interface and with the addition of their in-code search it is quite more practical than the dreadful search in Eclipse.

All you need is to register with github.com and follow simple instructions on getting started. We have a pro account with github, so we can keep our repositories private, for small teams you can however go to their competition - bitbucket.org, which has a pay-by-users model and lets you have free private repos.

2. Continuous Integration - making sure your code stays in good condition

Jenkins to run test and build. Well that is also one of the easy ones, there are some more sophisticated vendor solutions out there, but Jenkins with some extensions offers everything we needed. I added GIT, SVN, Google OpenID login, so that everyone from the organisation can log in and Apache Maven support.

The other cool thing Jenkins lets you do is have code statistics, like phpmd and copy/paste detection, code style checking and keeping the statistics for later. We also plan on making Jenkins directly fire up testing infrastructure in Amazon AWS via CloudFormation scripts so basically a file describing exactly your deployment and the relationships between machines. For the sake of simplicity and cost (which will come later in the post) just put Jenkins locally or on a traditional server.

3. Figuring out if everything belongs in the cloud

Now you have the means to code, collaborate and make it a really nice product so now you need to figure out which parts of your infrastructure should go into Amazon AWS.

As with everything, the cloud is only the answer to a very specific question, but certainly no to all of it. The true advantage of cloud computing isn't scaling up, as you can do that with normal infrastructures - it's scaling down in minutes. The advantage of being able to scale the number and size of instances is only relevant if you have fluctuating traffic. That means if the differences of usage during the day are quite higher than 1-2 servers of difference (in terms of computing power) than it's a classic case of lets-go-cloud.

There are some applications however, that are not really suited or would be missing the point of cloud computing. We have found that some applications simply don't belong there:

• Windows based stuff - it's just a question of cost. If  you don't need the scalability for windows boxes, it's far less expensive to put them with a traditional provider, like OVH
• All utility apps - everything you need to have running in single-server mode, so in other words there will never be need to make 2+ of them, but also making less than 1 would not be good. Every utility app, like Jenkins for instance, may or may not need additional nodes to work with, if it doesn't buy a normal server (or 2 if you like resiliency) and put it on there, make regular backups though. In our deployment Jenkins went onto a normal server.
• Everything requiring constantly needing hight computing power. Well every app that doesn't need to scale beyond one/two powerful servers and is constantly on-line is IMHO better off on a traditional server. 

What you have to take into account, that the AWS instances, even though even called "large" are nowhere near a £200 per month server, when it comes to computing power and usually costs the same or more. You get however the ability to run 1 tiny instance during the nights and 20 of them during heave days traffic which at the end might save you some major money.

Our traffic is not only fluctuating weekly, daily but is also strongly seasonal, as we are a ed-tech company. During the summer and all school breaks there is just need for a minimum infrastructure.

4. Moving domains to Route53

Route53 is Amazons DNS service. It offers a very nice interface for your dns administration and it's certainly easier to configure than bind9. This step of course isn't necessary, but it is good to move your domains so that you would have "one (key)ring to rule them all". As with AWS you get API access to almost everything so it is much easier to automate if everything is in one place, plus it's a lot tidier than having to log into somewhere else (or several locations) to control your domains.
We found a small Perl script that runs in CRON, that actually checks the instance tags of each instance for a specific tag and creates domains based on that name, so you can have a bit more simple to write name than the usual public DNS AWS instance domain names (e.g. ec2-54-228-109-17.eu-west-1.compute.amazonaws.com). In order for the script to work, you need to have the Amazon CLI tools installed.

This concludes the things you need to know to start getting familiar with AWS. In the next posts I will write about:

• Getting your team to have access to AWS services with IAM
• Managing some aspects of your deployments with the AWS Console
• Setting up and managing your workflows with Netflix's Asgard control panel
• Deploying your applications to your cloud servers with Capistrano
• Setting up and testing your auto scaling cloud deployments with jMeter and AWS CloudWatch

Hope you enjoyed reading, See you soon.

Amazon EC2 Instance inaccessible because of s3fs

Hi again!

Long time since my last post but always wanted to continue this blog - as it turns out some might be actually reading it ;)

During the time I was gone, I changed jobs a couple of times, even changed countries. I am now based in London, UK - a rather exciting city. Now working a lot with Amazon AWS and Java EE.

This blog post is about problems we had setting up auto scaling on Amazon AWS EC2 cloud. We were using a GUI front-end named Asgard (the home of the gods!) by Netflix trying to set up clusters and test them.

As we are moving more and more of our infrastructure into EC2, some problems become more and more apparent, especially the ability (or in our case - inability) of applications to work in clusters. There are some things that need to be considered, before creating a cluster of you application, which are not in the scope of this post.

Long story short, we needed to make some short cuts and instead of using Amazon S3 as it should - via its HTTP API, we went the s3fs way... it is a neat helper tool but can have nasty effects like memory leaks (saw it grab 50% of the machines RAM) and wild CPU usage... as it had in our case.

I was during a jMeter performance test of our two moved clusters, one is a PHP frontend app cluster and the other is a Perl service application cluster. Both of them started of as small instance clusters however the tests revealed that our Perl apps have a rather annoying problem with going really well and then just dying when overwhelmed. So we thought to go with medium instance sizes for the Perl cluster...

The medium instances hover did not start properly, they couldn't be ssh'd and the "get system log" option revealed that the setup didn't finish. The problematic thing about this was, that even though the small instances started without issue, the medium or large (we didn't try higher ones as it didn't seam there is a point) won't start. In the AWS console the instances passed all checks (which seam like a bit of shitty tests IMHO) but still there was zero movement.

After a few hours of trying different options to connect to the instances, we actually paid for Amazon's business level support - which is amazing. It costs you about 10% of your monthly fee or $100 whichever is higher, but assuming we would spend a week on figuring this out... well worth it. After about 30 minutes with one of their support staff, we finally found the problem.

We had a s3fs share mounted straight to /mnt and with no __netdev options (which basically means, that it need the network connection in otder to be mounted). After changing the mount point to some other folder and adding the fstab option, everything went back to normal and the instances

What we assumed is that the small instance was slow enough to start s3fs after it had started networking, therefore finishing the boot sequence correctly.

Hope this helps anyone stuck with the same problem. If an instance is not booting properly - look at fstab and use s3fs as a temporary option.

Cheers,
P

Symfony Dependency Injection - a really cool PHP tool

Hi!

Today I want to talk a little about a great design pattern called Dependency Injection and it's implementation by SensioLabs called simply Dependency Injection Component.

Now dependency injection is basically the inversion of control in you classes. It can help you to create decoupled structures that can be configured at run time which gives you a lot of flexibility. Let's take a look at an example:

Let's presume, you need to user a tool class inside some other class:

class User {


private $tool;


public function __construct(Tool $tool){
   $this->tool = $tool;
}


public function doStuff()   
   $this->tool->doSomeStuff();
}


As you can see, I've user the strategy pattern, so every time I want the USer to ->doStuff() it uses the ->doSomeStuff() method of the Tool class object.

Now this way is pretty neat, because it avoids  hard-coding the Tool into the User allowing for polymorphism. However creating a lot of such dependencies can be a pain if you have some more complicated stuff to do. Additionally there can be some problems with instantiating the Tool class before usage.

Symfony dependency injection can solve both those problems as it decouples elements and creates them upon usage, not upon configuration and helps to easily create complicated dependencies just before usage. Let's look a the example ripped out of my ZF project:


//retrieving mailer options from configuration

$mailerOptions = $this->getOption('mailer');


//we create a builder object, that will later build our classes
$serviceContainerBuilder = new sfServiceContainerBuilder();


//regitering the mailer service
//this means, that when we call the mailer service
//the Zend_Mail class has to be instantiated with parameters
//and a function called with some additional params


$serviceContainerBuilder->register('mailer', 'Zend_Mail')
->addMethodCall('setDefaultFrom', array($mailerOptions['from'], $mailerOptions['from_name']))
->addMethodCall('setDefaultReplyTo', array($mailerOptions['replyto'], $mailerOptions['replyto_name']))
->addMethodCall('setDefaultTransport', array('%mailer.transport%'));


//everything should be pretty straight forward
//I'm creating a builder instance and configuring it
//with the object class I wan't to have created
//and it's parameters
//the setDefaultTransport param is still open, when you add
//params like %param% sfDI knows that you want to set the param
//before service class instantiation


//setting to Zend_Registry to get the serviceContainer whenever I want
Zend_Registry::set('serviceContainer', $serviceContainerBuilder);



Some of you, who have some experience with dealing with Zend_Mail, know that you need to have a mailer transport of type Zend_Mail_Transport_Abstract.

In my application I have two bootstrapping (preparing basically) files:
1) the main application bootstrap file (from which it was I took the above example)
2) the PHPUnit bootstrap

The testing reasons I want to have a different transport in the second bootstrap that in the reall one. The first mailer should normally send the email but the second one should just log it in a file. Here the dependency injection copes in really handy:

1) Main bootstrap:



$serviceContainerBuilder = Zend_Registry::get('serviceContainer');
$mailerOptions = $this->getOption('mailer');


//I want the main transport to be a posrmarkapp.com using transport
$serviceContainerBuilder->register('mail.transport', 'FT_Mail_Transport_Postmark')
->addArgument($mailerOptions['transport']['api_key'])
->setShared(false);


//the setShared(false) simply tells the container that the mail.transport object isn't unique


//here is the interesting part
//I tell the Container to inject the newly configured container, as the 
//"mailer.transport" argument to the mail.transport container


$serviceContainerBuilder->setParameter('mailer.transport', new sfServiceReference('mail.transport'));



2) The second bootstrap (PHPUnit):



$serviceContainerBuilder = Zend_Registry::get('serviceContainer');
$serviceContainerBuilder->register('mail.transport', 'Zend_Mail_Transport_File')
->addArgument(array('path' => APPLICATION_PATH . '/..tests/log'))
->setShared(false);



This example shows just how you can create another transport for another purpose, just as easily.
Now we want to see it all in action:


//in some form processing command, we retrieve the container and create the service

$mailerService = \Zend_Registry::get('serviceContainer')->getService('mailer');


$tpl = //obviously we need some template


$mailerService->setBodyHtml($tpl->render());
$mailerService->addTo($emailTo);
$mailerService->send();



It's that simple, we can additionally set some parameters before getting the service, adding different behavior to the container.
Some of you might say that you can preconfigure the Zend_Mail class instance and set if directly into the registry. True, but remember that not all pages result in the usage of the mailer, so you would probably waste a lot of resources holding a lot if instantiated objects in the registry.

Hope you enjoyed the examples and start building your complicated class relations with the Symfony Dependency Injection Component. For more information go to http://components.symfony-project.org/dependency-injection/

cheers,
Peter