Amazon describes the new offering by stating, “AWS OpsWorks is a DevOps solution for managing applications of any scale or complexity on the AWS cloud.” Anyone proficient in Amazon would have used a similar statement to describe CloudFormation. Amazon is known for delivering very deliberate services that act as a member of their holistic framework. As a framework, their services leverage each other. For example, Elastic Beanstalk is built on top of CloudFormation, which in turn uses Simple Workflow Service as the coordination service. CloudFormation has the ability to instantiate virtually ALL of the Amazon services and also integrates with things like Simple Notification Service to alert users about events of interest. The Amazon framework has a long history of USING other services in the platform. That’s the beauty of it. Until now.

For the first time, Amazon introduced a service , OpsWorks, that has largely divorced itself of all knowledge of Amazon’s services. This service varies significantly from everything Amazon has delivered to date. The apparent design of the new service is to not use the existing services and instead to use open source libraries. For example, where CloudFormation has direct integration into the AWS native monitoring service (CloudWatch), OpsWorks instead uses Ganglia. CloudFormation has native integration into Elastic Load Balancers but OpsWorks describes integrations into HAProxy. And the list goes on… if you were to use OpsWorks to launch MySQL it doesn’t use RDS and you don’t get the new metrics, cross-cloud replication, auto-backup, etc. At first glance, this architecture might appear attractive but it’s important to note that Amazon has made HUGE improvements over the open source libraries. For example, ELB isn’t just a load balancer but it’s a horizontally scalable load balancer that even the best DDOS would have a hard time taking out.

With OpsWorks, Amazon has managed to create a non-open source, proprietary wrapper around the very popular Chef platform. It fails to use their own native services and also fails to be portable across clouds. This begs the obvious questions:

1. Why would we use OpsWorks over straight Chef (or Puppet)?

>> Our answer: We see no reason other than it might appear more convenient for a newbie.

2. Why would we use OpsWorks over CloudFormation?

>> Our answer: CloudFormation can be a pain. It’s designed for the advanced user. 3rd parties (like Transcend Computing) are making it more convenient but because CloudFormation is so wide in scope, users may find it daunting.

In summary, Amazon has release a proprietary, service oriented version of chef that lacks portability, fails to use their own native service and duplicates capabilities already in their platform. To add to the (lack) of the offering, the service is not available in GovCloud (nor should we expect it any time soon) and it does not support any of the modern and popular AWS features like VPC and Provisioned IOPS.

I have tremendous respect for the architects at Amazon. This service is a complete departure from everything they have done to date and in its current state, makes no sense. I’m hopeful that over time, Amazon gives a more detailed explanation of why they introduced this service and where they hope to take it. In its current state, this service has such remarkable shortcomings that it’s hard not to forecast an unfavorable outcome.

Cloud Monitoring

The Cloud Monitoring service provides constant visibility into the health of compute instances. Vital information such as CPU activity, memory consumption and network activity are captured. The data can be viewed over time ranges (e.g., last week, last hour, etc.) and with specific time increments (e.g., in 15 periods).

Alarms

Users also have access to the alarms that they’re familiar with in the AWS CloudWatch service. This allows operators to set automated routines to check for boundary conditions. Alarms that are triggered can perform a pre-determined action such as notifying personnel or running a backup routine.

Auto-Scaling

The TopStack Auto Scale service provides an elasticity capability to your cloud environment. When connected to the Cloud Monitoring Service, Auto Scale detects changes in utilization levels and increases or decreases the number of instances used. By automating the elasticity of the compute tier, systems can scale as needed to handle traffic bursts and release the resources when they’re no longer needed.

Auto-scaling can be configured to user-defined thresholds. Triggers can occur on a variety of signals (CPU, I/O, etc.) as measured over some period of time. Typically users scale the instance count by +1 or -1 but for larger systems there is an option to scale more instances in parallel. Advanced features include the automated joining to Elastic Load Balancers and the use of “cool down periods” to ensure that instances aren’t accidentally launched.

What’s next?

The TopStack family of AWS compatible services has focused on “platform services” that developers would take advantage of directly. We’ll continue to release new services (yes, a CloudFormation compatible offering is around the corner) but we’ll also be focused on delivering improved “support services”  for aggregated logging, registry/directory and configuration controls. To learn more about TopStack, take a look at the features.

In the cloud, virtual servers are constantly being started and stopped. This is a result of the dynamic and elastic nature of infrastructure-as-a-service. A side effect of this process is that it’s easy to lose track of the machines because their IP addresses keep changing. DNS 53 enables users to program changes to their DNS records:

• Like AWS, we offer a Hosted Zone (think DNS zone file)
• The SOA (Start of Authority) record and optional records such as:
  • A Record (IPV4 address)
  • AAAA (IPV6 address)
  • CNAME (canonical name)
  • MX (mail exchange)
  • NS (name server)

The DNS 53 service provides a RESTful interface to your DNS system:

Hosted Zone Actions

• POST CreateHostedZone
• GET GetHostedZone
• DELETE DeleteHostedZone
• GET ListHostedZones

Resource Record Set Actions

• POST ChangeResourceRecordSets
• GET ListResourceRecordSets
• GET GetChange

Why is DNS 53 Important?

The DNS 53 service is used by other services such as the Auto Scaling Service and the Orchestration service to automatically adjust host names as they come online. As your cloud grows in size, DNS 53 is an essential component for making your cloud manageable.  It marks another step in our on-going commitment to providing AWS compatible services available in your own data center, preferred cloud provider – or even your laptop.

Manage Via StackStudio

Like our other TopStack services, you can manage the service via standard RESTful API calls or by using our multi-cloud management UI, StackStudio:

DNS 53 is pre-integrated to the leading DNS server, BIND9, but can be extended to support other DNS servers.

To learn more about the offering, check out the documentation.

The TopStack Family

Remember, DNS 53 is only the latest entry into a growing suite of as-a-Service solutions including clean-room implementations of Relational Data Service, Elastic Beanstalk, ElastiCache, Elastic Load Balancer and Simple Queue Service. Transcend is committed to providing our customers portability and choice for their cloud workloads. Look for more exciting announcements in the near future!

The release of the services coincides with Amazon’s own update on the service. Just last week Amazon announced new capabilities for ‘long polling and batch requests. These features give developers more flexibility in how they interact with the queue. Long polling allows developers to avoid putting tight loops in the code that would block the process. Instead, the developer specifies a ‘watch period’ that can be up to 20 seconds. As soon as a message is available in the queue it responds back to the client. The polling attribute can be changed by calling the SetQueueAttributes command. By batching queued items, the caller can increase performance by return up to ten items in a single request.

All of the TopStack services can be managed by our multi-cloud UI, StackStudio. To manage TopStack SQS, just set up a new endpoint URL and use it the same as you would with Amazon.

The Scalable Queue Service runs on the leading cloud platforms: OpenStack and Eucalyptus. We’re currently supporting Eucalyptus version 3.1 and OpenStack Essex. Our integration tests are showing all green lights for OpenStack Folsom but we’re not planning on having official support for Folsom for another couple of months.

TopStack SQS was designed to run in a cloud friendly scale-out mode. From an architectural perspective, the solution is designed with no single master of the queue data. Instead, it leverages a peer-to-peer design that increases scalability and reduces single points of failure. Distribution of the queued data can be handled by either TCP or IP multicast, depending on your networking constraints.

Commands supported include:

• AddPermission
• ChangeMessageVisibility
• ChangeMessageVisibilityBatch
• CreateQueue
• DeleteMessage
• DeleteMessageBatch
• DeleteQueue
• GetQueueAttributes
• GetQueueUrl
• ListQueues
• ReceiveMessage
• RemovePermission
• SendMessage
• SendMessageBatch
• SetQueueAttributes

For more information on the compatible Amazon offering, see: Simple Queue Service.

What is Elastic Cache?

TopStack Elastic Cache Service is a clean-room implementation of the Amazon ElastiCache service. It can be installed in private clouds or at your favorite cloud provider. The service is API compatible with the popular Amazon offering, ElastiCache. The service allows users to quickly create in-memory, distributed caches using the Memcached protocol. It provides a convenient way to manage the clusters including the number of nodes, version of Memcached, ports and the sizes of the instances:

You can also configure common parameters and apply ingress security settings:

What is Memcached?

“Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.”  It was originally developed by Brad Fitzpatrick in 2003 for increasing performance of LiveJournal. The open source project was later extended by dozens of contributors including commercial support for persistent capabilities from the team at Membase, which later merged with CouchOne to form CouchBase.

How does Elastic Cache work?

Elastic Cache uses the infrastructure in your cloud to manage clusters of linked memcached instances. The service enables users to quickly create, modify and delete multi-node clusters, change their configurations and attach security policies. Key commands include:

• AuthorizeCacheSecurityGroupIngress, RevokeCacheSecurityGroupIngress
• CreateCacheCluster, DeleteCacheCluster, ModifyCacheCluster, RebootCacheCluster
• CreateCacheParameterGroup, DeleteCacheParameterGroup,
• ModifyCacheParameterGroup, ResetCacheParameterGroup
• CreateCacheSecurityGroup, DeleteCacheSecurityGroup, DescribeCacheSecurityGroups
• DescribeCacheParameters, DescribeEngineDefaultParameters, DescribeEvents

For full user documentation, see: docs.TranscendComputing.com

Where does it fit in the architecture?

The two most common uses of Elastic Cache are to cache data normally stored in a database or the caching of static web objects. Another common scenario is to use it as a means to share information between nodes. For example, many systems use memcached as a temporary storage area for session tokens between servers.

The diagram shows a fairly standard Web app. It uses an Elastic Load Balancer, a PHP server and a relational database. In addition, the PHP server is connected to the Elastic Cache. The application will reach out to the cache cluster as a fast (and cheap) way of managing data. Calls to the database can happen either in conjunction with the calls to the cache, or after a call to the DB where a miss occurred.

How can I get Elastic Cache?

Transcend Computing has made the Elastic Cache Service available for download as part of the TopStack offering.

Download here.

What’s in the download?

The download includes both StackStudio and two of our core TopStack services: Load Balancing and Relational Database (for MySQL).

What cloud platforms will TopStack run on?

These solutions are validated on OpenStack Essex and Eucalyptus 3.1.

What about CloudStack, OpenStack Folsom, …?

Our development and validations are prioritized by customer demand. If you have a need for another cloud platform or version, please let us know.

What is StackStudio?

StackStudio is our UI to manage multiple clouds. It’s an excellent solution for managing AWS, OpenStack, Eucalyptus and TopStack.

Will your other service be available for download?

Yes. Shortly, (this year) we’ll allow you to download our clean-room implementation of the AWS ElastiCache, Simple Queue Service and Elastic Beanstalk (Tomcat edition). Expect additional services in 2013.

Not picture are Raj Arora and Jeff Schneider

Hey Michael -what’s up with the Cloud Foundry t-shirt??

What do you think?

New Features

Version Control Entire Topologies

Users can now version control their entire topology and have multiple versions available in different environments. For example, you might have your version 2.0 product running in production while the QA department is busy testing version 2.1 and the developers are working on new features for a 2.2 release.

Manage Configuration Drift Across Dev/Test/Stage/Prod

After a version of a topology is stamped, it’s common for that version to have slightly different characteristics in each environment stage. For example, when a template is being worked on in Development, the scripts to fire alarms and page operators will most likely be turned off. In production, it’ll be turned on. You’ll see similar examples with backup/recover scenarios (e.g., don’t backup in Dev/Test) and in scalability (e.g., Dev doesn’t need 20 servers running the app tier but Prod might.)

To facilitate these minor changes between versions we’ve introduced the ability to create a Baseline that represents the common features and attributes of a topology. The baseline is then extended (think inheritance) to capture the minor variations introduced at each stage. When a user makes a change to a template they have the option of putting it in Baseline or specifically in one of the templates that extend Baseline.

Stack-in-Stack Composition

Embeddable stacks is a killer feature for organizations that want to share common architectures and platforms. A template can be marked as a shared resource allowing others to embed that stack in their project. For example, one team might create a shared template for auto-scaling their favorite NoSQL database. The template might contain directives on backing it up, patching it, notifying people if it were to go down, identifying capacity considerations and so on. Rather than having each team reinvent this solution, StackStudio allows users to have ‘pointers’ to shared resources. This significantly reduces the amount of time to create an application and increases the consistency in the platform topologies being used.

New CloudFormation Orchestration Resources

We’re continuing to add support for new resources to stay up to date with CloudFormation. The new elements that can now be placed in a CloudFormation topology include:

• CloudWatch Alarms
• CloudFront Distribution
• IAM Users and Groups (Identity & Access Management)
• Simple DB
• Embedded Stacks

Support for Eucalyptus & Piston Enterprise OS (OpenStack Core)

Users now have choice in their IaaS cloud stack! StackStudio can be used to manage either a Piston Enterprise OS cloud or a Eucalyptus cloud.

• Compute Service
• Block Storage Service
• Simple Storage Service
• Networking Service
• Identity and Access (Piston: Keystone, Euca: IAM)

Transcend Computing’s TopStack Web Edition On Eucalyptus or Piston Enterprise OS

TopStack Web Edition provides a set of Platform Services that are compatible with the Amazon PaaS offering. TopStack can be deployed in your private cloud running either Piston Enterprise OS or Eucalyptus. StackStudio can manage platform services for:

• TopStack Relational Database Service (ACE for Amazon’s Relational Database Service)
• TopStack Load Balancing Service (ACE for Amazon’s Elastic Load Balancer)
• TopStack Container Service (ACE for Amazon’s Elastic Beanstalk)
• TopStack Caching Service (ACE for Amazon’s ElastiCache)