[ANN] Stratosphere: AWS CloudFormation EDSL in Haskell

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[ANN] Stratosphere: AWS CloudFormation EDSL in Haskell

David Reaver

CloudFormation is a system that provisions and updates Amazon Web Services
(AWS) resources based on declarative templates. Common criticisms of
CloudFormation include the use of JSON as the template language and limited
error-checking, often only available in the form of run-time errors and stack
rollbacks. By wrapping templates in Haskell, we are able to easily construct
them and help ensure correctness.

The goals of stratosphere are to:
- Build a Haskell EDSL to specify CloudFormation templates. Since it is
  embedded in Haskell, it is type-checked and generally much easier to work
  with than raw JSON.
- Have a simple checking/linting system outside of the types that can find
  common errors in templates.
- Be able to also read valid CloudFormation JSON templates so they can be
  type-checked. This also gives us free integration tests by using the huge
  amount of example templates available in the AWS docs.

Most of the commonly used CloudFormation resources are implemented, and adding
new resources is very straightforward. (We created a web scraper that generates
a JSON file from a given CloudFormation documentation page URL, and from that
we generate a Haskell module.) So far, we have implemented resources we use at
Front Row Education, and we will add more resources over time.

The library is very much in a usable state as-is. However, we want to make any
sweeping changes while it is still young. If you have any suggestions at all,
you want us to add your favorite resource, or if you find a bug, please file an
issue on Github!

Also, we want to give a huge thanks to Brendan Hay, the author of amazonka, for
his ideas and feedback on the project.

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Stratosphere: AWS CloudFormation EDSL in Haskell

Alberto G. Corona
Very nice job.

I have tried AWS EC2 and I failed miserably since the permission system and his intricate configurations, his paranoid security, and  their secret formulas for pricing are beyond an average human intelligence.

It is possible to have an example stratosphere application that could configure and run a Haskell program in one or many EC2 nodes , close the instances and send back the results?

Are there  some elements still lacking to achieve such goal?

2016-04-19 20:56 GMT+02:00 David Reaver <[hidden email]>:

CloudFormation is a system that provisions and updates Amazon Web Services
(AWS) resources based on declarative templates. Common criticisms of
CloudFormation include the use of JSON as the template language and limited
error-checking, often only available in the form of run-time errors and stack
rollbacks. By wrapping templates in Haskell, we are able to easily construct
them and help ensure correctness.

The goals of stratosphere are to:
- Build a Haskell EDSL to specify CloudFormation templates. Since it is
  embedded in Haskell, it is type-checked and generally much easier to work
  with than raw JSON.
- Have a simple checking/linting system outside of the types that can find
  common errors in templates.
- Be able to also read valid CloudFormation JSON templates so they can be
  type-checked. This also gives us free integration tests by using the huge
  amount of example templates available in the AWS docs.

Most of the commonly used CloudFormation resources are implemented, and adding
new resources is very straightforward. (We created a web scraper that generates
a JSON file from a given CloudFormation documentation page URL, and from that
we generate a Haskell module.) So far, we have implemented resources we use at
Front Row Education, and we will add more resources over time.

The library is very much in a usable state as-is. However, we want to make any
sweeping changes while it is still young. If you have any suggestions at all,
you want us to add your favorite resource, or if you find a bug, please file an
issue on Github!

Also, we want to give a huge thanks to Brendan Hay, the author of amazonka, for
his ideas and feedback on the project.

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe




--
Alberto.

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Stratosphere: AWS CloudFormation EDSL in Haskell

David Reaver
Thanks Alberto!

You can certainly set up permissions and security groups on EC2 nodes using CloudFormation, run your job, and then tear down the CloudFormation stack. You can use stratosphere to specify the AWS resources like an EC2 instance, a VPC for it, some security groups, and an instance role. Then, you can use something like Ansible or even just a bash script to set up and run your job on that machine. Once it's done, you can tear down the stack you just built.

We actually do something similar to what you said for running pgbadger on our Postgres RDS logs. Our logs can reach a few GB per hour, so downloading them to a local machine can be pretty time-consuming, and processing can take long as well. In order to get RDS logs, you need to have permissions to do so on AWS. I made a stack in stratosphere that creates a large EC2 instance for processing, a strict security group that locks down the instance, and also sets up a temporary user that only has permission to download RDS logs. Using Ansible, I then install pgbadger and its dependencies, download the logs, run pgbadger, and then download the results. Once the job is done, everything that was just created is destroyed.

In summary, what you're saying is indeed possible and is a great use-case for CloudFormation. You just need to using something on top of stratosphere, as this library is simply used to build the CloudFormation templates. I'm considering adding support for running the templates directly in this library, maybe using amazonka-cloudformation. So far that hasn't been necessary for us.

Also note that CloudFormation has a way to estimate the cost of your created resources: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-paying.html

On Tue, Apr 19, 2016 at 12:21 PM, Alberto G. Corona <[hidden email]> wrote:
Very nice job.

I have tried AWS EC2 and I failed miserably since the permission system and his intricate configurations, his paranoid security, and  their secret formulas for pricing are beyond an average human intelligence.

It is possible to have an example stratosphere application that could configure and run a Haskell program in one or many EC2 nodes , close the instances and send back the results?

Are there  some elements still lacking to achieve such goal?

2016-04-19 20:56 GMT+02:00 David Reaver <[hidden email]>:

CloudFormation is a system that provisions and updates Amazon Web Services
(AWS) resources based on declarative templates. Common criticisms of
CloudFormation include the use of JSON as the template language and limited
error-checking, often only available in the form of run-time errors and stack
rollbacks. By wrapping templates in Haskell, we are able to easily construct
them and help ensure correctness.

The goals of stratosphere are to:
- Build a Haskell EDSL to specify CloudFormation templates. Since it is
  embedded in Haskell, it is type-checked and generally much easier to work
  with than raw JSON.
- Have a simple checking/linting system outside of the types that can find
  common errors in templates.
- Be able to also read valid CloudFormation JSON templates so they can be
  type-checked. This also gives us free integration tests by using the huge
  amount of example templates available in the AWS docs.

Most of the commonly used CloudFormation resources are implemented, and adding
new resources is very straightforward. (We created a web scraper that generates
a JSON file from a given CloudFormation documentation page URL, and from that
we generate a Haskell module.) So far, we have implemented resources we use at
Front Row Education, and we will add more resources over time.

The library is very much in a usable state as-is. However, we want to make any
sweeping changes while it is still young. If you have any suggestions at all,
you want us to add your favorite resource, or if you find a bug, please file an
issue on Github!

Also, we want to give a huge thanks to Brendan Hay, the author of amazonka, for
his ideas and feedback on the project.

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe




--
Alberto.


_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Stratosphere: AWS CloudFormation EDSL in Haskell

Brendan Hay
Great work, David. Congratulations on the release!

On 19 April 2016 at 21:39, David Reaver <[hidden email]> wrote:
Thanks Alberto!

You can certainly set up permissions and security groups on EC2 nodes using CloudFormation, run your job, and then tear down the CloudFormation stack. You can use stratosphere to specify the AWS resources like an EC2 instance, a VPC for it, some security groups, and an instance role. Then, you can use something like Ansible or even just a bash script to set up and run your job on that machine. Once it's done, you can tear down the stack you just built.

We actually do something similar to what you said for running pgbadger on our Postgres RDS logs. Our logs can reach a few GB per hour, so downloading them to a local machine can be pretty time-consuming, and processing can take long as well. In order to get RDS logs, you need to have permissions to do so on AWS. I made a stack in stratosphere that creates a large EC2 instance for processing, a strict security group that locks down the instance, and also sets up a temporary user that only has permission to download RDS logs. Using Ansible, I then install pgbadger and its dependencies, download the logs, run pgbadger, and then download the results. Once the job is done, everything that was just created is destroyed.

In summary, what you're saying is indeed possible and is a great use-case for CloudFormation. You just need to using something on top of stratosphere, as this library is simply used to build the CloudFormation templates. I'm considering adding support for running the templates directly in this library, maybe using amazonka-cloudformation. So far that hasn't been necessary for us.

Also note that CloudFormation has a way to estimate the cost of your created resources: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-paying.html

On Tue, Apr 19, 2016 at 12:21 PM, Alberto G. Corona <[hidden email]> wrote:
Very nice job.

I have tried AWS EC2 and I failed miserably since the permission system and his intricate configurations, his paranoid security, and  their secret formulas for pricing are beyond an average human intelligence.

It is possible to have an example stratosphere application that could configure and run a Haskell program in one or many EC2 nodes , close the instances and send back the results?

Are there  some elements still lacking to achieve such goal?

2016-04-19 20:56 GMT+02:00 David Reaver <[hidden email]>:

CloudFormation is a system that provisions and updates Amazon Web Services
(AWS) resources based on declarative templates. Common criticisms of
CloudFormation include the use of JSON as the template language and limited
error-checking, often only available in the form of run-time errors and stack
rollbacks. By wrapping templates in Haskell, we are able to easily construct
them and help ensure correctness.

The goals of stratosphere are to:
- Build a Haskell EDSL to specify CloudFormation templates. Since it is
  embedded in Haskell, it is type-checked and generally much easier to work
  with than raw JSON.
- Have a simple checking/linting system outside of the types that can find
  common errors in templates.
- Be able to also read valid CloudFormation JSON templates so they can be
  type-checked. This also gives us free integration tests by using the huge
  amount of example templates available in the AWS docs.

Most of the commonly used CloudFormation resources are implemented, and adding
new resources is very straightforward. (We created a web scraper that generates
a JSON file from a given CloudFormation documentation page URL, and from that
we generate a Haskell module.) So far, we have implemented resources we use at
Front Row Education, and we will add more resources over time.

The library is very much in a usable state as-is. However, we want to make any
sweeping changes while it is still young. If you have any suggestions at all,
you want us to add your favorite resource, or if you find a bug, please file an
issue on Github!

Also, we want to give a huge thanks to Brendan Hay, the author of amazonka, for
his ideas and feedback on the project.

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe




--
Alberto.


_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe



_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Stratosphere: AWS CloudFormation EDSL in Haskell

Alberto G. Corona
In reply to this post by David Reaver
David.
Thanks a lot for the hints. I looked at some CloudFormation scripts and they look even more horrendous than what I have imagined. 

2016-04-19 21:39 GMT+02:00 David Reaver <[hidden email]>:
Thanks Alberto!

You can certainly set up permissions and security groups on EC2 nodes using CloudFormation, run your job, and then tear down the CloudFormation stack. You can use stratosphere to specify the AWS resources like an EC2 instance, a VPC for it, some security groups, and an instance role. Then, you can use something like Ansible or even just a bash script to set up and run your job on that machine. Once it's done, you can tear down the stack you just built.

We actually do something similar to what you said for running pgbadger on our Postgres RDS logs. Our logs can reach a few GB per hour, so downloading them to a local machine can be pretty time-consuming, and processing can take long as well. In order to get RDS logs, you need to have permissions to do so on AWS. I made a stack in stratosphere that creates a large EC2 instance for processing, a strict security group that locks down the instance, and also sets up a temporary user that only has permission to download RDS logs. Using Ansible, I then install pgbadger and its dependencies, download the logs, run pgbadger, and then download the results. Once the job is done, everything that was just created is destroyed.

In summary, what you're saying is indeed possible and is a great use-case for CloudFormation. You just need to using something on top of stratosphere, as this library is simply used to build the CloudFormation templates. I'm considering adding support for running the templates directly in this library, maybe using amazonka-cloudformation. So far that hasn't been necessary for us.

Also note that CloudFormation has a way to estimate the cost of your created resources: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-paying.html

On Tue, Apr 19, 2016 at 12:21 PM, Alberto G. Corona <[hidden email]> wrote:
Very nice job.

I have tried AWS EC2 and I failed miserably since the permission system and his intricate configurations, his paranoid security, and  their secret formulas for pricing are beyond an average human intelligence.

It is possible to have an example stratosphere application that could configure and run a Haskell program in one or many EC2 nodes , close the instances and send back the results?

Are there  some elements still lacking to achieve such goal?

2016-04-19 20:56 GMT+02:00 David Reaver <[hidden email]>:

CloudFormation is a system that provisions and updates Amazon Web Services
(AWS) resources based on declarative templates. Common criticisms of
CloudFormation include the use of JSON as the template language and limited
error-checking, often only available in the form of run-time errors and stack
rollbacks. By wrapping templates in Haskell, we are able to easily construct
them and help ensure correctness.

The goals of stratosphere are to:
- Build a Haskell EDSL to specify CloudFormation templates. Since it is
  embedded in Haskell, it is type-checked and generally much easier to work
  with than raw JSON.
- Have a simple checking/linting system outside of the types that can find
  common errors in templates.
- Be able to also read valid CloudFormation JSON templates so they can be
  type-checked. This also gives us free integration tests by using the huge
  amount of example templates available in the AWS docs.

Most of the commonly used CloudFormation resources are implemented, and adding
new resources is very straightforward. (We created a web scraper that generates
a JSON file from a given CloudFormation documentation page URL, and from that
we generate a Haskell module.) So far, we have implemented resources we use at
Front Row Education, and we will add more resources over time.

The library is very much in a usable state as-is. However, we want to make any
sweeping changes while it is still young. If you have any suggestions at all,
you want us to add your favorite resource, or if you find a bug, please file an
issue on Github!

Also, we want to give a huge thanks to Brendan Hay, the author of amazonka, for
his ideas and feedback on the project.

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe




--
Alberto.




--
Alberto.

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe