The code is the key to determine if your company will be another level of Infrastructure management and automation. This has never so been clear as the last in recent years.
The number of opportunities for professionals like DevOps, SRE, or an Infrastructure Engineer in some markets exceeds the demand for Software Engineers.
Sounds that the tools that have been used successfully for a while seem to no longer serve teams and companies.
Nowadays, with the high demand for hybrid cloud, and even for companies that choose a unique cloud provider are failing. The tools didn’t provide support for containers, passing through virtual machines, and covering other services that want adopting best practices.
Faced with this scenario, many looked at Terraform as a tool that has achieved considerable success in provisioning infrastructure in different cloud providers as the final solution for all issues.
However, innovation is hardly made up of sudden jumps as many would like to tell.
Terraform has already had its importance, but given the points, I want to share in this article, it is proving to be a tool that will have to fight to continue receiving all these highlights.
The key: HCL
Hashicorp Configuration Language was created to be used in Terraform a descriptive way as a more flexible alternative than the YAML market standard. If we look at so many other projects like Kubernetes, Helm, Ansible, that uses YAML as their descriptive language, we realize that the decision to create this language started to be questioned as soon as it proved to be not as flexible as was thought, and soon the second version arrived: HCL2, leaving many negative comments and affecting projects that sought consolidation, even if partial.
This reminded me of Edsger W. Dijkstra who in 1968 wrote an article “Go To Statement Considered Harmful” demonstrating the harmful effects of its use.
“the use of the go to statement has such disastrous effects, and I became convinced that the go to statement should be abolished from all” higher level “programming languages” - Edsger W. Dijkstra
Recently, Hashicorp made a move that seems to move towards a more aligned future to the modern demand demanded by infrastructure as code tools. “CDK for Terraform: Enabling Python & TypeScript Support”.
Terraform’s biggest mistake could have been avoided if the Hashicorp team realized where the DevOps culture came from and where that career would go. Honestly, it was not so easy to predict it before.
A glorious time of green monitors and Unix systems, before the Linux era, the System Administrators already used bash, Perl, and Python to automate small processes and create command-line tools. The Python language is widely used in Linux distributions where the containers came from.
Looking for the provision and configure any feature in Google Cloud you can use API, CLI, or SDK in Python.
Just like in Google Cloud we see a pattern. To provision and configure any resource we have the option of consuming the API, CLI, or SDK in Python, the famous BOTO3.
Azure follows the same line as the others offering its API, CLI, or its Python SDK.
Python as a language in the world of cloud providers is so important that not only the SDK is provided in python, for example, Azure created its CLI in python, being the first client of its own SDK.
Going back to the career, however looking to the future, we see that the System Administrators became DevOps to be closer to the software engineers, reducing communication barriers and making the processes and tools more and more efficient and focused in the code. The market already hires and will increasingly look for SRE - Site Reliability Engineers or Infrastructure Engineers who are more developers than Systems Administrators.
The same principles, paradigms, and flows of software development are and will be adopted for us more and more…
Definitely, new times require a new approach in order to allow Infrastructure teams to scale and meet the growing demands already present in our day-to-day lives. Pulumi comes to occupy not only the place of Terraform but to consolidate the source code as a solid foundation for infrastructure.
Some of the reasons that have justified the migration of the current code base written for Terraform for real code running on Pulumi:
- Python as a mature and consolidated programing language, more than 30 years, provides support for descriptive, functional, and Object-Oriented programming, like another language that can cover more than one paradigm… Extensive support from cloud providers, easy integration of new team members, the relatively low learning curve. And many more benefits that a mature programing language and their community can offer to any project. https://www.pulumi.com/docs/intro/languages/python/
- Wide support for all features offered by cloud providers. Pulumi Azure provider covers 100% of the resources available. Pulumi works directly using Azure Resource Manager (ARM), instead of relying on a handwritten layer as in the previous provider. https://www.pulumi.com/blog/announcing-nextgen-azure-provider/
- Unit Tests, Property Tests, and Integration Tests, because Pulumi uses general-purpose programming languages to provision cloud resources, you can take advantage of native testing frameworks and perform automated tests of your infrastructure. https://www.pulumi.com/docs/guides/testing/
- Pulumi uses the concept of Stacks. A stack is an isolated, independently configurable instance of a Pulumi program.
- Policy as Code (“CrossGuard”). The crossGuard is Pulumi’s new Policy as Code offers. CrossGuard empowers you to set guardrails to enforce compliance for resources so developers within an organization can provision their own infrastructure while sticking to best practices and security compliance. Using Policy as Code, you can write flexible business or security policies. https://www.pulumi.com/docs/guides/crossguard/
If you are finding it difficult to scale your infrastructure as code created with Terraform I believe that this article has given you some important information as to why this occurred. I strongly suggest that you try to gradually migrate your code to Python using the libraries provided by Pulumi. Pulumi developed a tool to help in that task:
tf2pulumi — Convert Terraform projects to Pulumi Python programs.
$ tf2pulumi --target-language python
If you are in the process of making a decision, consider the background presented in this article that could save you months of rework. Perform a Proof of Concept(POC) using Pulmi.
For the next articles, I will keep my focus on creating infrastructure from scratch with Pulumi and Python, demonstrating how Pulumi can be flexible and simple to adopt.