Terraform and IT Infrastructure Automation

IT infrastructure is the backbone responsible for the digitalization of any industry, including oil and gas. Currently, Infrastructure as a Code (IaC) is gaining a lot of attention in the IT world, as it allows IT infrastructure to be defined, deployed, and managed rapidly and reliably at scale.

Terraform, an open-source, IaC software tool developed by HashiCorp, allows defining the infrastructure using a declarative configuration language (e.g., JSON, HCL – HashiCorp configuration language) and automated infrastructure management. As pictured below, Terraform offers automation capabilities in various aspects of IT infrastructure areas.

While Terraform offers tremendous infrastructure automation capabilities, it does present some limitations. This article describes some of the constraints and corresponding workarounds that can be used to mitigate the issues.

What are Terraform shortcomings?

• Terraform and external IT resources

Terraform assumes the management of IT resources (e.g., virtual machines) right from the start to the end of life. However, when an external resource is provisioned outside of Terraform (e.g., manually), administrators are required to “import” existing resources into Terraform. The “import” operation supported by the tool natively is very complex and requires a point-in-time configuration file (with many required parameters). This is difficult, especially when hundreds of resources are being imported into Terraform and each resource is configured differently.

• Terraform dependence on the state of resource

The Terraform model of automation heavily depends on maintaining the last known good state of the resource in its inventory. But when any administrator makes changes on the resource directly on the platform (i.e., for troubleshooting), it invalidates the state known to Terraform, making it difficult for administrators.

How to address these shortcomings?

The solution is the reliable and automated import of the existing resource, which requires a consistent, point-in-time configuration file. If we can auto-generate a configuration file that is clean and consistent with the platform, we can then reliably import an existing resource and bring that infrastructure item under Terraform management.

How to auto-generate configuration files?

First, we need to determine the “source of truth” relevant to a platform. Every platform, whether it’s a public cloud (e.g., Amazon, Google, Azure) or private cloud (e.g., VMware), or Kubernetes containers, maintains resource object inventory (metadata) that is true at any point-in-time, exposed over the API (application programming interface). This metadata can be programmatically fetched to generate a valid configuration file required for Terraform import.

 

Once the source of truth is known for the platform, a logic script can be written in any known language (e.g., python, PowerShell) to fetch the required point-in-time parameter values for automated generation of a configuration file. A script can be created with custom logic.

Here are some examples of resource object inventory databases — a repository of metadata for an IT resource that is maintained by every platform:

1. VMware: For VMware infrastructure, resource object inventory is called MOB (managed object browser) and is accessible via https://<vcenterIP>/mob.
2. Azure: Different resource APIs offered by the platform offer access to metadata objects inventory.
3. Google Cloud: Resource-manager APIs and Cloud Asset inventory in google cloud offer access to resource object inventory.
4. Amazon Web services (AWS): AWS systems manager inventory provides access to metadata for the resources provisioned in the cloud.

Benefits of automated generation of configuration files

• Reliably import existing resources into Terraform and at scale

Output configuration files can be used by the IT administrators to enable DevOps way of handling any existing application and bring any resource under Terraform management.

• Remove dependence on maintaining state

With import logic, the point-in-time state of the resource needed for Terraform operation can be easily regenerated. Therefore, administrators can benefit both worlds, i.e., employ Terraform to automate their L1 and L2 level operations, and at the same time, can work directly on the platform without impacting Terraform functionality for L3 activities.

• Skip Terraform learning curve

Creating a configuration file manually is an iterative process where you define some configuration parameters, apply them, and then verify if it reflects the same desired state of the resource. But when a configuration file is automatically generated, IT administrators do not have to spend time to understand each and every configuration parameter needed to manage a resource via Terraform.

• Better adoption of Terraform

Considering Terraform’s capabilities to cover a variety of infrastructure landscapes, the automated generation of configuration files makes it easy for administrators to better adapt the tool for a variety of their infrastructure automation needs.

• Cost savings

It allows the use of a common tool to manage the diversity of the infrastructure landscape with a common team; their skill set proves to be a lot more economical for organizations.

Import support references on key platforms.

With the growing popularity of Terraform, use a single tool that can help define IaC and manage multiple diverse platforms with a single pane of glass. In many developer communities, platforms have released utilities (import logic) to facilitate the generation of configuration files automatically.

• VMware:

Please refer to my GitHub repository, where a sample import logic script is written in Python language that facilitates the generation of configuration files for a VMware virtual machine and thus allows automated import.

• Google cloud platform (GCP):

Google released a utility called “gcloud beta resource-config” to autogenerate a Terraform configuration file for a resource provisioned in their cloud platform. Refer to this link for documentation.

• Microsoft Azure:

Microsoft developers working on the Azure cloud platform have released a command line utility called “Azure Terrafy (aztfy)” for auto generation of Terraform configuration files. Refer to this link for documentation.

• Open-source utility (Terraformer):

This tool was developed by “waze,” a subsidiary of Google. However, it is not an official Google product. This open-source tool can be modified and used across all major platforms, including Amazon, Google, AWS, IBM Cloud and Ali Cloud. Click this link for documentation.

Conclusion

Terraform’s ability to define infrastructure as code (IaC) and its popularity in almost all IT infrastructure spaces make it a valuable tool. However, the Terraform tool inhibits professionals because of its limitations in managing existing infrastructure at a scale. Workarounds are identified to mitigate those common challenges as the Terraform community grows. The remedies referred to in this article allow full Terraform adoption even with pre-existing IT infrastructure and further empower experts to leverage rich IaC capabilities to ease day-to-day IT operations and tasks automation.