Why We Built OpsMill and Infrahub – As we’ve reached a major milestone with Infrahub moving into a public beta, it’s an opportune time as founders to share our vision and motivations for building OpsMill and our first product Infrahub.
Why We Built OpsMill and Infrahub – Our experiences led us here
Having both spent years deeply involved in the networking and the infrastructure industry, we have each separately encountered the challenging and complex nature of managing large-scale infrastructure. This has highlighted the critical need for a platform that is easy to manage, easy to maintain, reliable, cost-efficient, and scalable.
Raphael’s Experience
At a certain point in my career, I started managing teams that built large-scale infrastructure requiring ever-increasing amounts of configuration and personnel across multiple countries. As a result, automation became a significant part of our engineering efforts.
What I found is that over time we spent as much time maintaining and improving our automation framework, which became a bespoke software platform after years, as we did focusing on our core business of network and infrastructure.
The amount of effort devoted to our automation limited our innovation. I recall searching for an off-the-shelf product to replace our foundation and allow my engineers to concentrate on our primary business. But I never found anything that could suitably do the job.
Damien’s Experience
Over the last decade, I had the opportunity to help a variety of companies of many different sizes define their automation strategy. Without exception, they discovered that building an automation platform required a huge effort. Despite all the good intentions they started with, many couldn’t pull it off. The few that actually built an automation stack struggled to maintain them and keep them in production.
Why is this still so hard?
As an industry we’ve seen a lot of progress in improving the efficiency of managing infrastructure over the past decade. Just look at the proliferation of GitHub, GitLab, Terraform, and GitOps practices, and the rise of many purpose-built point tools.
Despite this, we’ve talked to many teams that struggle to sustain the automation of their infrastructure, particularly in dealing with networking and security.
Building a better Source of Truth
With OpsMill, our overall goal is to make infrastructure more agile, robust, and well-documented, allowing in-house engineering teams to focus on their craft and automation without worrying about the maintenance and upkeep of the automation infrastructure.
We decided to start with what we consider to be the most critical component of every infrastructure automation stack–the Source of Truth (SoT).
A SoT organizes and manages the data that defines the expected state of the infrastructure. The SoT is critical because sound data management is a prerequisite to all automation.
In other industries and use cases, the automation ecosystem is anchored by a central platform that facilitates all data management and becomes the system of record. Examples include ERP systems like SAP in manufacturing and supply chain management, and CRM like Salesforce for sales and marketing.
Our vision for Infrahub is that it will serve a similar purpose for infrastructure automation.
Addressing complexity that leads to fragility
The key problem we see with the current state of infrastructure automation stack is fragility. Our assessment is that this fragility stems the lack of a platform that adequately addresses the two main SoT requirements: data extensibility and change control.
- Data Extensibility is non-negotiable because the Source of Truth needs to store all the information about your infrastructure at multiple levels (technical, design) and it also needs to capture all the services that are exposed to the customers/users and it needs to adjust to the business specific needs and requirements. Data management that is too rigid will inevitably lead to a spread of data sources and significantly increase the complexity and the maintainability of the system.
- Change Control is a necessity for ad hoc changes that have the potential to significantly impact production. The industry today is heavily using Git to provide Version Control but at the cost of giving up all features we are usually expecting from a database (Schema, API, query engine). Git is a fantastic tool and it revolutionized the industry but at its core it’s been designed to store text files not structured data.
With Infrahub. we’re excited to bring a new approach to infrastructure management inspired by the best of GitOps and Infrastructure as code and the best of a database with at its core a very extensible data platform that supports version control. There are many other capabilities of course, which you can read about in Damien’s introductory blog post. If you’re ready to jump in, access Infrahub in Github and join our Discord to connect with the Infrahub community where our team is available to chat and answer questions.
Without a platform that addresses these two critical needs, every organization that has successfully deployed automation has completely or partially redeveloped their data management platform on their own at a significant cost and effort. The complexity of such home-grown efforts leads to fragility across their whole automation stack.
