“Toto, I’ve a feeling we’re not in Kansas any more.”
Dorothy, The Wizard of Oz
We, the software vendors of the world, love
to talk about ridding the IT world of silos. So what, may you ask, is a silo anyway? According to the source of all knowledge (Wikipedia, of course), silo comes from the greek word σιρός (siros), which translates to “pit for holding grain.” For most of us, Wikipedia’s definition brings to mind large, metal containers that dot the golden landscapes of states like Idaho or Nebraska. But the term “silos” has also become a great visual analogy for many of the problems with IT and software development, so it’s worth delving beneath the surface to see what we can learn.
Why do we need silos?
My brother lived in Kansas for a few years and we’ve talked about the ways in which local farmers use grain elevators and silos. It’s amazing to think that farmers from hundreds of miles around bring their grain to these giant grain elevators so the fruit of their labor can be deposited into giant concrete silos. Why does this make sense for farmers? Because their corn is a commodity, i.e.,. all of the corn in that silo is basically the same and sold in bulk just like oil, copper ore, beans, etc.
For a long time corporations have done and continue to do something very similar with their data. They create large scale processes to dump related types of data into purpose-built databases. These “data silos” often mirror the organizational structure of the IT organization–network monitoring data goes here, systems monitoring here, etc.
Do Data Silos work?
Silos serve an important, rather obvious function for the commodities industry. They enable the storage of commodities. No one would put rice in oil or even rice with corn. Silos optimize for costs and efficiency since the value of the commodity is in its volume, or bulk–not an individual kernel of corn.
From a corporate perspective, data silos have the opposite effect of siloed corn–they increase costs, decrease efficiency and limit innovation. In the modern corporation, IT culture is rapidly evolving to a shared ownership model removing the organizational inertia to sharing data and ideas. When data is siloed, it actually creates unnecessary friction–hindering collaboration and problem solving across functional boundaries. Conversely, mixing related data can dramatically increase the value of the data by uncovering important patterns and correlating issues across different components. One only needs to look back at the recent past of data warehousing to see a mostly failed attempt to uncover this untapped value.
Who benefits from silos anyway?
So, who benefits from silos? In the world of commodities, silos empower industrial scale commodities trading and efficiencies. For the agriculture industry, in particular, siloed commodities have led to lower prices for consumers though, one might argue, some reduction in quality. Regardless, silo-based storage has mostly been a net positive.
What about data silos? In the world of modern applications, data silos are definitely not a help to application developers and operators. And, they rarely help business decision makers. Most importantly, they don’t help end users or customers. So, who do they help? One clear answer–software vendors. You depend on their products to derive value from the data they house for you. These software products and services either empower you to gain the fullest possible value from your data or hinder and hobble your efforts. In the past, software vendors could sidestep this thorny issue because their interests were aligned with the interests of the traditional organizational structure. No more. The architecture of the modern application (microservices, containers, etc.) demands the free flow of data to connect the dots and form a complete picture. Cross-functional, empowered teams of the Agile/DevOps revolution don’t defer to functional boundaries when solving problems. So, does it make sense to use machine data analytics tools that operate in this same siloed way, holding your team back from delivering value to your customers?
Better Together
The last 10 years have brought amazing changes to the power of software to deliver business value. The cause of this shift is the symbiotic relationship between powerful cultural changes in how companies organize themselves and get work done along with the tectonic shifts in how business value is delivered over the Internet to customers at massive scale and near real-time speed. Unfortunately, many corporations are still stuck in the world of data silos and losing out on the rich value their data has to offer by treating it like corn. So, it’s time to see how valuable your organization’s data truly is. Don’t let anyone, or any product, stand in your way.
