Data Governance Will Pull You Out of the Data Swamp

November 22, 2018

Growing data volumes and their increasing heterogeneity expand the range of analytical tasks. Unfortunately, despite all the progress, the gap between good and bad analytical solutions is widening compared to the past. These very poor analytical solutions are called data swamps.

How does a data swamp arise?

In the original sense, a swamp is an area saturated with freshwater, covered with peat and plants. In the realm of data integration, metaphorically, it refers to the failed outcome of implementing a Data Warehouse or Data Lake, which analogously becomes a swamp due to incorrect data “flow”. A data swamp is understood as any data platform or set of data from a data platform.

The figurative data “mud” cannot be reasonably used for analytics, or even worse, the resulting analytics may have negative impacts on the entire organization. These data analytics quality issues never arise by chance. They are the result of a series of more or less poor decisions, regardless of the technological foundation involved.

This can involve, for example, logical components like warehouse/lake or even service models like cloud/on-premise.

Therefore, a data swamp is not limited to data lakes, which are typically associated with them due to their “informal architecture” primarily focused on raw data.

Due to this approach, data lakes are more prone to “swamping” than traditional data warehouses, due to their significantly more flexible properties, which encourage the postponement of systematic maintenance of solutions, something that is not always possible with a data warehouse due to its formalized architecture.

Confronting the data mire with the aid of Data Governance

Is the solution to abandon the advanced concept of a Data Lake and revert to the older but tried-and-tested data warehouses? No! Even a data warehouse is not guaranteed to prevent becoming a data swamp, sometimes with very distinct contours.

Regardless of the type of data platform used, it is crucial to place much greater emphasis on data strategy and Data Governance – the latter being the only effective tool against the data mire.

Data Governance must always stem from the current data strategy, which defines the use of data assets in line with the company’s strategy, and cannot rely on generic standards.

Typically, a data strategy document is completely missing, even though it defines the relationship of the organization to its data assets. At a minimum, this document should include definitions of objectives, a list of related risks, principles, and concepts of the entire data management.

Data Governance should follow this defining document and outline the execution framework and resources to fulfill this strategy in accordance with the overall organizational strategy.

In practice, Data Governance is often unfortunately reduced to a passive, omniscient document, tucked away for worse times or for an auditor, depending on who or what comes first.

Another less common but also flawed approach is to limit Data Governance solely to the area of data quality, and only in a reactive mode. Both approaches are insufficient and do not respect the essence of the problem.

In times of data boom, it is impractical to use outdated tools for new problems. Properly conceived Data Governance does not complicate data management; rather, it simplifies and reduces costs.

It proactively addresses problems that are reactively resolved only by incurring high costs or are ignored by organizations, hoping nothing adverse happens. This often leads to unwanted cost increases or revenue declines.

Without comprehensive Data Governance, it is impossible to manage the life cycle of data or the entire data platform. Properly implemented Data Governance covers data architecture, metadata management, data security, master data management, operations, data integration, data quality, and technology.

Stuck in a data swamp?

The fundamental rule is to avoid entering a data swamp in the first place. But what if you find yourself in one already, and how can you ideally prevent this from happening? How to detect the signs that our data platform has become or is soon becoming a data swamp? For simplicity, let’s assume we have a data strategy and a tailored Data Governance framework. First, we need to answer the following questions:

• Do we have a clear idea of what we want to do with our data and how?
• Do we know how and for what purposes the data is used?
• Is it time-consuming to obtain data in the desired structure?
• Do we understand how data is generated and what its significance is?
• Do we have comprehensive, up-to-date documentation of our data platform?
• Do we have metadata describing data sets and data transformations?
• Can we search and analyze data on an ad-hoc basis?
• Are the data interconnected, consolidated, and complete (logically or physically)?
• Can we identify which processes can create value from our data for end-users or customers?
• Is our data of sufficient quality?

If you answered ‘no’ to most of these questions for your specific data platform, there’s a high likelihood that you indeed have an authentic data swamp. The extent of the problem and the cost of eliminating the data swamp depend on its depth and breadth and the possibilities to free up data flows to be meaningfully utilized.

In extreme cases, it may be more appropriate to build a new solution from scratch, avoiding past mistakes. In typical situations, minor adjustments implementing Data Governance tools and processes into the data platform are sufficient.

Liberate Yourself

How to proceed specifically in “drying out” a data swamp?

1. Define and Describe Data Sets

Start by defining and describing the data sets, including their purpose and content. Having a clear definition of the content, along with the corresponding metadata, allows for meaningful utilization of the data by end users. Collecting thousands of items for each business case is not efficient; it is much more effective to select only those items that are truly necessary and useful.
This approach may seem to contradict the concept of data lakes, but it significantly facilitates the “drying out” of your swamp.

2. Establish Data Ownership

Only data owners can accurately define how data should look and what quality it should have to be useful for further processing such as analysis, reporting, and others. The problem often underestimated in practice is the transfer of data responsibility to IT departments, which usually lack the requisite expertise.
The issue of data security, which again can only be defined by its owners, must also be addressed. Without clear responsibility, every platform will eventually turn into a data swamp, whether it is a flexible Data Lake or a rigid Data Warehouse.

3. Define a Security Model Including Data Access Rights

A defined security model facilitates access to data assets. Data should be easily sortable and analyzable. The technological aspect, which significantly affects the efficiency of de-swamping, should not be forgotten.
The goal should be maximum automation, not only in processing but also in areas of advanced analytics such as Natural Language Processing, Cognitive Intelligence, and Machine Learning.
These technologies allow for the preprocessing of data analyses and significantly improve and speed up the analytics for end users. Without this overlay, proper long-term utilization of data stored in the data platform cannot be ensured.

4. Engage Specialized Firms

It is not shameful to turn to experts from specialized firms when drying out data swamps, as they have extensive experience in processing data mire and can often completely save the data platform. After cleanup, maximum attention must be paid to maintenance and adherence to all rules to prevent the swamp from returning. The effort and cost are well worth it compared to re-drying the swamp.

Operate on Quality Data, Not in a Data Mire

Every data swamp has an escape route; the question is what it is. Sometimes it may mean incurring higher costs, but the result is an enterprise truly optimized by data, generating long-term higher profits than a system operating on autopilot. Quality data is the greatest asset of organizations.

You have likely heard this many times before. It certainly holds true, but it cannot be managed and utilized without efficiently functioning data platforms. These platforms must not even remotely resemble data swamps, where an individual or organization can get lost or, in the worst case, utterly drown.