Data Virtualization Interview

January 22, 2024

What is data virtualization?

Data virtualization represents one methodology for the integration of data. Consider accessing and processing the data in your time, without the need for physical data model application. We’re trying to create a paradigm where you go from actually moving the data from one place to another, to now synchronization. We also want to establish a consolidated view of data from different sources and present it to a consumer in the desired format without any need to really understand the technology behind the data or how the data is physically organized.

The reference architecture for data virtualization is comprised of key components of the data virtualization layer, which includes data integration, data governance, data security, and access control. These are some of the key building blocks of the reference architecture for data virtualization.

What are the benefits of data virtualization?

First and foremost, most organizations are trying to standardize their technology stacks and really to be technology agnostic. We have various products in place and various APIs, whether they are standardized or proprietary. So, with virtualization, organizations come up with data abstraction, which is technology agnostic. So, it really doesn’t depend on which technology runs behind. That makes it easier for the businesses and organizations to focus on what data they actually need to process for their business needs, rather than trying to understand the technologies that run behind.

Another benefit is acceleration of data access. It’s really about making data available in real-time from the AI sources of data, and eventually avoiding any replication, any synchronization, or any copying of the data. The end goal is that access to this data will be accelerated through the virtualization data layer.

Another obvious benefit is reducing data movements and duplication of data. It is known that many organizations are struggling with understanding where their source of truth comes from, and which data goes for the different purposes and different needs of business units. From the enterprise architecture perspective, data virtualization provides a layer which will allow access without the need to replicate data, or to have all the monitoring or logging of all the processes in place which are either transforming or moving data for specific purposes. Instead, data virtualization gives a solid layer which is focused on the business understanding of data and what data needs to be brought to the end consumer rather than moving and transforming the data for each business unit.

Finally, data virtualization has improved data quality and security through centralized data governance. This technology or methodology for access to data provides a central platform for data governance and all the aspects of data governance, including data quality, data security, and various parts of the data management lifecycle. From the centralization perspective, it’s of the principles of the well-built and well-designed architectures of the data solutions that have everything in one place, rather than having multiple different processes which are then difficult to consolidate and synchronize.

What use cases does Adastra implement with data virtualization?

So, I’ll just go quickly through the list of common data virtualization use cases that are seen across many industries and many organizations. First and foremost, [data virtualization] is efficient for real-time data integration.

Another big use case for data virtualization is business intelligence and analytics. Here, data virtualization methodology and data virtualization technology are used to create single-view or unified view of data for intelligence and analytic application.

The next use case is data warehousing. It is about creating logical data warehouse which provides a unified view of data from different sources. The greatest benefit of data warehousing use cases is that it enables organizations to manage their data warehouses. I mentioned earlier that data governance security is one of the biggest use cases where data visualization plays a key role in cloud migration. Data virtualization can be used to migrate data to the cloud without the need for physical data movement or replication. It is about enabling access to cloud data, whether there are existing various types of data storages in cloud databases or provided through any cloud-based APIs. It really provides a layer of abstraction where, with proper configuration of the access to these data, end consumers can reach relevant business data which will drive their business.

Additionally, we are witnessing a huge increase in the Internet of Things and the devices that can communicate with each other. Data virtualization here is used to integrate data from various IoT devices and sensors, allowing organizations to analyze and gain insight for real-time data. But also, there are some examples that I can make from the Telco industry and the streaming industry where, using IoT, you can actually have a view of what you have. For example, directly looking at the football matches or basketball matches, you use those different cameras which communicate with each other to seamlessly move the focus from one camera to another to create a sensation of a 3D effect for the people who are watching that.

One of the last common use cases I mentioned here is master data management where data visualization technology is used to create a single-view master data from different sources. These are some of those key data virtualization use cases.

What tools does Adastra use for data virtualization?

From a tool’s perspective, cloud-based solutions offer native tools. For example, Azure offers a data factory to support data virtualization by leveraging data pipelines that can transform data from various sources into a consumable format, similar to AWS. There is AWS glue that can pull information from multiple sources.

But those are great for native applications within those respective cloud environments. There are also tools that allow external consolidation of information such as Denodo and Informatica, to name two. Also, SAP HANA has its own database-based capabilities with virtualization capabilities based on top of that. Denodo is a data virtualization platform that specifically can combine data from multiple sources, including databases, data warehouses, on Prem and across the cloud, and then present it in a in an accessible format through API’s or through a virtualized layer if required, to access necessary data, enhancing that additional layer of security and governance on top as needed. That is separate from your existing environment. So, it’s a platform within itself that can support all of that.

Informatica as well can support the integration and virtualization capabilities. It has its own governance layers as well, but not necessarily all-encompassing in kind of one platform-based solution through different modules that can be connected. So, these are the most common tools. If you’re going cloud native or if you have multi-cloud or hybrid options, you might go with something like the node or Informatica or SAP HANA to tie everything together.

When it comes to where Adastra has experience in providing data virtualization, we have a span of skill sets and knowledge leveraging these tools and in multiple different industries; in in finance, manufacturing, retail, healthcare, public sector, telecommunications, energy and utilities, transportational logistics and hospitality to name a few.

Where we excel and where we shine is not necessarily only about the integration component, but also understanding the whole architecture from a reference-based perspective: what are the key components to modernize the platform and then associate key use cases to have that virtualization layer fulfill and satisfy. So, for example, it’s the idea of asking an organization what your use cases are, well, you know, they don’t always know what their whole vast library of use cases might be, but they do have a few key use cases they want to attack first. The goal with data virtualization is to enable that, as use cases get produced and as they get added on, the virtualization layer can dynamically support things as they are needed because they would provide a consolidated view across all those multiple different sources and storage layers to provide access to an API or a user-requested format that might be needed.

In terms of data access and data provisioning as part of a data fabric, virtualization would allow for speed to delivery methods to support that. Adastra’s expertise lies in, yes, the integration and understanding of the architecture, but also implementing some of the key use cases. This varies industry by industry.

In finance, we’ve done fraud detection where we’ve integrated data from multiple sources from transactional data, social media sources, and other external sources as well, to detect fraudulent activity in real-time. This is all done through a virtualized layer that does not interfere with operational orchestration or synchronization from regular systems. The goal here is that this projection is an abstracted layer that can detect the activity and action it in real time.

In addition to that, from a finance perspective, we’ve done regulatory compliance use cases where we’ve provided that central view of access and usage, understanding lineage across how data is used, where data is used, and then making sure that organizational regulations, as well as external regulations and those policies, are enforced throughout, wherever data might be used throughout multiple departments.

In manufacturing, a few of the key use cases we’ve done there are around supply chain optimization. This involves integrating data from multiple sources including supplier data, logistic providers, manufacturing process data, IoT data and external data to sort of eventually optimize supply chain effectively; in ways, this creates a virtual data digital twin – not to the extent that it has full digital twin capabilities, but enough to run what-if scenarios and then drive models to improve operational efficiency.

In the manufacturing space, we’ve done use cases around predictive maintenance: reading all sensor data and equipment data to predict when maintenance is needed on machines. The goal there is to reduce downtime and improve efficiency on the retail side.

For some other key use cases around customer experience, we define what that customer journey will look like. This does not only involve tagging on a lot of overhead for operational systems that bring in Point of Sale data in real time. Rather, it’s more about being able to abstract and then couple that with customer feedback, different product sources, social media posts and then understanding what that customer journey looks like to enable better downstream actions, next best actions, and personalized marketing campaigns.

We’re also working on use cases for retail around inventory optimization. This is similar to the supply chain optimization with suppliers and data, but also includes distribution centers, Point of Sales systems, and understanding inventory levels in real time to reduce either overstock or stock outs and then automate responsive actions to order more supply or distribute supply across different stores or distribution centers to effectively manage the stock and inventory supply.

With healthcare, what we’ve done there is with the patient data integration, bringing in some health data, lab results, and other related external source information, to provide not only a comparative comprehensive view of patient data and doctor data and therapy data, but also to enable better patient care. Then, they use the virtualized layer to create data products that could eventually be monetized and then accessed by subscribers of that information to use for aggregated consumption in experimental research and so on.

Within the public sector, we’ve done more around administrative services, so understanding city-based events and regionality of where they occur. Our work in this sector also involves understanding emergency services like police, fire, and other citizen services like library or garbage collection from municipalities. In this, the focus is on understanding what challenges might exist, how many calls are received in the call center, analyzing all that information to have access to data for reporting from a  data democratized point of view where accessible data can be simply requested to then do analysis of certain event data that might have occurred that are not necessarily within the technical teams but might be enabled by a business user on the front end.

From an energy and utilities perspective, there’s many use cases here around energy consumption and optimization. This includes Smart Meter stuff that we’ve done on the Microsoft side, analyzing weather data and patterns to regulate energy consumption and usage and distribution. As well, we strive to ensure that we have good information governance and quality to optimize consumption and define what the key usage time frames are like. Part of it is also trying to understand what Peak Hours are like and trying to find a pricing model that would be sustainable for the future.

From transportation and logistics, fleet management was one large one where we brought in data from multiple sources, not only within the Data Lake environment, but also virtualizing GPS data, maintenance logs, vehicle data mileage statistics, and so on. The goal there was to have a Master of Information around their fleet and then leveraging that to optimize delivery logistics around their fleet, mileage use, reducing fuel consumption, and improving efficiency and maintenance and so forth across the fleet

With telecom, there are two key use cases here. We did one use case around customer churn prediction in line with the customer journey. There are many different inputs here that can contribute to customer churn. It’s not necessarily the types of services offered, but it’s also related to call center data, user experience data, leveraging social media data, system data, and so on. It is about understanding the customer’s lifetime value and the number of interactions they’ve had to develop an

Some of the key challenges where Adastra can help navigate based on our area of expertise to enhance the data virtualization capabilities, is not only our experience with these specific use cases or the technologies in Azure and AWS and with Denodo and Informatica, but also, we understand the aspects around data quality and integrity and how important they are when it comes to data provisioning and access.

The idea behind data virtualization is the integration of multiple sources with different standards and different quality and different levels of accuracy. We know data quality is integral in terms of keeping it enriched, up to date, complete, conformed, and consistent across, so that whoever is accessing or whatever it’s being accessed for has it in a consistent manner.

We know the security and privacy concerns around virtualization of data, specifically around working with PII data, key financial and general Ledger information or specific healthcare data that might be relevant to always keep secure and ways to abstract or mask that information through virtualized layers. We understand the performance and scalability of optimizing access either through queries or APIs. Overall, it is about ensuring that there is high performance of data in real time that it can meet business demands.

Challenges

There are some compliance regulations around security and privacy as well. There are also some consent-related regulations that we’re aware of through GDPR, CCPA, or what comes in through Bill 64 in Canada, or certain data retention policies or DSAR requests or other classification policies that we’re aware of. When it comes to virtualization, we don’t want to open up everything, but we want to open it in a very secure and compliant way. Those are some of the challenges we’ve been able to navigate through in many organizations.

The two last that I’ll mention – one around of course integrating it all. We talk about data quality difference in standards but integrating whether it fits into a common data model, whether it fits into just a generalized landing area to be used or leveraged. These are architecture decisions and challenges that that we’ve come to help organizations with, in terms of understanding what their and vision goal is with not only cloud modernization, but what they want the business to achieve and where data virtualized layer can help support that.

The last thing that I think is probably the most challenging that is less technical is the organizational buy-in and the adoption and/or cultural challenges. So, getting the support from stakeholders across the organization, understanding the overall value and return on investment that virtualized later will bring, if there’s any change in management aspects around roles and responsibilities, and really making sure that everyone is doing what they need to do and held accountable to ensure its overall success.

I’d say those are the key challenges. Those are the key use cases where Adastra has helped and those are the key tools that we know and use consistently to deliver data virtualization to our customers.

Some specific clients that we have implemented these changes for include fraud detection at some large Canadian banks, such as TD. For manufacturing work, we have worked with Magna and Woodbridge Foam. For retail, we have worked with Loblaws, MAG (Mark Anthony Group).