Data Management with the User Experience in Mind

Reading Time: 8 minutes

These are exciting yet challenging times, especially in the data space. Generative AI (GenAI), with its challenges, brought hope in a new reality of what could become of our data. There is still some hard work ahead, but now we have a renewed idea that AI can bring astounding value. I have seen this firsthand, but I have also seen that data management is critical to support more everyday tasks such as ad-hoc analysis, metrics, optimization, planning and integration, on both small- and large-scale projects. In fact, it has always been the core of these critical activities.

A decade or so ago, we supported business users’ efforts to access data with shadow IT and saw that when business users do gain access to data, it can spark creativity and ingenuity that leads to innovation and growth. Now many of us data architects (and yes, I include myself here as having been in the trenches) now have to figure out how we can both empower these efforts and support them safely. This has always been a tall order, but today there are many new technologies that help to both store data and speed up access.

We have faster databases, data lakes, and parallel processing, with options to move and store information both on premises and in the cloud. We have ETL and ELT in modern form-data pipelines, which can automate data processing. We have platforms built for analytics, ML, and AI. We have many reincarnations of technologies for persistence—but at the end of the day, it is still storage. These advances are great, but I believe that this is only one element of the user experience.

Who are the Enterprise Users, and what do they want?

Typically, when we think of users, we think of business analysts or data scientists, and these are important users. But I feel that we should also think of end users, business partners, applications, integrations, AI applications, and whatever the future brings, as a new class of users. I also feel that we should also consider the “arms” of IT, such as the developers, change control managers, architects, and support personnel. This is because all of these users play an important role in data management, and the degree to which you can address their experiences can make or break your implementation.

If we evaluate what these users want, we can better understand what our architecture must address. Fundamentally, we have those who need to access information; however, along with that comes consistency, usability, and the processes and capabilities to support everything. From my experience, breakdowns in any of these areas can thwart chances of adoption and render a solution untenable. Think about it. A business analyst creates a wonderful report or dashboard, and users don’t have access to the underlying data. What are the processes a user must follow to gain access? How complex and perhaps frustrating is this process likely to be? The dashboard can be lightning fast, but if the process to gain access is cumbersome, it puts a strain on the support structure and hardens users’ hearts against the solution. I am sure you can think of many other similar scenarios.

How do we manage data for great experiences?

First, we need to establish what comprises a great experience, and I believe we can sum it up in the statements below:

• “I can find what I need.”
• “I can access what I need.”
• “I can do what I need to do.”
• “I understand what I am using.”
• “I know where to get help.”

I feel that these statements provide a good starting point for discussion, and if you think of others, think about how you can consider them in your architecture. What I have done next is to define an influence diagram for user experience, keeping these statements in mind.

The factors I have listed that influence user experience are as follows:

1. Consistency – (“I can find what I need.”)  This speaks to having known, established processes and paths to information.
2. Accessibility – (“I can access what I need.”)  This is the main purpose for data management—providing secure access to data for different types of users.
3. Performance – (“I can do what I need to do.” ) How long do users have to wait for information? How quickly do reports and dashboards refresh?
4. Usability – (“I understand what I am using.”)  A data management platform is usable when it empowers understandability, broad usage patterns, and necessary capabilities, and is both simple to use and trustworthy.
5. Supportability – (“I know where to get help.”)  A data management platform that is supportable means it supports and simplifies security, change control, triage, monitoring, security and auditing along with mechanisms for testing to instill trust.

Enabling User-Centric Data Management with a Data Fabric

Data fabric is a system that helps bring together and manage data from different places in real time, no matter where it’s stored, how it’s formatted, or how quickly it’s needed. It’s built using flexible components like tools to combine data, organize it, use AI to analyze it, and add meaning to it. This makes it easier for people to access the data they need, speeds up how quickly they can get useful insights from it, and keeps data secure, private, and well-organized across multiple distributed systems.

Data fabric can significantly enhance a data management architecture. Below is the influence diagram again, re-drawn to demonstrate how a logical data fabric, enabled by the Denodo Platform’s logical data management capabilities, can simplify the influencers of a great user experience.

Consistency

Users have an instinctive need for consistency, and this can apply to many different levels of the data management infrastructure, from points of access, to security, to processes. These are just a few areas that highlight challenges of consistency, and from my experience, it is critical to understand the business user’s perspective with regard to consistency.

With the Denodo Platform, users no longer need to wonder where to go to get access to information—one place alleviates confusion. Also, when you manage the security in the Denodo Platform with Global Security Policies, you have consistent security that is understandable, manageable, and auditable. You can provide security based on roles and/or user attributes, potentially relieving users from having to even request access. In the event that your organizational model requires requests, users can make those requests directly in the Denodo Data Catalog, and those requests can be integrated with incident management systems and processes.

Accessibility

For business users, access to data means being able to quickly and easily retrieve the information they need from a variety of databases or storage systems to drive decision-making and strategic planning. It involves not just the ability to pull data, but also the ability to analyze, interpret, and leverage that data effectively in their day-to-day operations and long-term business goals.

Managing multiple data warehouses or data lakes in a distributed setup presents some exciting opportunities alongside significant challenges. The key here is connectivity. Having seamless access to data across different storage spots can really boost our ability to analyze data and make smart decisions. Yet, the complexity of such a setup can’t be ignored. This complexity results in long lead times for data access and challenges around security and compliance, which contributes to even more complexity, as users find workarounds to gaining the access they need.

The Denodo Platform provides universal access to data, regardless of where data is and how it is managed. Let a logical data fabric, enabled by the Denodo Platform, help you bring a great experience to your users.

Performance

Performance is an important issue, especially for dashboards and reporting. Today, many persistence options are very fast, so expectations are high. The challenge here, however, is justifying the costs, time, and overhead to move data onto these fast systems. In some cases, there are even multiple systems at play, for business reasons. In some situations, we cannot move data for legal, policy, or regulatory reasons. As a result, we are confronted with the challenges of providing good performance across multiple sources.

The Denodo Platform is the most advanced technology for optimizing performance across a multitude of sources. It leverages active metadata, artificial intelligence (AI), and years of experience to provide optimized querying across systems. It has access to an embedded MPP engine for more complex queries, along with a variety of caching options for optimal performance across myriad scenarios.

The Denodo Platform also enables you to manage your data estate however you see fit. You can move data where you need to, for any reason, like migrating to new technology, moving to less expensive or faster storage, moving to a different format or storage location, or moving to a different level in the architecture. You can move data knowing that the Denodo Platform’s optimization power is always working to provide the best possible performance.

Usability

Usability is related to consistency in many aspects. A person’s comfort level can dictate whether they deem something usable and therefore valuable. A business user’s level of familiarity with tools can determine how productive that user can be. Today, we have a wide range of users from systems, data scientists, and developers, to end-user consumers of reports. Balancing needs across this wide breadth is essential.  The Denodo platform offers API- and SQL-based delivery methods supporting formats like REST, GraphQL, and ODATA, to name a few.

In addition, the Denodo Platform empowers business users to perform data prep with an intuitive low/no code web interface. There is also a web interface for more advanced users to create data products. Business users can use the Denodo Platform’s semantic layer to create products and share them, with the knowledge that security is intact.

We at Denodo are continually enhancing our offering to make using data even simpler by implementing AI for naming and descriptions, and empowering every user to get helpful answers from their organization using generative AI (GenAI).

Supportability

Most of the areas we touched on above also have an influence on supportability. Along with “I know where to get help,” fundamental statements like “this data doesn’t look right,” “I can’t find what I need,” or “I don’t have access” are also expressions of the need for support. Major issues such as data breaches and systems outages will also require support.

There are additional considerations to be aware of, such as system changes, enhancements, policy changes, resource management processes, new projects, new technologies, and really, anything else that can create change around the enterprise ecosystem. A data management architecture designed with supportability is essential to a great user experience.

The Denodo Platforms has built-in monitoring and auditing of all activities occurring within it. This reduces the burden of trying to figure out where to begin support, in the event of issues. With its low code/no code, layered approach to development, combined with AI-powered assistance, locating the source of issues is streamlined.

The Denodo Platform also provides a testing tool that you can integrate into automated CI/CD processes. You can perform a myriad of tests to validate data and security. This is a great way to verify that enhancements to data products continually meet requirements.

Considering all the moving parts in our infrastructures today, and their costs, the Denodo Platform gives you peace of mind that many of the adjustments needed, due to source changes, can be managed with little to no disruption to the business. In addition, it supports high availability configurations that enable upgrades, maintenance, and code migrations with little to no downtime.

Keeping Users in Mind

It’s clear that the future of data management hinges not only on the robustness of technologies but also on their adaptability to meet the evolving needs of a diverse user base. As we’ve just reviewed some of the intricacies of providing consistency, accessibility, performance, usability, and supportability, I hope that I’ve shown the Denodo Platform’s potential for  simplifying and enhancing the data management infrastructure with the needs of different users in mind.