Designing Elegant Data Products


What is a data product?

The way organizations think about data and how they access trustworthy information is changing quickly. Demand for insights is growing exponentially and more efficient strategies to manage data are emerging. At the center of this change is a growing shift in mind set. Organizations are beginning to think about data as a product, a packaged offering that is reusable and refined. This movement gets away from the project-based mindset where every request for data is fulfilled with a new one-off data pipeline.

The key benefits of data products are:

Reusability
Reusability
Easy access
Easy access
Shareability
Shareability

As with any product, the way that data products are designed and presented to users makes a significant difference. In this blog post we will look at how to design elegant data products.

When we talk about data products, we are referring to them in the context of a larger IT strategy or data mesh. This is not to be confused with a data product as a part of a core business strategy where a data product targeting customers is an organization’s primary revenue generator. We are not talking about data products such as Google Analytics or Bloomberg.

Gartner defines a data product as:
“a curated and self-contained combination of data, metadata, semantics and templates. It includes access and implementation logic certified for tackling specific business scenarios and reuse. A data product must be consumption-ready (trusted by consumers), kept up to date (by engineering teams) and approved for use (governed). Data products enable various data and analytics (D&A) use cases, such as data sharing, data monetization, domain analytics and application integration.”

This very detailed and complex definition may be accurate; however, a more elegant definition might come from J. Majchrzak who defines a data product as “an autonomous, read-optimized, standardized data unit containing at least one dataset (Domain Dataset), created for satisfying user needs".

While both definitions are accurate, one is simpler and easier to consume. Data products are the same way; elegant designs are easier to consume, increasing their value.

For a deeper dive into data products, check out our recent post.

What is an elegant design?

How do we know if a design is elegant? Albert Einstein has was credited with saying, “Everything should be made as simple as possible, but not simpler.” An elegant data product, therefore, is ideally as simple as possible in order to obtain the best outcome.

Elegant solutions should be;

  • Focused and efficient
  • Coherent enough to handle edge cases in core logic, not thought not bolted on capabilities
  • Powerful enough to be applied to multiple applications

Why is elegant design important? Less complexity makes things much easier and enjoyable to consume, driving greater value. A simple but effective solution will outperform complexity.

Data product mindset

The first step to designing and creating elegant data products is to adopt a data product mindset. Often, this can be the biggest hurdle.

To adopt a data product mindset, you need to escape the project mindset. This is the idea that each time a data request comes into the data engineering group, a new project needs to be created and executed. This project mindset is much more reactive, with data engineers constantly scrambling to build data pipelines per the stakeholders’ requirements. Once one project is done, it is time to forget it and move on to the next one.

The product mindset is much different. Data engineers, analysts and data stewards think more proactively about data. Instead of waiting for one-off data requests, analysts, engineers, and managers work together to create data products before they are needed. This approach requires more research and insight to create data products that will be most useful to the greatest number of users, driving greater value per output.

Data products are also reusable, so they stay relevant thought the lifecycle of the data product. This lifecycle includes ongoing maintenance and improvement. As data products take on a life of their own, feedback is incorporated into new versions.

The biggest challenge in implementing data products and building effective and elegant data products is creating this mindset. When you shift to a data product vs. a data project strategy success is measured by outcomes not outputs. While data products do evolve, effective planning and design upfront will set the foundation for elegant data products.

Key traits of good data products

Useful and powerful data products typically exhibit certain traits. Designers should keep these traits in mind as they create their data products.

Discoverable

For data products to have the most impact they need to be discoverable. Even if you create a fantastic product, if no one knows it exists, it will not fulfill its potential. Data product marketplaces are great ways to get data products in the hands of users. Some data product marketplaces will use AI and predictive analytics to suggest data products to users, similarly to the way Netflix suggests new movies or shows to viewers. Elegance is not always about how you design the product but also about how you bring it to market and get it in the hands of users.

Quality

Clean and accurate data is a must-have attribute for any data product. Without data analysts can trust, your data product will not be valued by decision-makers. Designing and building data products must include a reliable process that cleans and normalizes data as it is merged and integrated.

Once you have a process, you need to both ensure and prove to your audience that it works. This requires tracking and sharing data quality metrics to measure variability and completeness, among several other qualities.

Secure

Keeping data safe is a requirement of any IT strategy but building security into your data product can be nuanced. Elegantly designed data products can provide granular access to data assets. Designing access rules that consider the roles of users and data attributes balance access and security. These access controls and data masking also provide efficient use of data tables.

Sophisticated encryption to ensure data is protected as it moves from database to analysis is also an important trait for great data products.

Observable

To ensure continuous quality, great data products have built in observability capabilities. Data products are only as good as the quality of the data they deliver. If decision makers do not trust the data produced by data products they don’t have much value. Data products should be designed with integrated monitoring features that detect anomalies and errors. This reduces the potential that bad data will make its way into an executive’s analysis or be used to train AI models.

Scalable

One of the benefits of adopting a product-based approach is that the more a data product is used the more value it contributes to the organization. Data products are much more flexible and can be applied to several use cases, increasing their utility. Consequently, data products must be designed to scale and meet the growing demand of users.

Collaborative

For data products to be powerful enough to solve multiple problems, input from various sources is required.

Building a diverse team to participate in building data products and the frameworks to support them is vital. Multiple stakeholders play a role in successful data products including data product producers, domain owners and consumers.

Data product producers take the lead and are most invested in the success of a data product. They may have data engineering skills or data analysts’ skills but must understand the needs of the consumers. Those with a background in product management or product ownership understand the product mindset.

Domain owners also play a vital role. They typically are responsible for ensuring proper governance. Domain owners need to be incorporated into the process as governance is a particularly important factor separating success and failure, so the right controls and policies are paramount.

Data product consumers are also a key piece of the ongoing lifecycle of data products. Their engagement and feedback provide the input to improve the utility of data products. They can rate their satisfaction with individual data products and how well they fit their needs. Tracking data product consumers behavior is also a big part of incorporating consumers into the process.

Accessible

Like discoverability, effective accessibility is an important trait of quality data products. Easy accessibility improves the process of getting data products and using them for analysis as simply as possible. Quicker access leads to faster time-to-insight. One of the barriers to rapid access is importing data products into your BI tool or AI model-builder tool. Elegant data product designs enable data products to be accessed from within whichever analytics package is preferred by analysts.

The second and perhaps more difficult barrier to overcome is gaining the authority to access data. Setting up the right protocols to enable access makes the process much safer and more efficient. Clearly defining who is responsible for enabling access is an important part of defining elegant protocols. In a more distributed framework, domain managers who oversee data collection in their group have the authority to provide access.

Subscriptions and data contracts define the duration for which a user has access and how data can and can not be used. By standardizing these agreements up front, users don’t have to go through the process each time they want access to a data product, simplifying the process.

 

Customizable and Interoperable

To meet the needs of users’ data products should be adaptable to specific business requirements and user preferences.

Instead of bolting on awkward data features, elegant data products should also be designed to interoperate with other data products. With interoperability built into the design, data products can be easily combined to create richer and more valuable super data products.

Auditable

As data products evolve, some changes will be improvements, but not all. Changing data products can also expose vulnerabilities, such as security and compliance risks. To ensure that data products are of the highest quality, they must include audit trials and versioning data. Quickly identifying errors and pinpointing the source will help to keep your data product running securely and efficiently.

Use-Case Driven

To be comprehensive and consistent, data products need to solve users’ problems effectively every time. To do this, like any product, they need to be designed with the end user at the center of the process. Whether the user is a data engineer, data analyst, business analyst, business executive, customer, or partner, having a comprehensive understanding of their needs is key for success.

Comprehensive data products also incorporate a wide breadth of data sources to ensure their coverage of use cases is not limited or inconsistent. Enriching data with partner or 3rd party data can add additional depth to your data product. For example, using zip code databases to fill in missing address data and standardize it can make data products more comprehensive and consistent.

Users must be able to clearly understand what the data within your data product represents to be applicable to their use-case. This can be a challenge as data originates from all over an organization. Proper metadata management is important in creating powerful data products and ensuring context is preserved. Making certain that users understand the terminology used to describe the data in the data product is also important. Incorporating business glossaries is one way to help standardize terminology.

Lifecycle Management

One of the biggest differentiators between data products and data projects is the performance of data products and their ability to be constantly improved and enhanced. Even if we do our best to design a data product to meet the needs of our audience, it will not always hit the mark or simply require change. Building a mechanism to capture feedback from users is key to continuously delivering great data products.

Tracking data products and understanding how they resonate with users is key to connecting products with users. A data product marketplace littered with aging and irrelevant data products does not lend itself to an elegant process. Data products can be archived and retired marking the end of life for these products, reducing noise. Ensure that you curate your data product marketplace to optimize the experience.

Process

Elegant data products do not just happen on their own, the right process needs to be in place to support the creation of elegant data products. Without it, there is a tendency to add more data that adds complexity. A process ensures data is added deliberately. Elegant designs are produced by iterative and collaborative processes.

Iterative design processes support elegant design because each step or cycle gets you closer to a simpler, more powerful solution. The first versions of data products may not be the optimal solution, so they need to evolve. Features that are unused or disrupt the path to the best outcome can be eliminated through iteration. New users can find innovative applications for data products that spawn new features or a split from the original data product into something new and more impactful. Your process should embrace and institutionalize feedback to better understand how your data product meets its objective. As data products evolve, and feedback is collected, ideas emerge for new data products.

Building great data products is no small feat. Doing it from scratch without a solid technological foundation can be even harder. Data product platforms can make the process much easier. Extrica is a modern data analytics platform that is designed from the bottom up to streamline the creation of data products. To learn more about the capabilities of Extrica and how the platform can help you create elegant data products schedule a demo.

Get in touch to unlock the real potential of your data!

Trianz would be pleased to set up Extrica demo for you and conduct proof of value to showcase the benefits of Extrica.

data mesh lab