The Value of Small Data

In “How Data Democratization Can Unlock Game-Changing Business Performance” we mentioned that organizations can see huge benefits from starting small and using common back-office applications such as spreadsheets to analyze data. However, starting small is not only about people, tools, and processes; it has also a lot to do with data, and it spans areas beyond only business decision-making. Small Data can also help organizations create data models without having to harvest large volumes of data to train the models.

Bigger is not necessarily better

Small Data is not a new concept, but it has been overshadowed by the more catchy and buzzy term of Big Data and all the attention it has captured in recent years due to its key role in developing AI systems and advanced analytic platforms. Each has its place, and it is key to understand what they are good at and when to use them to better leverage their potential.

Big Data is all about volume, velocity, and variety. When data comes from multiple sources (transactions, IoT devices, industrial equipment, sensors, social media feeds, video cameras, audio recordings, images, etc.) so fast that systems can barely keep up, it then becomes impossible for a human to understand that data, let alone identify patterns and insights that can be useful in any way. Instead, analytic platforms are used to crunch that data and reveal patterns and identify insights that are relevant to the business. Not all data is useful and therefore it becomes crucial to have a data strategy that helps organizations choose which data is relevant before collecting it and analyzing it. At the end of the day, it is the job of Data Engineers and Data Scientists to produce value out of all that data, business insights, and actionable information, and then share it in the form of visualizations and reports.

Small Data, on the other hand, constitutes a small set of data that can be easily captured and managed using standard back-office applications such as spreadsheet software. Such small data sets can come from a subset of already curated big data chunks or from specific business data sets that aren’t big enough to require further processing. A Small Data dataset provides enough information to analyze problems, identify patterns, and discover insights just by looking at it and visualize it in different ways, mostly by leveraging the already powerful charting and visualization capabilities of spreadsheets.

Small Data – Big Insights

What Data Analytics reveals will always depend on what the Data Scientists are looking for or program into the reports and visualizations they create. The business leaders’ understanding of the organization, market conditions, competition, and everything that affects the business is not near being replaced by those tools or AI software. Intuition and cognitive understanding of the business is still critical asset for business leaders, and Small Data enables these leaders to determine if they are doing the right things at the right time. Small Data provides execution excellence by providing an instant understanding of what is going on. With the advantage that the data can be manipulated and visualized in any way that they desire, because they know how to relate variables in such a way that Data Engineers and scientists can’t, given that they don’t have that clarity about the business.

Leveraging Small Data the right way

There are two main challenges business leaders face to get the most out of those small datasets spread across their organizations:

1. Effective use of software: Being able to pull data into Excel and organize it can be challenging. To gain insightful information it will generally be required to perform calculations, match different data sets to correlate data, and then create visualizations a bit more complex than a simple bar chart. Investing time in how to efficiently use Excel will speed up the analysis and decision-making processes.
2. Data organization and availability: Data may be stored behind a closed system and some database tools or API scripts may be needed to pull the data. Other data may be spread in Excel files on users’ computers or in the cloud. Having a structured data policy in place is a must, as data centralization and data quality is the cornerstone for any further analysis.

Beyond any technical challenge, the real one is having the bandwidth (time and mental) to work with the data and uncover those insights only someone with the knowledge of the business can uncover.

Key Takeaways

Using fancy names to refer to common tasks and processes may prevent some people to do what intrinsically these concepts attempt to achieve: leveraging data.

Anyone can leverage data, but depending on the volume you may require additional tools, skills, and technology. For the general case, in business contexts, there is a high chance that pulling up Excel and working with the right data will take you very far.

About Encora

Fast-growing tech companies partner with Encora to outsource product development and drive growth. Contact us to learn more about our software engineering capabilities.

Leverage Encora's Partnership with Snowflake