Executive Summary
These key metrics can be helpful clues to whether your organization's data projects are successful: Any data in your organization should be accessible anytime, anywhere, or without silos, and you can monitor it.
Your data team and their systems should be able to deliver one new insight from the data in one business day.
There is no repetitive work to get insights of a similar or periodic nature. For example, if you must obtain daily sales reports and predictions by creating a data pipeline once, you can enjoy these reports daily without your data team having to rework them.
Cost-effective or scalable. The costs required to obtain additional insights are not linear or exponential to these extra costs. For example, you have ten dashboards that cost $100, so the costs incurred shouldn't be too big when you want to increase it to 20 (<< $200).
Data is protected and properly managed, meaning that data can only be accessed by those entitled to it.
Data and insight are keys to winning in the business world. It can answer many important business questions, from the root cause of ineffective production cost, marketing cost, customer loyalty factors, factors that drive most revenues, and what should be anticipated soon for a business to stay healthy. But, when data scatter across departments, it can be a daunting task. This data scatter is why some companies end up in a costly but ineffective data project.
Every company can end up with data silos. The silos happen for many reasons, from company culture to technical aspects, mainly because each department has a different style of using, storing, or processing data into insights. According to a Starburst and Red Hat report, more than half of businesses (52%) use around five or more platforms to store their data. The use of multiple platforms causes factors that make data difficult to obtain or gain insight because each platform most likely has different input and output formats. One software and another do not have a direct data connection, and they have their way of handling data in it. So a successful data project must deal with these silos that make data accessible anytime, anywhere, and you can monitor it.
As stated in the same report, 60% of companies need more than one working day, even 27%, to create a data pipeline in three and sixty working days. But, in a company with no data silos, as mentioned in the first paragraph, you can access data anytime and anywhere. So the data team's work becomes just three steps: 1. Accessing the data needed in the project 2. Creating a pipeline 3. Presenting results according to requirements So the data team should complete one data pipeline project in one working day because the faster the data pipeline project is completed, it will incur the fewer costs.
Ideally, you can monitor your business situations daily through company data, especially if growth or changes in business conditions every periodic time is vital to know. Of course, it depends on how fast your business cycle is, so you and other executives can quickly make decisions. Loss of business opportunities can be avoided, especially if the data shows what you should do. For example, the data project must obtain daily sales reports and predictions by creating a data pipeline once. You can enjoy these reports daily without your data team having to rework them. In these situations, data teams and their systems must have automatic synchronous capabilities that allow the same pipeline to run again when new data inputs are available without the same channel having to be recreated or manually rerun again and again. The reason is apparent, cost. So, there should not be repetitive work to get similar or periodic nature.
Your company data project must be scalable, meaning that most of the cost is building the system, not using the system. When the data system is ready, it can handle as many projects as you like without linearly or exponentially increasing the cost. And the price of using the data system, let's say, to hold ten dashboards will be similar to when taking 100 dashboards or more.
Data security can be a nightmare for everyone, especially regarding customers' data, and you have no right to make the data public, so security is mandatory. You have to make sure all security concerns are checked. Even so, this system should not limit or reduce the activity of those who have authority over it. And also, the system must be able to restrict or share data access with anyone who has the right. Apart from the technical side of security, it's a good idea for your data system to have activity records or logs of data access so that information such as who, when, and what can be traced; is very helpful in tracking data security.
Many data projects (85%) fail (Gartner, 2017). Therefore, a clear understanding of the difference between successful and unsuccessful data projects of the enterprise helps increase the potential for success.
By Mujirin, S.Si, M.Si - VP Digital Transformation
References:
*Starburst (2021):https://www.starburst.io/info/2021-state-of-data/
*Gartner (2017): https://designingforanalytics.com/resources/failure-rates-for-analytics-bi-iot-and-big-data-projects-85-yikes/
