This post continues the examination of the need to conduct an initial quality check after utilizing a calculation process for one or more Key Performance Indicators (KPI) for the first time. In part 1 of this blog topic (see Part 1 in the previous entry (post #9)) an overview of the purpose for the quality check was presented as well as a specific focus on how to examine the business rules used. In part 2 of this blog topic we examine the fidelity of the algorithm used to calculate the KPI value as well as referencing the point in time of the data used for such calculation.
Algorithms Correctly Respond to the KPI
In the eighth blog post of this series entitled Initial Run Through, the concepts of area and perimeter were referenced. These concepts illustrate the part of the quality checking process specific to examining that the correct algorithms are used to calculate a particular KPI. The area of a shape is specific to certain aspects of it’s characteristics. In general, two numbers are going to be multiplied together, the length and width. This is straight forward when finding the area of a parallelogram. Webster’s online dictionary defines a parallelogram as “a quadrilateral with opposite sides parallel and equal”[i]. The means of calculating the area of a trapezoid is different. Webster’s defines a trapezoid as “a quadrilateral having only two sides parallel”[ii]. While the trapezoid also is a quadrilateral, its characteristics are slightly different then a parallelogram, and thus the formula for calculating the area is also slightly different ((base a + base b)/2) X (height). The point here is that the algorithm may appear to be calculating values with great precision; yet the team needs to be sure that the precision calculated is specific to the KPI assessed.
The Date Range is Reflective of the KPI Parameters and a Frozen File
“Ladies and Gentlemen, may I have your attention in the terminal. Generic Airlines Flight 1234 enroot to Rochester, NY is in the final stages of boarding for an on-time departure. The doors will be closing in 10 minutes and will not reopen.” Sound familiar? This page over the PA system is most prevalent in two situations where people are rushing to make their connections:
At this point in the performance year, there are still situations of data collection for the fourth quarter. As we move forward, this reality is recognized, and we will work through how to make progress while data is still coming in for this last 25% of the performance period; however, by this time, all data for the first, second and third quarters should be complete. Additional data should not be in the process of being added for the first three quarters. If this is the case where data is still being modified, then the team needs to deem the data set as not being ready. It is recommended that the strategies outlined in blog posts #4-7 be utilized to validate these three quarters of data once finalized.
The data set that was used to conduct the initial run through of data should be date stamped as to when the data set was extracted, from what system, and by what processes or people. All of this information should be included in the data dictionary that accompanies the data set. Additionally, as the calculation of a KPI may become more discrete, any subsets of data created should also contain timestamp information and be included in the data dictionaries of the subset(s) created.
An Unexpected Surprise
As the team continues to work with data sets, it is quite possible that an unforeseen glitch in the data may arise. Such a glitch may occur even after great care has been demonstrated to check for completeness and quality of the data. A common glitch that occurs at this stage is the sudden discovery of a significant set of data that has not yet been entered into the application storing the data needed. As a result, the team is now facing the reality that the data set currently being used to assess a KPI(s) is incomplete. The team should discuss the feasibility of back tracking the analytic process to include this information. Alternatively, the team may need to speak with other data leaders as to proper courses of action which may include the discontinuation of analysis for a specific KPI or proceeding with limited information. Moving forward known limitations raises a value statement once again: Do limitations that have surfaced are overpowered or outweigh the potential conclusions?
Blog #10 Question: Does your initial run through of data align with the intent of the KPI?
Blog #11 Preview: First Quarter Analysis
[i] Source: https://www.merriam-webster.com/dictionary/parallelogram retrieved 11/25/2017.