Back in Blog Post #1, together we began the journey through this blog to assess the Key Performance Indicators (KPIs) that we labeled the KPI Assessment (KPIA) process. As the end of the annual performance period nears, an analysis team is like a fire battalion getting ready for the next emergency call. The analysis team will do whatever it can to prepare to complete an analysis in as timely a manner as possible while preserving the highest levels of accuracy. The last blog article outlined some considerations for calculating the quarterly values for each annual KPI. The question remains, how can you assess fourth quarter results and generate the final KPI value before the fourth quarter ends? The answer is quite simple: you can’t.
One of the wisest people I know says “focus on what you can, not what you can’t.” In that spirit, although it is not possible to calculate the fourth quarter and final KPI values at this time, you can project what they will be with a high sense of reliability.
Just before I began to write this blog article, I watched the election returns occur for the special senate election that in Alabama. I wasn’t watching commentary on a news station, or listening to the radio, instead I was watching real-time updates on the New York Times’s Election 2017 returns website. The data used on the site to allow for comprehensive monitoring was awesome. The data displayed was not only representative of what was happening at the moment, but there were various tools that the observer could use to interpret the validity of the votes that had been processed up to that moment. What you will notice when visiting the site is that there are more widgets to monitor the validity of the data than there are for analysis of the data itself. These verification tools are what make the site especially awesome. I viewed the site for more than an hour (beginning at 9:30 PM to about 10:45 PM (EST)). It was really interesting to watch the variation in the confidence of the data at any one moment. The site registered an update every minute whether new results had come in or not. Once 90% of the votes were counted the data verification widgets had little fluctuation, and shortly thereafter the race was called.
I bring up the New York Times election result site as a point of reference to think about when calculating fourth quarter values before the end of the fourth quarter. Actually, given that the fourth quarter is not yet completed, you won’t be calculating fourth quarter results, rather you will be projecting the anticipated value. Projections are nothing to take lightly. They can be very dangerous if interpreted incorrectly and are subject to change. The main purpose of a projection is to give you an idea about the direction that something is headed toward. Is the value going to me more or less likely X? When making a projection as to where the fourth quarter may end up, the quality of the data set should be in better shape than real time election results pouring in. Real time data analysis has to factor in a level of uncertainty that is acceptable. Given how close the race was in the December 2017 Alabama special election, 90% of the votes needed to be in for teams of analysts to be comfortable with making a call about a winner before the remaining votes were tallied. Unlike processing a data set that is changing in real time, examining a data set that is still active, yet with the majority of the data collected, the levels of uncertainty should be less. With only 15 days or so until the end of the fourth quarter, out of a total of 92 possible days (84% of the quarter has been completed),[i] there is sufficient data to make an informed projection for the fourth quarter.
Fourth Quarter Projections
Having sufficient data does not mean that a projection is always a good thing to use. A projection is not a guess. As defined by Merriam Webster[i], a projection (as related to statistical analysis) is “an estimate of future possibilities based on a current trend.” Whereas a guess[ii] is defined “to arrive at a correct conclusion about by conjecture, chance, or intuition.” One is based on a series of facts from the past that estimate the future, whereas the other is based on conjecture and intuition, a true chance. Therefore, projections require methodology and past evidence. When projecting the values of the fourth quarter to best prepare timely evaluative analysis there are four things to check for:
Please note that this can be a tricky situation. By considering that enough time has passed also assumes that data has been readily collected. If data has not been collected, then the validity of the projection will be lower because it will solely rely on past performance data.
Once the fourth quarter KPI values are calculated and subsequently the year-end KPI, the team is ready to identify the limitations, deductions, completeness, and special circumstances. Both a limitation and special circumstance for these values is that they are projections. Like quarters one through three, each team member should sign-off on the projected value for the fourth quarter and for the year as a whole signifying that the prescribed business rules were followed, the values were cross referenced, and notes relevant to the KPI value(s) are recorded correctly.
With these projections in place, the team can meet to do one final data quality check. The steps involved, and the next phase will be discussed in the next blog article #14.
Blog #13 Question: Is it ok to project?
Blog #14 Sneak Peek: The KPI Assessment wrap-up.
Resources as cited:
[i]https://www.merriam-webster.com/dictionary/projection retrieved on December 13, 2017
[ii] https://www.merriam-webster.com/dictionary/projection retrieved on December 13, 2017
[i] Even when considering business days only, out of a total of 63 business days in the fourth quarter and 9 remaining 85% of the fourth quarter has been completed. The one percentage point difference when compared to all possible days is not significant.
Now that the first quarter Key Performance Indicators (KPI) have been calculated and reviewed, and the purpose of the first quarter’s results have been identified, it is time to move on to the remaining quarters. Similar to the analysis for the first quarter, the second and third quarters should be analyzed in the same fashion. The data for the first through third quarters should be considered frozen and ready for summative analysis (Refer back to blog articles five through seven for guidance on determining the readiness of a “frozen” file.) The fourth quarter; however, will require a different process for review as it is still an active data set, since it is the middle the fourth quarter when this process has begun and does not end until the thirty mark on the timeline outlined in Blog #1. Therefore, the fourth quarter’s data is still being collected and verified.
After running the data for each KPI for the first quarter there are various items the team should consider:
Each of these considerations should be examined when completing the calculations for quarters two and three as well.
Nothing is perfect. There is always some sort of caveat about any data point. At this stage of the process, the team should be looking for limitations related to the collection rather than the significance, or lack thereof, of the data used to determine the value. Let’s look at an example to illustrate the difference. The temperature hit 76 degrees Fahrenheit. This temperature was highly unusual for the city that it was measured in as it was February and usually the temperature is 27 degrees Fahrenheit. What caused this unusual temperature record?
Similar to the process outlined for limitations, the focus at this stage should be on the data collection process rather than the context which will happen during the KPI evaluation process. At this stage it is important to consider related data points that bolster the meaning of the KPI data point. Consider the following KPI:
By the end of the 2017 fiscal year, 85% of vacancies that arise will be filled with candidates who have a college diploma.
For this example KPI, let’s say that the actual percentage for the second quarter was 90%. Additional deductions that can be made could include the percentage of:
The team should once again, review the data used to calculate the quarter specific data point to check to make sure that is reflective of the data set. Data quality checks that were referenced in KPI Assessment (KPIA) blog articles five through seven, should be used to once again conduct a verification for completeness.
4. Special Circumstances
When being interviewed, a common (and quite effective) question asked is: “is there anything else that I should know?” Often this question ends up being a safety net that catches information that is pertinent and useful but might not have been anticipated. When thinking about special circumstances that should be recorded about each quarter’s data this question comes in useful. The data dictionary for the data sets that were used to calculate the KPI(s) is a great resource to explore notations made during the data retrieval and analysis processes. These notations may be important to explain in greater detail as potential limitations or notes that will help later-on in the evaluation process. Items that are not already listed in the data dictionary can also be added. For example, based on notes from the data dictionary, there were some concerns about data quality during the first quarter. When the KPIs were calculated for each quarter, there was more variation with the first quarter data than the second or third. Upon further investigation, the team recalled that the team member that was usually in charge of the data collection of the KPI in question was on maternity leave during the first quarter. The person who was filling-in to manage data collection in the first quarter did not have as much experience as the team mate on maternity leave. The quality of the data was noticeably better once the team member on maternity leave returned to work. When the KPIE team constructs a narrative around the meaning and implications of the KPI values calculated during the assessment process, this type of special circumstance may become an important footnote to explain certain variations in the data.
Once the value has been determined for each KPI for the first, second, and third quarters, and the team has identified the limitations, deductions, completeness, and special circumstances, the team needs to come to consensus on the finality of the values calculated. For each quarter each team member should sign-off on the value calculated signifying that the prescribed business rules were followed, the values were cross referenced, and notes relevant to the KPI value(s) are recorded correctly.
Although the end of the fourth quarter has not yet occurred, there are steps to tentatively complete the KPIA process and commence with KPIE. In the next blog article, we will focus on the fourth quarter.
Blog #12 Questions: Should there be limitations to quarterly data?
Blog #13 Sneak Peek: Fourth Quarter
“Shouldn’t we have already analyzed first quarter data?” Ideally, yes, as well as the second and third quarters. The best way to make use of a Key Performance Indicator (KPI) is to continually monitor its progress, thereby increasing the likelihood that the KPI will be met. Continual review of a KPI throughout the year allows for timely decisions to be made to improve performance in real time and increase the prospect that the annual goal is achieved.
The use of a KPI during the time of performance has a different purpose then when engaged in the Key Performance Indicator Assessment (KPIA) process. During KPIA, the focus shifts from the formative (monitoring) to the summative use of data whereby the final values of KPIs are calculated and verified. Once final KPI values are calculated and reviewed they are then ready for use in the Key Performance Indicator Evaluation (KPIE) process where an in-depth review takes place to provide narrative and insight regarding the summative KPI values.
The questions listed below are illustrative examples of those that arise when a process moves from the preparation phase to the action phase.
KPI Use Continuum
Determining where your organization is on the KPI use continuum is important in order to maximize the efforts that have already been invested in the KPI formative process.
As shown in Figure 1, organizations generally fall into one of four categories:
Category 1: The organization continually monitors KPIs
Category 2: The organization monitors KPIs quarterly
Category 3: The organization monitors KPIs annually
Category 4: The organization has KPI data but does not yet have a plan to analyze the KPI data.
Figure 1: KPI use continuum
An organization may fall in-between one of these categories, where it may be closer to one category than another with the exception of category 4. Category 4 represents an organization where there is no systematic process yet developed to use data to measure KPIs although the data may exist.
Organizations in categories one and two perform some sort of formative KPI process by the very nature of the time period associated with the time of KPI review. When it is time for the annual process, where summative information is determined, those that represent categories one and two have additional formative data that can be used to conduct a data quality check for the summative values that were calculated. Additionally, as will be discussed in the KPIE process, these values can add to the contextual narrative associated with the final numbers that were calculated. Similar to categories one and two, those organizations in category 3 simply need to follow the process outlined. Unlike categories one and two, category three organizations will not have formative data points available for comparison since they are just engaging in the review of their KPI data. A special blog within a blog will occur later on to address those organizations that identify with category four.
In addition to the final annual values calculated for each KPI, it is also important to calculate the values of KPIs for each quarter based on data that has been verified at the end of the year. Calculating quarter specific values will serve as important points of reference when establishing trends that occurred during the year.
Importance of First Quarter Data
Is the first quarter the baseline or the frontline? In many cases the emphasis of first quarter data is one or the other. When the first quarter is the baseline, then KPI performance during this time period is used to compare the performance of the remaining three quarters of the year. The first quarter is often used soley as a baseline value when a new initiative is launched at the beginning of the analytic period and tweaks to the process are made during those first three months. Alternatively, a baseline using the first quarter may be implemented when looking at a new initiative that is started in the second quarter of the year.
If the KPI is one that is routinely used, then the first quarter results may be the frontline value representing a value that is expected to sustain at similar levels the rest of the year. There is less mystery with a frontline number, it is usually predicated by historical data and is anticipated within a range of certainty. Baselines are often diagnostic and set a bar for growth or reduction expectations.
Even though the first quarter data may be a frontline, it often becomes more of a baseline as time moves on as it becomes a comparative data point. For example, consider that renewable energy consumption is a KPI for a company. The amount of solar energy the company can utilize is expected to remain consistent at X kilowatts of power a month due to being in a sunny location year-round. The first quarter produced X kilowatts of power as expected, but the second quarter yielded a decline in power consumption by one-fifth. The second quarter results were not expected. By the very nature of the unexpected second quarter result, the first quarter becomes a baseline value of what is expected versus what actually happened in the second quarter.
The team should make sure that the first quarter data is calculated and cross checked. The team will also want to make note of the following (all of these are described in the next Blog article (#12)):
Blog #11 Call to Action Question: Where is your organization on the KPI use Continuum?
Blog #12 Sneak Peek: The Quarterly