Home > Statistics > Trend Indicator of Output > 2009 > Examination of the accuracy of the Trend Indicator of Output

Examination of the accuracy of the Trend Indicator of Output

The former Monthly Indicator of Total Output has been revised and is now published under the new name of Trend Indicator of Output. The development of the Indicator was done during 2008 alongside its routine calculation without breaks in publishing. This report endeavours to evaluate how well the Trend Indicator of Output would have anticipated the initial quarterly accounts data on value added during 2007 and 2008 with the revised calculation method that became available at the end of 2008.

Due to the limitations imposed by the source data the calculations described in this report do not cover all industries included in the usual calculation round of the Trend Indicator of Output. The time span selected for the examination was the period of eight quarters from the beginning of 2007 to the end of 2008 because quarterly accounts have only been calculated with the presently used method starting from data concerning 2007.

The structure of this report is as follows: the research frame is presented first, inclusive of the demands it imposes on the data and the limitations that arise from them. This is followed by a presentation of the results. Generalisability of the results is discussed in the concluding chapter.

1. The research frame

The objective in these examinations is to investigate as closely as possible the accuracy of the Trend Indicator of Output by means of comparisons with results from genuine calculations. Because the statistics were developed alongside the calculation process throughout 2008, comparing the released figures with quarterly accounts would not have described the situation at the end of 2008 with the exception of the last quarter of 2008. The adopted solution was to use the revised calculation method with the data that according to the release schedule would have been available for each calculation round. Thus, the data on the last quarter are the only ones based on calculations of the published Trend Indicator of Output. The monthly calculations of the quarters prior to it have been made retrospectively with the data that according to the release calendar would have been available at the time of performing the calculations.

In accordance with the research frame, the initial, unrevised monthly data were gathered from the source data of the Trend Indicator of Output on the industries for which they were available. In practice, this meant that for service industries in the public sector, for which calculations are based on wage and salary sums in the Tax Administration's payment control data, unrevised data were not available so they had to be left out of the examination. Monthly source data on primary production seldom become revised so in respect of primary production the accuracy of the Indicator was examined by using in all calculations the source data that were used in calculations in December 2008. The effects of these limitations on the generalisability of the results are discussed in the concluding chapter.

Among the examined industries, the calculations for renovation building and financial and insurance activities had to be based on estimates of experts, as no direct, reliable source data are available. It is impossible to make such expert estimates retrospectively without taking into account data on real development or data having come to light after the reference period of this evaluation. For these industries change in the value of output had to be calculated basing on assumptions. In renovation building it was assumed that up to the first quarter of 2008 the volume of output would go up at a steady rate of 3.0 per cent and then continue to rise at the rate of 3.5 per cent compared to the respective month of the year before. The annual change in the development of the value of output in financial and insurance activities was assumed to always be the same as the annual change in the last quarter of the previous quarterly accounts. These standard assumptions do not fully equate to a genuine calculation procedure. However, correct or "suitably" wrong data could not be used because then the accuracy for these industries would habe been determined direct by their selection. Instead, the above assumptions about development could have been realistic expert estimates and thus introduced to the calculations uncertainties which evaluations of development in renovation building and financial and insurance activities are subjected to even during each genuine calculation round.

As a rule, the producer prices used in the Trend Indicator of Output do not become revised so for their part the price series used in the calculations of data for December 2008 could be used in all calculation rounds.

The monthly time series of the Trend Indicator of Output are benchmarked to the value added series of quarterly accounts. This has been taken into account in the calculations.

It is possible that the source data used in the examinations were of somewhat poorer quality that those used in a real calculation situation. Especially in manufacturing there are industries that require special attention, often because of a couple of particularly dominant enterprises. In these cases the source data are formed by resorting to estimations and expert evaluations in addition to mechanical calculation processes. The initial data on 2007 obtained for these examinations had not been calculated quite this precisely, as these methods were not yet used, or not at least as systematically as today, when the source data for 2007 were calculated.

2. The results

The calculation of the Trend Indicator of Output is based on indicators that describe the development of output either direct or indirect, and on corresponding data on prices. However, in quarterly and annual accounts the output approach focuses on value added by industry and on gross domestic product, which is received by adding taxes on products to the value added of the whole economy and by deducting product subsidies from it. Although intermediate consumption is not taken into account in the calculation of the Trend Indicator of Output, it should ideally be capable of predicting value added expressly in quarterly accounts. This is why the accuracy of the Trend Indicator of Output is assessed by using the initial quarterly accounts data on value added as the point of comparison.

Because for the time being there are only eight quarters in which the data of the Trend Indicator of Output can be compared, no efforts are made here to draw conclusions about the statistical significance of the results. The calculated parameters are only initial, descriptive variables to enable the formation of a tentative perception of possible bias or some other inexactitude.

The main emphasis in the presented results is on the initial, unrevised data of the Trend Indicator of Output, from which the estimated development in a quarter is obtained by summing up. The secondary interest is in the so-called flash estimate of quarterly accounts, which is based on the Trend Indicator of Output, or to put it more precisely on the released data on the third month of each quarter. In them the data on the first and second quarters have already become revised to some extent but the data on the third quarter are based on the initial, unrevised source data. The flash estimate is not published as instant data on a quarter but is supplied as such only for Eurostat's needs.

The accuracy has been examined with growth percentages describing development since the respective point in time in the year before. The deviations presented next have been calculated by deducting the annual change of quarterly accounts calculated with initial data from the annual change indicated by the Trend Indicator of Output for a quarter. Thus, positive deviations mean that the Trend Indicator of Output has overestimated the development in the quarter concerned while negative ones point to underestimation.

2.1 Value added of whole economy

The obtained average for the deviations was -0.2 percentage points. Thus, basing on these calculations the Trend Indicator of Output would have underestimated the development of value added in the national economy in 2007 and 2008 by 0.2 percentage points, which being close to zero does not indicate a clear bias in either direction.

Irrespective of the sign, the average deviation was 0.6 percentage points, so the Trend Indicator of Output cannot be regarded as a precise predictor of quarterly value added. In three of the eight quarters the absolute value of the deviation of the initial data was of the magnitude of around one per cent.

Under the presumption that the service industries administered by the public sector develop more evenly than those of the private sector and the comparisons could have been extended to them, the deviations describing the accuracy might look more favourable. The weight of public sector services in the value added of the whole economy is good 21 per cent. Accuracy concerning this can only be ascertained once more data accumulate.

Figure 1. Deviations of Trend Indicator of Output as percentage points, development of the whole economy. These comparisons could not be extended to service industries administered by the public sector.

Although the flash estimate is based on a larger amount of information than the estimate of quarterly development compiled from initial data, it did not prove to be more precise in predicting quarterly accounts during the examined period. The obtained average for the deviations describing the development of the national economy was +0.4 percentage points. Figure 1 shows that over the examined period the flash estimate almost always indicated larger growth than the growth shown by initial data. Irrespective of the sign, the average deviation was 0.7 percentage points.

The initial data of the Trend Indicator of Output underestimated the last quarter of 2008 which, as already pointed out is thus far the only one genuinely calculated with the revised calculation method and the data sources used in it, i.e. inclusive of public sector services, by 0.5 percentage points (growth in quarterly accounts -2.7%, with initial data of the Trend Indicator of Output -3.2%). The flash estimate, again, overestimated the growth by 0.2 percentage points (growth according to flash estimate -2.5%).

2.2 Primary production

Estimations for primary production proved really precarious. The Trend Indicator of Output predicted especially poorly the second, third and fourth quarters of 2007 (respective deviations -7.9, +8.0 and -11.7 percentage points). Calculations for 2008 proved less inaccurate: the average deviation was -0.8 percentage points, and calculated from absolute values 2.4 percentage points. The average deviation for the entire examination period was -2.2 percentage points, and calculated from absolute values 5.0 percentage points.

Despite the unreliability of the results the sign indicating the direction of development was correct for all quarters. However, it should be borne in mind that in both years the third quarter was overestimated and the other quarters underestimated, so strong signals of systematic bias can be seen. This may be connected with the fact that data on farming crops are received annually and then arbitrarily divided between months as there is no volume indicator that could be relied on. For the time being, the crop is divided between the months of the third quarter of the year in the calculation of the Trend Indicator of Output.

Figure 2. Deviations of the Trend Indicator of Output in percentage points, primary production

The poor accuracy in primary production raises the question of whether data of this poor quality are worth publishing. Primary production will be included in the calculations even in future (weight in the whole economy 3% according to 2007 annual accounts). However, figures concerning primary production need not necessarily be published in connection with the Trend Indicator of Output if they cause outright problems to the users.

No separate comparisons with the flash estimate were made in respect of primary production because the source data do not become revised.

2.3 Secondary production

On the average, the Trend Indicator of Output anticipated the development in secondary production without bias, for the average of the deviations was near zero (-0.1 percentage point) but there was a clear difference between 2007 and 2008. The average for the deviations in 2007 was -0.9 percentage points whereas in 2008 it was +0.6 percentage points. Thus, development was underestimated for 2007 and overestimated for 2008. Partial contributors to this many have been the estimation methods introduced in the source statistics for manufacturing, which were not yet used when the source data were calculated for 2007. However, the volume of the data is so small that no clear conclusions can be drawn about the impact of the methods.

Figure 3. Deviations of the Trend Indicator of Output in percentage points, secondary production

The average obtained for the absolute values of the deviations over the whole examination period was 1.2 percentage points, so the Trend Indicator of Output would not have anticipated the development for secondary production very accurately. There were large deviations in the latter part of 2007 and early part of 2008 (the largest, +2.8 percentage points, for the 2nd quarter of 2008) which became considerably revised from the flash estimate (although the deviation for the 2nd quarter of 2008 still remained at +1.9 percentage points). When examining the absolute values, the figures were less accurate for 2008 (average 1.5 percentage points) than for 2007 (average 0.9 percentage points).

2.4 Services

The service industries administered by the public sector, including public administration, education, health care and social services, cover approximately one-third of all service activities. Unfortunately these had to be excluded from the examinations because unrevised source data were not available from the past. Under the presumption that public services develop more evenly than the services administered by the private sector (trade and other services), the accuracy may be better in a genuine calculation. The accuracy in the public sector services will also be studied further as more data accumulate.

Private sector services were underestimated in the examined period, for the average deviation was -0.4 percentage points. Because of the low number of compared quarters, at least in these examinations statistical significance was not yet tested. However, in the light of these results it seems that the Trend Indicator of Output underestimates systematically the development in private services, for seven out of the eight deviations had a negative sign.

Figure 4. Deviations of the Trend Indicator of Output in percentage points, services. These comparisons could not be extended to service industries administered by the public sector.

The initial source data on services derived from a direct data collection in which the emphasis was on large enterprises. The flash estimate, which in addition to the data from the direct collection also exploits data on small and medium size enterprises from the Tax Administration's payment control file, overestimates the development in private services as systematically as calculations based on the initial data underestimate it. It seems quite likely that the development of large enterprises deviates from that of small and medium size enterprises, which would explain the differences caused by the different sources and at least partly also the possible bias.

Apart from the possible bias, the prediction for private services was fairly accurate during the examined period, for the absolute values of the deviations were mostly quite low (fourth quarter of 2008 had the highest one, -1.1 percentage points). The average of the absolute values was 0.4 percentage points, with no significant variation between the years.

2.5 Relative accuracy

The deviations can also be examined relative to the growth figure of initial quarterly accounts data. For instance, if the deviation is -1.1 percentage points, the seriousness of erroneous interpretation caused by the deviation depends on whether the growth figure of quarterly accounts at the same point in time is -12.1 per cent (primary production, 2nd quarter 2008) or -1.6 per cent (services, 4th quarter 2008). In the latter case the absolute value of the deviation is over one-half of the absolute value of the growth figure of quarterly accounts, so this kind of a deviation inadvertently distorts the depiction of the trend.

Table 1 shows the deviations with absolute values of at least one-third of the absolute value of the growth figure of quarterly accounts. Most of these are deviations relating to primary production and can be explained with difficulties in allocating the received annual data on the output of crop production to the right time periods.

Table 1. Observations with largest relative deviations of absolute value from growth figure of quarterly accounts

Time series	Time period	Trend Indicator of Output, initial data (%)	Quarterly accounts, initial data (%)
Whole economy	2008Q3	2.1	1.4
Primary production	2007Q2	10.4	18.3
	2007Q3	13.8	5.8
	2007Q4	4.5	16.3
	2008Q 3	-3.0	-6.1
	2008Q4	-13.8	-9.6
Secondary production	2008Q2	7.2	4.4
	2008Q3	3.1	2.1
Services	2008Q4	-2.7	-1.6

 

When the whole economy is examined, one large relative deviation can be found in the third quarter of 2008. We know from quarterly accounts (at the time of writing this) that in the quarter concerned a turn occurred in the trend, which pushed Finland's economy into recession. The calculation method of the Trend Indicator of Output is incapable of reacting quickly to structural changes in, for instance, intermediate consumption and this could have been one of the reasons for the magnitude of the deviation.

In the second and third quarters of 2008, the time series on secondary production showed larger growth figures than the ones arrived at in quarterly accounts. The above-mentioned structural changes could possibly also explain these deviations.

In the last quarter of 2008, the time series on service industries showed a steeper decline than quarterly accounts. In section 2.4 above on results for services the initial data of the Trend Indicator of Output were suspected of being biased downwards. If this is true and the bias is caused by the data collection having been focused on large enterprises, even this large relative deviation could be put down to the same thing as above. Thus, if we follow this line of thought, large enterprises would have cut down their output by more than small and medium size enterprises.

3. Conclusions

Due to the low number of observations, the generalisability of the results is poor. Tests to prove the statistical significance of the bias will not be meaningful until data become available for a considerably longer examination period. Therefore, these results should only be viewed as additional, descriptive information about the statistics.

A further major problem in this examination of accuracy was that the comparing could not be extended to all industries. The absence of public sector services from the calculations means that the presented figures on accuracy only depict the ability of the Trend Indicator of Output to anticipate the private sector's share of the value added of the national economy. In the Trend Indicator of Output the industries included in the comparison cover approximately 79 per cent of calculations for the whole economy and 66 per cent of calculations for services. The weights are based on the 2007 annual accounts.

In the light of the presented results, the Trend Indicator of Output is capable of predicting the development of the value added in the national economy at least fairly well. Primary production was the only activity where the prediction was poor.

It would be pertinent to ascertain whether improvements could be made to the calculations concerning primary production. Figure 2 shows that in both of the examined years the third quarter became overestimated and the other quarters underestimated. The reasons for this may be that the data on crop production that are received as annual data are divided arbitrarily to the months of the third quarter in the Trend Indicator of Output. The congruence of the Trend Indicator of Output with quarterly accounts could possibly be improved with the help of an expert in farm accounting. Although some shortcuts were made in the calculations concerning primary production, these should not have any bearing on the obtained results.

In respect of secondary production it was observed that the Trend Indicator of Output underestimated development in 2007 and overestimated it in 2008. The overall predictive ability was also not quite as good as had been wished. The deviations should be monitored together with the source statistics to see whether some systematicity can be seen in them.

The Trend Indicator of Output seems to anticipate the development in services quite accurately as the deviations were uniformly small. Basing on Figure 4 it seems likely that the initial data of the Indicator underestimate while its revised data overestimate development systematically. This should be monitored together with the source statistics.

Section 2.5 on relative accuracy discussed certain possible structural factors that the calculation method of the Trend Indicator of Output is at the moment unable to take into consideration. For instance, no estimations are made in calculations for intermediate consumption, which becomes transmitted to the time series from quarterly accounts in connection with benchmarking. Solving problems like this would probably improve the accuracy if in consequence of the found solutions the calculation method becomes capable of reacting more quickly to structural changes.

The point of departure in the examinations discussed above was assessment of the quality of the source data of the Trend Indicator of Output. In reality, the accuracy could only be studied this simply if the calculation methods of the Trend Indicator of Output and quarterly accounts were fully identical. At least for the time being this is unfortunately not the case. Only once fully identical calculation methods are used in quarterly accounts and in the Trend Indicator of Output (complying with the output approach to gross domestic product) can the other factors be excluded from possible future studies and the examination focused on the quality of the source data.

---

Last updated 8.4.2009