20 - Supervisory Board Newsletter October-December 2014
Executive Summary
With Resolution no. 17/20141 the Supervisory Board (SB) terminated verification of the reasons giving rise to Work Orders rejected by Telecom Italia due to the "Presence of a multiplexing device", after having identified a steady decline in the phenomenon.
Regarding the Complaint S02/13, concerning malfunctions of the Wholesale CRM, as a result of the improvements recorded with the help of the new indicator that shows the level of severity of the critical aspects of such CRM, the SB had ordered closure of the procedure with Resolution no. 15/20142. In November, however, there was a new - temporary - worsening, explained by Telecom Italia, during the hearing of 2 December, with problems on the IT systems. The SB therefore initiated an audit to acquire additional information on the excessive reschedulings of the EDD.
The deadline for closure of Complaint S03/13, originated by the Operator Welcome Italia which complained about an excessively high number of faults and deteriorations, has been rescheduled. In addition, a joint Telecom Italia-Welcome Italia technical team was also set up, with the participation of the SB as facilitator, in order to analyse and solve the problems reported. A first meeting was held in June at the premises of Welcome in Massarosa, followed by a progress meeting in September at the premises of the SB. A subsequent meeting was held in Ancona, at a Telecom work centre, in October, while another progress meeting was held on 21 November. At the proposal of the SB, the two operators have initiated an analysis aimed at identifying an agreed technical definition of "deterioration" and "disruption" in order to facilitate operations for both.
During the quarter, the SB obtained the usual documentation from Telecom Italia in relation to the Undertakings now due for compliance, together with progress reports on issues requiring further investigation.
As regards the Undertakings Groups nos. 3 and 4, the SB adopted, during the Board meeting of 28 October, Resolution no. 18/20143, with which it initiated a procedure aimed at identifying appropriate Key Performance Objectives (KPO) in order to allow the SB to provide more pertinent opinions on the performance of Open Access, as provided for by the Undertakings.
Telecom has sent the 2015 Annual Plan for the Quality of the fixed access network (Undertakings Group no. 5).
Finally, monitoring carried out by the Supervisory Office (SO) to check the correct performance of the Single Queue mechanism continued, with visits to the geographic areas of Sardinia, Campania, Naples, Basilicata, Apulia, Trentino Alto Adige and Friuli Venezia Giulia.
Monitoring progress on Undertakings
Undertakings Group no. 1
(start up of the New Delivery Process)
All the main OLOs have completed migration to the New Delivery Process; the only exception is Wind as regards LLU and WLR services.
Work Orders having passed through the single queue and closed with a positive outcome as at December 2014 were approx. 3,900 for the Asymmetrical Bitstream Service, 3,250 for the LLU and 250 for the WLR.
Undertakings Group no. 3
(establishment of a performance monitoring system for SMP services)
The SB considers it necessary to proceed with the rapid identification of a new basket of more meaningful performance indicators compared to the current one, given the changed scenario compared that of the approval of the Undertakings with Resolution no. 718/08/CONS; in order to reiterate this position, it sent a letter to the Authority on 13 October 2014.
The SB adopted, during the Board meeting of 28 October, Resolution no. 18/20144, with which it initiated a procedure aimed at identifying appropriate Key Performance Objectives (KPO) in order to allow the SB to provide more pertinent opinions on the performance of Open Access. Identification of the KPOs has been assigned to the University of Catania as the Certifying Body. The same University has also been asked to validate the elementary data used in the processes. This became necessary following the introduction by Telecom Italia of the new NGOM Delivery process system.
Undertakings Group no. 4
(guarantees of transparency of the monitoring system)
In the fourth quarter the SB received from Telecom Italia the monthly reports for the months of September, October and November 2014, as well as that for the third quarter of the year. During the period in question, some documents contained information deficiencies, for which the SB asked Telecom for justification. The Company explained that delays in measuring some data, in particular data linked to KPI 1-Delivery, were due to problems that arose during the switch to a new IT system currently being solved. Telecom therefore sent the updated reports complete with the previously missing data.
Undertakings Group no. 5
(guarantees of transparency of the Technical Plans for fixed access network quality)
As regards Undertakings Group n. 5, in the fourth quarter the SB received the actuals of the 3rd Quarter 2014 of the Technical plans for fixed network access quality.
Telecom also sent the 2015 Annual Plan and a rescheduling of the 2014 Annual Plan: the reason for the review of the document was a change in the desaturation plans of bitstream and ADSL services linked to the closure of commercialisation of DSLAM ATM and migration from ATM to Ethernet.
The SB also sent Telecom Italia a letter requesting information concerning the On Going Project, asking to have the data on the repetitivity of the first three quarters of 2014.
Undertakings Group no. 6
(guarantees of transparency of the Technical Plans for fixed access network development)
Telecom Italia sent the SB a new version of the actual data for the third quarter 2014 summary and the programme for the fourth quarter of the year. Compared to the previous version, the number of FTTCab cabinets envisaged for ONU devices was revised downwards: from the 3,352 initially planned in the original programme to 3,267. The programme for the first quarter of 2015 was also sent.
The number of Property Units reached by optic fibre in the primary section of the network (connected in primary) was considerably lower than the planned number (449,000 compared to 520,000), while progress with regard to the secondary section of the network was decidedly ahead of schedule (21,000 compared to 7,000 planned).
Undertakings Group no. 8
(integration of the regulatory accounting and determination of the transfer charges)
In October, Telecom Italia sent to AGCom, and published on the Wholesale portal, the new Service Contracts of copper access services for 2015.
In December 2014, Regulatory Accounting checks for the year 2011 were completed, while those relating to Regulatory Accounting for the year 2012 are in progress, both for the fixed as well as mobile networks.
A procedure was initiated under Resolution No. 1/12/CONS concerning the “Identification of Regulatory Obligations for Access Services for Next Generation Networks” for setting internal transfer identification and application criteria for access services on next generation networks. In December 2014, however, this procedure had not yet been initiated. A decision is also pending concerning the implementation rules for definition of the transfer charge in relation to interconnection services for IP-based traffic.
Undertakings Group n. 12
(obligation to report the activation of services not requested)
Telecom Italia has sent the SB a report on the figures for Q3 2014 relating to complaints of the activation of services not requested, received by Open Access technicians: no unrequested activation was made.
Investigation proceedings initiated
Complaint P01/14 - Identification of specific KPOs (Key Performance Objectives) concerning the internal/external equality of treatment monitoring system - Implementation of Undertakings Group no. 3
The SB has initiated a procedure aimed at identifying Key Performance Objectives (KPO), as provided for by the Undertakings, to determine tolerance thresholds considered acceptable regarding performance differences recorded in the delivery of SPM Services to the Retail and Wholesale segments by Open Access.
The Telecom Italia Undertakings envisaged the identification of specific Key Performance Objectives (KPO) which, compared to the performance highlighted by the KPIs, were to have provided indications on the quality levels of Open Access activities in the provision of its services. These KPOs were to be identified annually by the Authority, but so far this has not happened. The SB nevertheless believes the KPOs to be essential for a proper evaluation of the performance of Telecom Italia with reference to the principle of internal-external equality of treatment.
For this reason, in the Board meeting of 28 October 2014 the SB approved Resolution no. 18/20145, initiating the procedure "P01/14 - Identification of specific KPO (Key Performance Objectives) concerning the system for monitoring internal/external equality of treatment - Implementation of Undertakings Group No. 3 on the establishment of a performance monitoring system for SPM services". The objective is the definition of internal KPOs capable of reinforcing the monitoring activities carried out by the SB. The Supervisory Board also points out that such activity will be suspended if the Authority does not proceed with a similar activity.
Also regarding the KPO identification process, it was necessary to identify a specialised certification body able to identify, for each KPI, a range of oscillation ("tolerance threshold") of the differences between the performance recorded on the Retail segment and those on the Wholesale segment, within which the process performance can be considered equivalent, also on the basis of statistical evaluations. The SB saw fit to select, in continuity with the work already started, the University of Catania. On 2 October 2014 a first meeting was held with the certifying body to start the validation of both the data resulting from introduction of the NGOM system, as well as the KPOs, while on 6 November a first kick-off meeting on the topic was held with Telecom Italia and subsequently a working programme was defined.
Complaints from Alternative Operators
Complaint S01/13 – Fastweb/Network access discrimination in the installation of LLU and Bitstream systems
Fastweb had notified the SB in February 2013 of alleged violations of Undertakings Group n. 1 with reference to the following two cases:
- the alleged inconsistency between Open Access' refusal to activate installations for Fastweb due to excessive distance from the exchange, followed by the activation of the same type of installations for Telecom Italia retail customers.
- the alleged inconsistency between Open Access’s refusal to activate an LLU line for Fastweb, followed by the activation of the same service for Telecom Italia retail customers.
With Resolution n. 5/20136, the SB began a review of the merits of the allegations made by Fastweb. The Supervisory Office (SO) performed on-site inspections to verify the actual presence of the reported equipment. From the analyses conducted by the SO, it emerged that in the vast majority of the rejected orders the equipment was effectively present; however, in some cases, while the issue of a KO was in fact correct, it was due not to “presence of equipment”, but rather for the most part to errors in the compiling of the order by the OLO. A number of cases were found in which the KO appeared to be unjustified, almost always originating from a misalignment of the Telecom Italia computer systems. Following a specific request by the Authority, the SB sent AGCom the documentation collected during visits and subsequent analysis carried out by the SO.
Pursuant to its Internal Regulation, the SB then sent Telecom Italia a preliminary notification (Resolution no. 18/20137), attaching the final Technical Report on the on site inspections conducted. After consulting Telecom Italia and Fastweb in two separate hearings, during its Board Meeting held on 9 December 2013, the SB passed Resolution no. 25/20138 in which it determined the closure of the procedure, ruling that based on the checks made, the KOs issued by Telecom did not show any conduct such as to represent breaches of the principle of equality of treatment between Telecom Italia Retail and the OLOs.
At the same time, the SB made a number of Recommendations to Telecom Italia, including a request to take suitable action to eliminate problems related to the less than full reliability of the LLU pre-sale analysis tool and reserving the right to being specific monitoring activity on rejected Work Orders.
Simultaneously, with Resolution no. 28/20139, passed at the same Meeting held on 9 December 2013, the SB decided to have a specific analysis conducted on refusals to activate LLU equipment due to presence of network multiplexers, in order to assess the effectiveness of the action recommended and check any breaches of the Undertakings.
During the 2014, Telecom provided the updated data on the progress of the process of de-commissioning of the equipment, with the split between "LLU areas" and "non-LLU areas”. Subsequently, it also sent to the SB the progress of the monthly incidence of the KOs due to the presence of equipment on lines not active due to LLU and BTS WOs from which a gradual reduction of the phenomenon was seen. Subsequently, the SB decided to proceed with the closure of this verification activity with Resolution no. 17/201410, adopted during the Board meeting 28 October.
Complaint S02/13 – Fastweb/Malfunctions of the CRM system for Wholesale customers
In 2013, Fastweb reported malfunctions to the SB in the Telecom Italia Wholesale CRM system. The origin of these malfunctions was to be sought in the series of various system releases and are thought to have originated critical aspects including the blockage of multiple Wholesale Work Orders.
The SB had opened a complaint with Resolution n. 6/201311, inviting Fastweb to provide more details about the critical aspects in question. In June 2013, also Vodafone sent a letter containing other elements of analysis on the same subject.
The SB had asked Telecom Italia to provide detailed information to investigate the events covered by the complaint. Telecom sent numerous files containing detailed information on the anomalies found on the systems during the periods reported. The SO examined the material received, which describes the anomalies in releases 2.0 and 3.0 of the CRM, relating to the backlog of order processing and CRM outages.
During the hearing on 16 October 2013, Telecom Italia described to the Supervisory Board the plans for release of the Wholesale CRM and the action taken to improve the platform. In November, Fastweb confirmed during a special hearing the persistence of the critical aspects already highlighted; The Supervisory Board inasmuch recognised the need to define an indicator suitable for objectively measuring the level of operation of the Wholesale CRM system.
During the meeting held on 9 December 2013, Telecom Italia proposed to the SB a new wholesale order percentage indicator which, at the date of measurement, had “in process” status in Wholesale CRM, but no formal order sent to Open Access.
on 11 June 2014, a new hearing was held during which Telecom provided further details concerning the timing for solving the critical issues and the modes for collecting the data used by the foregoing new indicator as requested by the SB. The initial results of the new indicator nevertheless evidence the persistence of the critical aspects reported; therefore, on 16 June SB sent a letter to Telecom requesting information about action plans and times for the implementation of envisaged interventions, with periodic updates for measurement of KPI values and indications of the volumes of correlated unresolved and backlog volumes by individual service.
These periodic updates showed a marked improvement in the situation; therefore, after numerous extensions of the date for conclusion of the investigation (Resolution no. 15/2013,12; Resolution no. 29/201313; Resolution no. 14/201414), the SB closed the procedure, with Resolution no. 15/201415, passed during the Meeting on 24 September. With this Resolution, deeming that the malfunctions do not constitute breaches of the Undertakings, in view of the fact that the said Undertakings do not govern or require a precise obligation for results as far as the quality of the services delivered by Wholesale CRM are concerned, the SB recommended to Telecom Italia to provide a procedure suitable to check the effective operation of the software, asking the Operator to send weekly reports of the Wholesale CRM supplemented by specific analyses on the Work Orders for the period.
Thanks to the operational indicator of the CRM system for Telecom Italia Wholesale customers, the SB continued to keep the performance of said system under constant observation in order to rapidly and efficiently detect any operational problems which, if not identified and resolved in a timely manner, could have constituted a source of problems such as to jeopardise the principle of equality of treatment.
Precisely within the scope of weekly monitoring of the operational indicator of the Wholesale CRM system, the SB identified two episodes of an abnormal increase in the backlog of Work Orders for LLU LA services on 3 and 10 November 2014 of which it gave timely notification to AGCom with the communication of 25 November 2014. In order to acquire further evidence concerning the various issues regarding this sudden increase in the Work Order backlog, on 2 December 2015, the SB held a hearing for Telecom. During the meeting, the incumbent confirmed that on the above-mentioned days there was an effective and abnormal increase in the number of orders in the backlog in the pre-delivery phase for LLU LA services.
In particular, the first abnormal episode on 3 November 2014 was due to an anomaly in the Wholesale CRM system on 2 November, caused by the incorrect population of a field in the interface to other systems in the chain. The problem was identified and, as a result of intervention by the competent function of Telecom Italia, solved; the improvement was in fact also evident from the analysis carried out by the SB which found, in the case in point, that the value of the indicator had decreased from 17.39% on 3 November to 6.99% on 7 November.
The second episode of a sudden increase in the order backlog was detected on 10 November 2015 and was due to an anomaly in the systems external to the Wholesale CRM system. The identification of this anomaly (that was not systematic) and its subsequent resolution were particularly complex. Two separate interventions were, in fact, required. The first, implemented on 21 November, and the second on 28 November.
Pending final solution of the problem, Telecom Italia provided workarounds which mitigated the negative effects of the problem, as also demonstrated by the decrease in the indicator. The total backlog (of all OLOs) as at 24 November 2015 was 4,702 WO's, equal to 3.1% of the backlog and the backlog of only orders with expired EDD was 3,902 pieces, equal to 2.9% of the total backlog. Of these latter orders, only requests for activation or migration of access services on active lines can generate disruptions to end customers. In this case of 24 November, these orders were 1,529, equal to approx. 1% of the total backlog.
The SB therefore initiated an audit to acquire additional information on the excessive reschedulings of the Expected Delivery Date (EDD). With regard to the case in point, in fact, in the meeting of 2 December 2014, the SB asked the Operator to perform specific interventions and take concrete actions to reduce multiple rescheduling of the EDD and failure to send to the OLO (according to the timing envisaged in the process) notifications relating to the various order statuses. As a result of the intervention of the Supervisory Board and in accepting its requests, in the communication of 12 December 2014, Telecom Italia declared that it had initiated the following activities:
-
EDD rescheduling
A periodic check was activated with the objective of releasing, in addition to any orders with expired EDD, also those in the backlog since EDD-2. By December 2014, two additional interventions should have been released (one designed to reduce process latencies due to NP and the other to optimise acquisition of the WO files) that will help reduce any order downtimes and increase the probability of executing orders within the original EDD.
-
Transmission of notifications
Telecom Italia has implemented a dashboard that verifies the correspondence between planned notifications and those actually produced. If it is found that a notification, although planned, has not been produced, an ad hoc monitoring ensures it is correctly reproduced. The dashboard has been enriched by reports, which is in the process of being optimised, indicating the number of notifications planned, those produced and those incorrect. By the end of next December, the IT procedure will be further strengthened with regard to the aspects relating to the correct association between notifications produced and notifications actually present in the files sent to the OLOs.
Complaint S03/13 – Welcome Italia/Physical deterioration of lines and fruitless service interventions, SLA compliance and size of penalties
Welcome Italia complained of critical aspects associated with frequent deterioration and repeated faults in access and transport, charging of alleged fruitless service interventions by Telecom Italia and the penalties paid by Telecom Italia for failure to comply with the SLA.
During the hearing held c/o the SB in 2013, Welcome Italia complained of problems related to service disruption and deterioration of the access network; general service disruption in the transport network; the provisioning of bitstream services over ethernet networks; fruitless service interventions, the SLAs and the assurance penalties for bitstream services, deemed to be insufficient deterrents and not commensurate with the inconvenience suffered by the Operator.
The SO drafted the customary Technical Report following which the SB opened Complaint n. 3/13 through Resolution no. 9/201316. The SB asked Welcome to provide detailed data and information about the alleged faults.
Welcome sent the requested information; subsequently, the SB deemed it advisable to recommend that Telecom Italia should send further information, so that it may conduct cross analyses between the various groups of elements, a request that was fulfilled by the incumbent Operator.
In 2014, the SB met Telecom Italia during hearings on 28 January and 15 May and Welcome Italia on 12 February and 11 June. Telecom Italia recognised the reported disservices, although they underlined the improvements in the fault rate during the period 2011–2013.
In agreement with the SB, it was decided to set up specific technical discussions to analyse and solve the difficulties encountered. A first meeting was held in June 2014 at the premises of Welcome in Massarosa, and a progress meeting took place on 23 September at the premises of the SB. A subsequent meeting was held in Ancona, at a Telecom Italia work centre, in October, while another progress meeting was held on 21 November. At the proposal of the SB, the two Operators have initiated an analysis aimed at identifying an agreed technical definition of "deterioration" and "disruption" in order to facilitate operations for both Telecom and Welcome.
The final deadline for concluding the investigation, initially set for 29 August 2013, was extended a number of times during the same year, by Resolution n. 17/201317 and Resolution n. 23/201318. In 2014 this deadline was further extended, first with Resolution no. 7/201419, and subsequently with Resolution no. 12/201420 and Resolution no. 20/201421.
Specific Analyses
Analysis of Wholesale and Retail KOs
In 2010, the SB launched the “KO Wholesale Analysis” Project in order to analyse the reasons for rejection of Wholesale Work Orders. It then ordered the start of a similar analysis into the causes for rejection of KOs relative to the offer of Telecom Italia Retail services. no cases of unequal treatment between Retail Work Orders and Wholesale Work Orders were identified. An improvement has been noted in the efficiency of the process, in Telecom Italia's favour, with regard to the number of WOs needed to fulfil a customer's request: this difference should progressively disappear as all the OLOs subscribe to the New Delivery Process.
With Resolution no. 26/201322, the SB decided to continue the analysis on Wholesale customers, to assess full compliance with the principle of internal-external equality of treatment by Telecom Italia in the Delivery process area.
During the Meeting of the SB on 13 March, Resolution no. 9/201423 to extend Order analysis to Retail Work Orders was approved.
The SO was asked to report periodically to the SB on the results of the checks made, proposing any more suitable action to be taken, and to draw up a final report of the action taken and the results attained.
During the Board meetings in October and November 2014, the results of the investigation were presented, highlighting the comparisons between the data for the two-year period 2009-2010 and current retail/wholesale performance.
Some overall improvements in the wholesale delivery chain were verified and areas for further improvement of process efficiency were also identified.
The pre-alert system for exchanges at risk of saturation (“Amber Light”)
Complaint S01/10 was opened following the sales closure of the bitstream service in approximately 500 exchanges due to network saturation.
Based on the results of analyses conducted when the Complaint was made, the SB advised Telecom Italia to establish a mechanism to signal exchanges close to saturation.
As a result of the "End of sale" of ATM technology, the SB continued monitoring the state of saturation of exchanges as regards bitstream services, paying particular attention to exchanges equipped with saturated ATM DSLAM that, since they have no availability of DSLAM Ethernet, do not allow the sale of new connections for ADSL services. As at 31 December 2014, the number of these connections was 57, with an increase of 4 units as compared to 30 September 2014. In contrast, the decrease in the number of exchanges equipped with saturated ATM miniDSLAMs and without the alternative of IP Ethernet technology continued. In fact, the number of exchanges dropped from 677 recorded on 30 September 2014 to 600 recorded at the end of December. As regards the saturation of the exchanges served by Ethernet DSLAM, at the end of 4Q2014 there were 51 saturated exchanges (+25 more than 3Q2014) and 94 saturated cabinet backpacks (+1 unit compared to 3Q2014), while on the same date 11 cabinet DSLAM and 29 exchange DSLAM were in “amber light" status.
With Resolution 94/12/CIR, AGCom - acknowledging the need to promote rapid transition to Ethernet/IP technology, instructed Telecom Italia to pass on a series of financial incentives to the OLOs for the duration of the migration period and imposed a reduction in prices for bitstream services provided on the Ethernet platform. The Authority also asked Telecom Italia, as a condition for the recognition of ATM "End of Sale", to make available certain functional elements relating to: the process of service acquisition and provisioning, the adaptation of protocols to render compatible the OLO modems and the link analysis tools on the section relating to the delivery kits (for the OLOs).
By December 2013, Telecom Italia had made available all the functional elements required by the resolution. On 28/02/2014, the company announced on the Wholesale Portal that it would proceed to implement provisioning of new ADSL bitstream access exclusively with Ethernet technology for Bitstream exchanges already covered (or will be) by that technology. In the same communication, Telecom Italia also announced that Bitstream exchanges with DSLAM ATM where there is an equivalent alternative to the Bitstream Ethernet service, would have been progressively placed in "saturated" in accordance with a pre-defined plan.
Analysis of the correct application of the Single Queue
Undertakings Group n. 1 envisaged the creation by Telecom Italia of a so-called Single Queue mechanism for handling Work Orders in connection with SPM services; with Resolution no. 12/201224, the SB launched activity to check the correct management of this system by Open Access.
During 2012, the Supervisory Office carried out on-site checks in a number of geographic areas, making spot checks at local Open Access operating sites to analyse the WOs that had been entered into the Single Queue and subsequently executed. No anomalies were found in the analysis. Accordingly, the SB closed the enquiry in question with Resolution no. 17/201225.
Once again in 2013, the SB decided to continue such monitoring (see Resolution n. 3/201326. During the year, checks were carried out c/o various AOL belonging to all Regional Areas in Italy. Resolution n. 3/201427 closing the activity also indicates that checks in 2013 did not reveal any cases of unequal treatment between Telecom Italia Retail customers and customers of OLOs regarding the application of the Single Queue mechanism.
Monitoring will also continue in 2014 (Resolution n. 4/201428); in particular, during the third quarter were conducted visits were made to the geographic areas of Sardinia, Campania, Naples, Basilicata, Puglia, Trentino Alto Adige and Friuli Venezia Giulia.
The checks undertaken so far have not revealed any cases of unequal treatment between Telecom Italia Retail customers and customers of OLOs regarding the application of the Single Queue.