The Goal of Controller Training

If your company meets the CRM regulations, Operator Qualification requirements, and any other regulatory requirements, will you be satisfied? Will you be confident that controllers have the knowledge, skills, and abilities to perform excellently? What are the goals of your controller training program? It could be a useful exercise to define those goals in terms of going beyond compliance and achieving an excellent program that emphasizes both excellent performance and excellence of process. The diagram below illustrates an approach to training by Craig Watson, our Training Consultant. Does your training program include those elements? For more information, read the article Craig wrote in the Pipeline and Gas Journal. http://www.pipelineandgasjournal.com/business-control-room-training

The Control Room Management (CRM) regulation states that companies have to train each controller to carry out the roles and responsibilities defined by the operator. Have you compared the list of responsibilities that controllers must fulfill to the elements of the controller training program to ensure that it includes training on each responsibility and all of the responsibilities?

The regulation emphasizes training on responses to abnormal operating conditions, using a computerized simulator or tabletop scenarios. Have you developed scenarios for each of the abnormal operating conditions in your operating procedures? Do some of those scenarios combine the occurrence of several abnormal operating conditions that might occur in sequence or at the same time? If your company has invested in a computerized simulator, is it meeting the intent of simulating abnormal events in ways that enable controllers to demonstrate competencies in their responses? Does your company have a process for evaluation of the controller responses on the simulator?

Do you have adequate resources and technical support to keep the simulator functioning properly?

Since controllers have responsibilities for communications during emergency response, it is required that controllers be trained on those responsibilities. What types of communications training do you provide? It makes sense to develop tabletop scenarios based on the emergency operating procedures and/or emergency response plans. It is my observation that some emergency response plans are quite explicit on the incident command system; yet lack explicit instructions for controllers on their responsibilities for communications. PHMSA has issued at least one Advisory Bulletin that includes communications responsibilities for control rooms. Does your CRM Plan address the guidance in applicable Advisory Bulletins? (PHMSA–2012–0201 Pipeline Safety: Communication During Emergency Situations)

How are you providing controllers a working knowledge of the pipeline system? I think this requires more than a hydraulics course, most of which are heavy on theory and light on practical applications. We recommend developing a detailed outline or checklist that includes the specifics of each pipeline segment, each facility, types and sizes of equipment the controller must use, and normal operating procedures that provide guidance to controllers. Controllers must be able to explain the details of the system and demonstrate how to control and monitor the system in order to complete their training. They need working knowledge, not theoretical principles. Look at the hydraulic demonstrator we used in training with Chinese pipeline controllers. See the sags, the over bends, the hydraulic gradient. We demonstrated slack line conditions, pressure increases and decreases, and other principles.

Have you developed a list of pipeline operating setups that are periodically, but infrequently used? Do you have procedures for those setups so that controllers can review the procedures prior to performing the operation? This should not be very complicated; it just needs to be done. The list needs to be reviewed and updated as part of the annual evaluation of the controller training program.

What is the process for the annual evaluation of the training program content and for the identification of potential improvements? One of the hindrances to the evaluation of the training program content is that the content either does not exist or it is in the head of some expert. If it does exist, it is scattered throughout the control room or
in multiple learning management systems or other sites on company networks or
hard drives.

As a starting point for compliance, look at these findings from CRM inspections from 2012-2015 and ensure that your CRM plan and implementation do not have these deficiencies about “Controller Training:”

(Notice of Proposed Violation (NOPV) 07/23/15)

Non Identified Company (NIC) failed to establish a controller-training program in accordance with Section 195 .446(h). Section 195.446(a) required the controller training program to be implemented by August 1,2012. NIC’s controller training program did not clearly define controller roles and responsibilities and did not include training for responding to abnormal operating conditions likely to occur simultaneously or in sequence. The training program did not identify setups that are periodically, but infrequently used, and did not indicate how the controllers were trained on the procedures used for such setups.

(Notice Of Amendment (NOA) 04/20/15) This company had multiple deficiencies! It must have been the same inspector as the NOA 10/10/12 below since it has some of the same language in the deficiencies.

NIC’s procedures for training as presented had a number of general deficiencies in the training procedures reviewed as identified below:

• NIC’s training section lacked adequate detail and requires more content, substance, and formality to meet the requirements/intent of the code.
• New system asset training has not been addressed in NIC’s training section of the CRM plan. Classroom training, hydraulic review, simulation, and beginning low flow conditions have not been included in the manual.
• NIC has not reviewed controller and cross-training assessment documentation.
• NIC has not included interviews with supervisors as well as a trainee’s activities in their training plan. Also, NIC has not constructed formality and implementation references associated with identified improvements in training.
• NIC has not included procedures to crosscheck to confirm O&M, Emergency, CRM, and other specific plan sections are addressed in the Roles and Responsibility section and that all roles and responsibilities are covered in the CRM training program.
• NIC has not included the training module completed list for controllers and supervisors in their training plan.
• NIC had not emphasized that controllers should ‘”Think leak first” in the
  training material.
• NIC has not addressed explicitly explaining differences between consoles in the training section.
• NIC’s procedures for training did not include responding to abnormal operating conditions likely to occur simultaneously or in sequence.

NIC’s procedures for training as presented had a number or areas where PHMSA noted general deficiencies in the training procedures reviewed with respect to the use of a computerized simulator or non-computerized (tabletop) method for training controllers to recognize abnormal operating conditions. Each issue is identified below:

• NIC did not adequately address information about control room participation in field driven tabletop exercises or spill drills as verbally described by NIC personnel during the inspection.
• NIC plan does not include consideration for working with executive management to allow controller input to the selection of field tabletop scenarios.
• NIC has not included simulating AOCs not addressed elsewhere, (communication outages for example) in their training procedures.

NIC’s procedures for training as presented had a number of areas where PHMSA noted deficiencies in the procedures reviewed with respect to providing a controller a working knowledge of the pipeline system, especially during the development of abnormal operating conditions. NIC must amend their procedures to address each issue as identified below:

• NIC did not include HAZOPS, trouble logs, and other monthly reviews of lessons learned in their training program.
• NIC did not include covering older selected lesson learned (war stories) with new controllers in their training section.
• NIC did not incorporate operating experience components into their
  training program.

NIC’s procedures for training as presented had a number or deficiencies in the procedures reviewed with respect to pipeline operating setups that are periodically, but infrequently used, providing an opportunity for controllers to review relevant procedures in advance of their application. Each issue is identified below:

• NIC did not include seldom-used procedures/set-up in the training section.
• NIC did not include equipment position regarding power failure in their
  training program.

(NOA 03/24/15)

• NIC did not have adequate written procedures for the controller-training program.
• NIC did not have an adequate procedure that identified who is responsible for reviewing the training program and for making improvements.
• NIC did not have adequate procedures that addressed training on abnormal situations that may occur sequentially or simultaneously.
• NIC did not have adequate procedures for training on infrequent operations.

(Warning Letter 02/09/15)

NIC failed to establish the content of controllers training program that includes the abnormal operating conditions (AOC) scenarios likely to occur simultaneously or in sequence. NIC Controller Training Plan, Section 2 (program) states:

• NIC’s Controller Training Plan is designed to ensure that all NIC Controllers working on NIC’s DOT-regulated pipeline facilities are CRM and OQ-qualified to perform their roles, responsibilities and Operator qualified covered tasks, to document that qualification and to reduce the probability and consequences of incidents and accidents. All NIC employees performing these roles, responsibilities and covered tasks will be qualified under this program.

During the inspection, PHMSA requested NIC to provide the Controllers training elements to ensure whether all the controllers are trained on multiple abnormal operating conditions (AOCs) that could likely occur simultaneously or in sequence; NIC provided two (2) lists of AOC scenarios dated 8/14/2014 and 10/10/2014. PHMSA reviewed all training scenarios and found that each scenario constitutes a single event (AOC) scenario instead of having multiple events (AOC) that could likely to occur simultaneously or in sequence as required by §195.446(h)(I).

(NOA 03/27/15)

NIC did not establish adequate written procedures for responding to abnormal operating (AO) conditions as required by § 192.631 (h) (1). An operator is required to establish and implement a controller training program for each controller to carry out their roles and responsibilities. At the time of the inspection, NICs’ CRM procedures were inadequate because NIC’s AO procedures did not consider simultaneous or sequential scenarios.

(NOA 12/10/14)

• NIC’s CRM plans and procedures did not adequately define requirements for recognizing and responding to abnormal operating conditions that are likely to occur simultaneously or in sequence. NIC must also include training on lessons learned from the review of operating experience, in accordance with 195.446(g)(2), including critiques of all recent accidents/incidents.
• NIC’s CRM plans and procedures did not adequately define requirements to train controllers on their responsibilities for communication under the operator’s emergency response procedures. The training program should require that controllers demonstrate knowledge and proficiency in communicating during an emergency.
• NIC’s CRM plans and procedures did not adequately define requirements for training to provide controllers a working knowledge of the pipeline system, especially during the development of abnormal operating conditions. Training should ensure that controllers have practical knowledge of how fluid dynamics, electrical power, communications, etc. impact operations. Training should also include information about how pressure and flow in all pipeline segments are impacted by control actions.

 (NOA 12/10/14)

• NIC did not review the training program content at least once each calendar year, but at intervals not exceed 15 months, to identify potential improvements. That is, NIC did not present to the OPS inspector records to demonstrate that it had reviewed the training program content in calendar year 2013 to identify potential improvements.

 (NOA 09/09/14)

NIC’s control room training review process references document F-195.446 (h) which no longer exists to record controller training content as a means to identify potential improvements to the training program. PHMSA reviewed the procedure and found that the document used for recording the control room training review no longer exists, and should be removed as referenced in the procedure.

(NOA 07/28/14)

NIC Control Room Supervisor training had NOT been completed prior to Control Room operation. The Control Room Supervisor failed to complete required training prior to Control Room operation on March 1, 2014. The Control Room Supervisor completed “Training for Managing Operator Fatigue” on June 4, 2014.

(NOA 09/30/13)

The NIC Control Room Management Plan failed to have a detailed written process on how training scenarios for recognizing and responding to abnormal operating conditions that are likely to occur simultaneously or in sequence are determined, in accordance with §195.446(h)(1). Specifically, the CRM Training procedure did not address reviewing historical alarm logs to identify candidate scenarios for training.

(NOA 07/18/13)

• NIC’s written control room management procedures were inadequate because the procedure, Section 10.7 Annual Training Requirements, specifies Control Room Training “will be reviewed once per calendar year” as opposed to once per calendar year, not to exceed 15 months as prescribed in §192.631(h).
• NIC’s control room management procedures were inadequate because Section 10.3 Specific CRM Training Requirements does not include a subset list of AOCs required to make training more specific and meaningful as prescribed in §192.63(h).

(NOA 07/12/13)

• NIC’s written control room management procedures were inadequate because the procedure depicts a bullet list that addresses infrequent setups, but contains no actual listing or description of such setups as prescribed in §192.631(h).

 (NOA 11/28/12)

• The CRM Plan needs to include procedures to review the training program content and identify potential improvements.
• The CRM Plan needs to include a list of all the tasks performed by the controllers and to include how the training will demonstrate the proficiency on each task.
• The CRM Plan needs to include a list of some of the most common abnormal operating conditions (AOC’s) that may occur simultaneously, and include training to recognize and react correctly to those situations when simultaneous AOC’s occur.
• The CRM Plan needs to include a procedure for having controllers perform their responsibilities during emergency response drills and a method to document the controllers’ performance.
• The CRM Plan needs to include training that will provide controllers a working knowledge of the pipeline system.

(NOA 10/10/12)

NIC’s procedures for training as presented had a number of areas where PHMSA noted general deficiencies in the training procedures reviewed. Each issue as identified below:

• NIC’s training section lacked adequate detail and requires more content, substance, and formality to meet the requirements/intent of the code.
• NIC’s use of the terms, “Experienced controllers” and “Inexperienced controllers” is not consistently implemented throughout the training section.
• New system asset training has not been addressed in NIC’s training section of the CRM plan. Classroom training, hydraulic review, simulation, and beginning low flow conditions have not been included in the manual.
• NIC has not reviewed controller and cross-training assessment documentation.
• NIC has not defined what “promptly” means regarding identified improvement implementation.
• NIC has not documented all elements of training in their training plan.
• NIC has not included interviews with supervisors as well as a trainee’s activities in their training plan. Also, NIC has not constructed formality and implementation references associated with identified improvements in training.
• NIC has not included procedures to crosscheck to confirm O&M, Emergency, CRM, and other specific plan sections are addressed in the Roles and Responsibility section and that all roles and responsibilities are covered in the CRM training program.
• NIC has not included the training module completed list for controllers and supervisors in their training plan.
• NIC has not included the power point presentations that are used for controller training in their plan.
• NIC had not emphasized that controllers should “Think leak first” in the
  training material.
• NIC has not addressed explicitly explaining differences between consoles in the training section (Console 7 versus console 3).

NIC’s procedures for training did not include responding to abnormal operating conditions likely to occur simultaneously or in sequence.

NIC’s procedures for training as presented had a number of areas where PHMSA noted general deficiencies in the training procedures reviewed with respect to the use of a computerized simulator or non-computerized (tabletop) method for training controllers to recognize abnormal operating conditions. Each issue is identified below:

• NIC did not adequately address information about Control Center participation in field driven tabletop exercises or spill drills as verbally described by NIC personnel during the inspection.
• NIC plan does not include consideration for working with executive management to allow controller input to the selection of field tabletop scenarios.
• NIC has not included simulating AOCs not addressed elsewhere, (communication outages for example) in their training procedures.

NIC’s procedures for training as presented had a number of areas where PHMSA noted deficiencies in the procedures reviewed with respect to providing a controller a working knowledge of the pipeline system, especially during the development of abnormal operating conditions. NIC must amend their procedures to address each issue as identified below:

• NIC did not include HAZOPS, trouble logs, and other monthly reviews of lessons learned in their training program.
• NIC did not include covering older selected lesson learned (war stories) with new controllers in their training section.
• NIC did not incorporate operating experience components into their training program, specifically those identified through section G review as discussed verbally during the inspection.

NIC’s procedures for training as presented had a number of areas where PHMSA noted deficiencies in the procedures reviewed with respect to pipeline operating setups that are periodically, but infrequently used, providing an opportunity for controllers to review relevant procedures in advance of their application. Each issue as identified below:

• NIC did not include seldom-used procedures/set-ups in the training section (example: the leased two lines from DOE regarding flowing at a specific rate once a year and reverse flow considerations.)
• NIC did not include equipment position regarding power failure (well heads fail closed, some tank valves failed closed other equipment does not) in their
  training program.

(NOA 09/13/12)

NIC’s training program is inadequate. The training program does not address the backup SCADA system and/or the backup control room. NIC has a backup SCADA system and this is not covered in the training program.

 (NOA 07/06/12)

• NIC’s training program does provide a review of actual data for the occurrence of simultaneous or in-sequence abnormal operating conditions.
• NIC should consider and amend its training program to include provisions specifically for Day Controllers. Day Controllers may not operate a console for an extended period of time. Changes associated with roles and responsibilities, procedures or console equipment may have occurred since the last time a Day Controller operated the console. NIC did not include provisions for necessary training as the result of changes that have occurred prior to Day Controllers operating the console.
• NIC’s controllers have not been consulted regarding the use of color and/or descriptors associated with Alarms. The only evidence that the controllers may have a problem with the color or associated understanding of alarm descriptors is through the use of the simulator or specific alarm review activities. NIC should incorporate into the simulation aspect of the training program a specific review of all colors and significant alarms utilized on the console so that acuity and alarm descriptor understanding can be confirmed.
• While NIC does use and develop computerized simulations for training, these simulations did not appear to cover all pipeline assets associated with any one complete console, a terminal, specific pressure reductions prior to implementation, or difficult pipeline operations. The NIC training plan should describe how simulation scenarios are developed and selected based on various operating considerations.

NIC’s training program does not provide specifics regarding terminal operations and associated configurations that are infrequently used.

(NOA 04/24/12)

NIC’s Pipeline Controller Training Program did not provide adequate specificity for processes that will be used to review the NIC controller-training program as required by §195.446(h). Section 4 Measurement and Verification explains that one of the metrics that will be used to confirm the effectiveness of their controller training program will be improved controller competency. The measurement of improved controller competency will be by the use of competency assessments. Interviews with NIC personnel indicated that each controller will be assessed by a competency board. However, the training program provides no guidance as to who will perform a competency assessment or how that competency assessment will be made. NIC’s procedures must be amended to fully describe the controller competency assessment process including the makeup of the competency assessment board as well as a detailed description of a competency assessment.

I suppose the initial goal for the controller training program should be to avoid receiving a notice of amendment, a notice of probable violation with potential fines, or other enforcement action from PHMSA. If one takes the time to read the deficiencies above, it is evident that most of those deficiencies should have been avoided by writing a CRM training program procedure that addressed the regulations, FAQs, inspection questions, and advisory bulletins. Then, the company needed to implement the written procedure, including documenting the training of controllers and supervisors, and the annual review of the training program content