Have you ever showed up for work and there were new features on the SCADA display, new procedures, or new policies that you are supposed to use during the shift even though there had been no opportunity to review those changes? Can you remember times when something unexpected occurs on the pipeline because a technician is performing maintenance, but no one had provided notification that the maintenance was going to be performed? Does your company have a Management of Change (MOC) procedure, but is it sometimes circumvented due to a “rushed” business development project?
Although it should not be necessary for a regulatory agency to require pipeline companies to manage changes that affect pipeline safety and to require notifications prior to maintenance
and during emergency conditions, one section of the Control Room Management (CRM) regulations requires:
Why did PHMSA think it was necessary to include that CRM requirement? Can you think of any pipeline accidents where poorly managed changes or where maintenance being performed on active systems contributed to an accident? Look at the NTSB Pipeline Accident Reports for the Olympic Pipeline 1999 accident or the Piney Point Pipeline 2000 accident. Look at the National Energy Board of Canada 2012 tank overflow report on the Trans Mountain Pipeline. Have there been any operating errors or near misses in your company caused by change management failures? I could cite examples, and I suspect each of us could do the same. There are good reasons why OSHA includes Management of Change requirements in its Process Safety Management Standard to address: the technical basis for the proposed change, impact of the change on employee safety and health, modifications to operating procedures, necessary time period for the change, and authorization requirements for the proposed change.
Consider this advice from the Chemical Safety Board (CSB) and apply it to your safety processes:
- Define safe limits for process conditions, variables and activities — and train personnel to recognize significant changes. Combined with knowledge of established operating procedures, this additional training will enable personnel to activate the MOC system when appropriate.
- Apply multidisciplinary and specialized expertise when analyzing deviations.
- Use appropriate hazard analysis techniques.
- Authorize changes at a level commensurate with risks and hazards.
- Communicate the essential elements of new operating procedures in writing.
- Communicate potential hazards and safe operating limits in writing.
- Provide training in new procedures commensurate with their complexity.
- Conduct periodic audits to determine if the program is effective.
The CSB indicates in reports that inadequately managed changes have been cited as contributing factors in major industrial accidents. I have facilitated many pipeline incident and accident investigations, and inadequate change management was often a basic cause. Do you remember these statements from the proposed CRM rule (9-12-08): Changes to the pipeline system are important and can affect the ability of a controller to do his job. System changes can affect the hydraulics of the pipeline and change the response to control inputs. It is important that controllers be aware of changes being made and that controllers are involved early in the change process to help identify and alleviate any undesirable effects on controllers and control room operations. Similarly, changes to the SCADA system, or to the instruments it monitors, can also affect a controller’s understanding of conditions on the pipeline and his recognition of the need for control actions. The proposed rule requires operators to establish thorough and frequent communications between controllers, management, and field personnel when planning and implementing changes to pipeline equipment and configuration. Maintenance procedures must ensure that problems with SCADA or field instrumentation critical to controllers are resolved promptly and properly documented. SCADA system modifications must also be coordinated with controllers and affected pipeline operating personnel. It is not always practical to coordinate changes before they are made, particularly when a change is in response to an emergency. In those instances, operators must make affected personnel and controllers aware of the change as soon as practical and document why this occurred. When field equipment, pipeline configuration, or SCADA changes are planned in advance, coordination should also be done so that controllers who are off duty get informed of these changes prior to implementation. Controllers shall have time to study the implications of targeted changes and to become familiar with the anticipated system changes before they are initiated. Finally, controllers shall be represented by a controller, controller supervisor or by someone very familiar with control room operations when changes that can affect pipeline hydraulics, configuration or control system changes are considered so that controller perspectives and potential impacts can be considered early in the planning process and appropriate adjustments and training can be developed. Whenever possible, operators should thoroughly test changes on an off-line system.
Management of change procedures shall also include how operators will inform controllers of changes before they operate the system, especially the controllers who are not on shift at the time the changes are made. Most pipeline companies have a formal MOC procedure or process. Some of these processes are complex and require use of software for identifying action items, assigning action items and due dates, tracking completed action items, and making notifications to personnel who are affected by the change. Because both the process and the software can be cumbersome, there may be a reluctance to use the MOC procedure. There may be a tendency to avoid classifying a change as a MOC type change. One person I questioned said, “It was just going to be a minor display change and it was a Friday.” Guess what? A controller made an error using that display and contaminated a tank of fuel oil. Other companies do not have a formal MOC procedure and those companies do an even worse job of managing changes because they can justify why nothing is a change that affects the control room. Here are some examples from Notices of Amendment and a Warning Letter that indicate that it may be necessary to do some improvements in both CRM written procedures about change management and in implementation of those written procedures. (NOA 07-23-15) XXX did not establish adequate written procedures for implementing section 7 of API RP 1168 as required by §195.446(f)(1). During the inspection, it was determined that the procedures in the XXX CRM manual do not require implementation of section 7 of API RP 1168 for control room management change and require coordination between control room representatives, operator’s management, and associated field personnel when planning and implementing physical changes to pipeline equipment or configuration. In addition, the CRM plan does not have a process/procedure to assure changes in field equipment (for example, moving a valve) that could affect control room operations are coordinated with the control room personnel. (NOA 07-23-15) XXX did not establish adequate written procedures for requiring the field personnel to contact the control room when emergency conditions exit as required by § 195.446(f)(2). During the inspection, it was determined that the procedures in the XXX CRM manual do not require field personnel to contact the control room when emergency conditions exist and when making field changes that affect control room operations (NOA 04-20-15) XXX’s Change Management Plan as presented had deficiencies in the procedures reviewed with respect to implementing section 7 of API RP 1168 for control room management change and requiring coordination between control room representatives, operator’s management, and associated field personnel when planning and implementing physical changes to pipeline equipment or configuration as identified below:
- XXX’s Management of Change (MOC) documentation is not formalized and does not describe the actual changes made and what was changed.
- XXX’s calibrating procedures for reviewing alarm set points did not include all applicable details.
- XXX did not include asset changes in the CRM manual regarding the MOC process.
(NOA 03-27-15) XXX did not establish adequate written procedures to address the communications between control room representatives, operator’s management, and associated field personnel as required by §192.63l(f)(l). At the time of the inspection, XXX’s CRM procedures were inadequate because they did not clearly describe the Control Room portion of XXX’s Management of Change (MOC) procedures. Leads and supervisors viewed all MOC’s and decide whether the CR is affected. This policy needs to be amended so that it is written into the overall MOC procedure.
(Warning Letter 12-24-13) XXX did not follow Management of Change (MOC) procedures when changes were made to the Pipeline Control Procedure Manual procedures. Additionally, changes were made to the critical alarm type description and response priority. XXX did not follow their Change Management Procedure, Section 7 of the XXX Control Room Management Plan. Section 7.2, Communication During Changes to Pipeline Equipment states that “pipeline systems/processes which are generally covered by the MOC process include but are not limited to: purchase or sale of physical assets, new equipment coming online, retired equipment going offline, operations and maintenance manuals, new or revised procedures, operating responsibilities between pipeline controllers and field personnel or third-party operations, field maintenance activity affecting pipeline control room operations, control system changes, SCADA system changes.” At the time of the inspection, PHMSA representatives discovered that several procedures from different manuals were consolidated into a Pipeline Control Procedure Manual. The lack of documentation and interviews with control room staff demonstrates that XXX did not follow their MOC procedure for modifying their Pipeline Controller Procedure Manual.
(NOA 09-04-13) XXX’s CRM procedures did not adequately require coordination between control room representatives, operator’s management, and associated field personnel when planning and implementing physical changes to pipeline equipment or configuration. O&M Manual Section 8.8 did not require that configuration changes to SCADA screens relating to changes in field equipment must be made prior to the equipment changes going into operation.
(NOA 11-28-12) The CRM plan needs to be amended to incorporate current MOC procedures and define when a MOC document needs to be completed for CRM activities. The plan also needs to address adequate training for controllers prior to implementing the change.
(NOA 10-10-12) XXX’s Change Management Plan as presented had a number of areas where PHMSA noted deficiencies in the procedures reviewed with respect to implementing section 7 of API RP 1168 for control room management change and requiring coordination between control room representatives, operator’s management, and associated field personnel when planning and implementing physical changes to pipeline equipment or configuration.
(NOA 07-06-12) The change management portion of the CRM plan did not sufficiently include the provision in Section 7.3 of API RP 1168 regarding training occurring in advance of implementation of MOCs. XXX did not define emergency MOCs as per API RP 1168 Section 7.4. The definition is needed for clarity if XXX intends to implement emergency MOCs.
(NOA 4-30-12) The XXX Control Room Management Plan, section 700 Change Management, subsection 702 Change Management and Control Room Participation (§192.631(f)(3)) fails to ensure that a control room representative will participate in meetings, where changes that could directly or indirectly affect the hydraulic performance of the pipeline, are being considered, designed and implemented by engineering planning and the MOC process. XXX must amend the procedure to ensure that Gas Control will participate in meetings, where changes that could directly or indirectly affect the hydraulic performance of the pipeline, are being considered, designed and implemented by engineering planning and the MOC process.
The other aspect of change management concerns notifications from field personnel to the controller prior to maintenance, prior to any field changes, and/or when an abnormal or emergency condition occurs. That has always been a requirement at most companies, but one that has not always been followed. Even now, when it is a regulatory requirement, I receive smiles from both field personnel and controllers when I inquire about whether or not field personnel always notify the control room as required by work permit systems, O&M manual instructions, and CRM procedures. What is about human beings that make us reluctant to take the time to talk to another person before performing a task that may affect the other person?
Our reluctance may stem from laziness, time pressure, complacency, poor judgment, introversion, distractions, or an independent spirit. I often say jokingly that “real men don’t document,” and I could also add that “real pipeliners don’t like to talk to other pipeliners” about upcoming tasks. Whether we like it or not, it is important to notify controllers in control rooms before we perform maintenance or make any changes. It is important for controllers to document those notifications so that everyone, including the next shift, is aware that maintenance is occurring or has occurred. It is important for companies to coordinate changes that affect controllers with those controllers. Use your MOC process, communicate all changes, and audit the effectiveness of your processes. Require SCADA and field personnel to notify the controllers before performing any work tasks that might affect pipeline operations. It’s a requirement, but it makes good sense to coordinate and to communicate.
