My favorite SCADA person works in a control room on the west side of Houston. When we did a workload assessment there in 2012, I found it interesting that he had a console adjacent to the controllers’ console. He and the Controller could look at issues as they occurred and I observed that my favorite SCADA person was interested in continuous improvement of displays, how the field devices were working, the accuracy of information, and the reliability of communications. In this control room, the SCADA person was also a qualified and trained Controller. He worked on shift once or twice each quarter to maintain qualifications and to determine from his perspective if the information being provided to Controllers was adequate. Since the
definition of “adequate” only implies good enough, his goal for his work was to surpass adequacy.
That scenario may not work for your control room and your SCADA personnel. Even if your SCADA personnel work in a separate room or separate city, it is important that SCADA personnel adopt a mindset of continuous improvement and customer satisfaction, with the Controller being the customer. Whether your SCADA Squad has one person or dozens of people, pipeline safety can be compromised if the Controller
does not have displays that are accurate, the points in the field and on the displays do not have the same information, there is no procedure for loss of data communications or SCADA systems, and/or a backup SCADA system is not tested for reliability.
That is the intent of the SCADA related portion of the Control Room Management (CRM) Regulations:
* Provide adequate information. Each operator must provide its controllers with the information, tools, processes and procedures necessary for the controllers to carry out the roles and responsibilities the operator has defined by performing each of the following:
As it turns out, it takes a heroic effort by many people just to provide adequate information. We have met SCADA Squads all over North America. While they do not dress like superheroes, they must be alert, conscientious, and trained so that they perform their tasks correctly time and time again. Technology advances have made it easier to implement SCADA systems and technology advances can make it easier
to make mistakes.
A few weeks ago in Texas, I met a Screen Guy and also observed a point-to-point verification when a Field Tech at a facility and a Database Guy in his darkened cubicle communicated well about all the points that were being commissioned. They followed their company’s procedure and documented the necessary, required information. This company also has SCADA personnel, male and female, who had specific responsibilities. It appears this SCADA Squad is providing adequate information to its Controllers.
PHMSA publishes enforcement actions on a monthly basis. It is not easy to find the CRM related reports, but there is useful information from CRM inspections that resulted in Notices of Amendments, Notices of Probable Violations, Warning Letters, or Corrective Action Orders. One has to look at all the reports and find the ones of interest. I have found 38 publicly available reports that we use to ensure our clients’ CRM plans are addressing what some companies had not done prior to an inspection.
Of the four SCADA-related areas in the CRM regulations, it appears most companies we work with have standard processes for display development that are compliant with API Recommended Practice 1165 or more up to date human machine interface standards such as those in The High Performance HMI Handbook. Bill Hollifield, one of the authors, recently told me a second edition would be published soon. The enforcement actions related to SCADA displays were when the CRM plan did not describe how the company was going to comply with API Recommended Practice 1165.
One of the SCADA-related items that is still being discussed, debated, dissented is the definition of a safety-related point for point-to-point verification. As companies “count the costs” of compliance, the more safety-related points there are on a system, the more time it requires verifying points. Our advice is to err on the side of pipeline safety. These examples from enforcement actions may be useful for your SCADA Squad.
One company received a Notice of Probable Violation because it had “failed to define safety-related points in it Control Room Management procedures…Company could not demonstrate that it had a written process or criteria for determining which points are safety-related.”
Another company received a Notice of Amendment because “its plan did not adequately define safety related points nor sufficiently identify when field equipment additions, movements, or changes could impact pipeline safety.”
A third company received a Notice of Amendment because its “procedures were inadequate because the definition of Safety Related Points contained in GCOP-400 6.32 states that points indicate harm and failures, rather than what is needed to insure the pipeline remains safe as prescribed in §192.631(c)(2).”
The next SCADA-related item is the testing of an internal communication plan. In my simplistic view a company has to have a procedure for loss of communications. It may have a procedure for partial loss of communications and another for total loss of communications. The company needs to decide if it will operate its pipelines during either a partial or a total loss of communications. If it decides to operate at those times, the internal communication plan must have detailed method for ensuring pipeline safety control and monitoring during the SCADA or data communications outage.
The internal communications plan must be tested each calendar year, not to exceed 15 months from the previous test. Some companies appear to struggle with whether or not they will continue to operate, particularly if the company has reduced complement of field personnel so that it is difficult to staff facilities with people who can communicate status reports to the Controllers. Other companies wonder if it is necessary to test the internal communication plan on each of its pipeline systems and/or with all Controllers each calendar year, not to exceed 15 months. That might be ideal, but the purpose is to test the effectiveness of the plan. The SCADA Squad would have to do even more heroic feats.
It appears most companies have processes and procedures for testing of backup SCADA systems, but some have difficulties documenting all of the information in the inspection questions. Some companies have more than one backup SCADA system; some also have backup control rooms. Each and all have to be tested with documented results.
I do not know much about the technical aspects of how SCADA works, the necessary hardware and software, and how it all fits together in points, bits and bytes. I am just an operations guy, thankful that SCADA engineers, technicians, analysts, and field technicians know what to do to design, implement, maintain, and address any problems that occur. Whether your SCADA Squad looks like superheroes or just dress in ordinary clothes, express appreciation to them like I did to my favorite SCADA person and to all the SCADA people at our clients. They love to see us arrive on site!
Charles Alday © 2015 Please Distribute to Others.
