Redundant power supply failure in the control system card caused the process unit to stop operating
Thread Content
This post was last edited by 2010Lingyun on 2022-8-31 at 15:18. I. Event timeline: At 17:10 on August 4, 2022, a call was made to the process control department; instruments in some processes of VCM Workshop 1 were showing no data due to a disconnected connection. The instrument technician on duty arrived at the control room at 17:12 to conduct an inspection and notified the instrument engineer to go to the engineer’s station. Upon inspection, it was found that the card holder at the back of the VCM-SYS02-F cabinet had lost power. Further checks revealed that the power switches F14 and F52 for the 24V power modules in the power supply cabinet were tripped. Attempting to restore power did not result in any display, and the 24V power modules were overheating, indicating that they were damaged. Immediately remove one of the redundant power supplies on the front of the VCM-SYS02-F cabinet and install it at the faulty location to restore power supply. Subsequently, an instrumentation technician should obtain spare parts from the warehouse for further repairs. II. Overview of the control system: The control system in VCM Workshop 1 is PKS. When this system was upgraded from C200 to C300 in October 2021, only the redundant power supply for the controllers was replaced; the redundant power supply for the cards remained unchanged (it has been in use for 14 years). III. Main causes of the accident: 1. Aging and overheating of the electronic components in the power supply module, as well as the cessation of fan operation, led to inadequate heat dissipation and damage, which caused the circuit breaker to trip ； 2. Inspections were inadequate and not thorough enough; as a result, the malfunction of the redundant power modules was not detected in time. IV. Corrective measures for equipment accidents 1. Develop a proper equipment maintenance plan; in accordance with the operating cycle, perform maintenance tasks such as dust removal and cleaning on the power supplies and fans ； 2. Strengthen comprehensive and thorough inspections of equipment, and address any issues found promptly ； 3. The control system has added an alarm display for power module power loss, promptly alerting inspection personnel to take corrective actions. Summary: Downtimes of the control system have significant impacts, making inspection tasks particularly important. Careful and thorough inspections are necessary to prevent system outages. In the absence of sufficient automation professionals, solving these problems requires the use of new technologies for intelligent upgrades! Through on-site system upgrades by the manufacturer’s engineers, professional system inspections, and on-site technical training, the system has been operating well to this day.
This post was last edited by jlshnlhj on 2022-8-31 at 20:05. A failure in the redundant power supply caused the process unit to stop operating; it seems that both power supplies failed. It’s unlikely that both paths fail simultaneously. One of them might have failed first but went undetected; however, there should have been a system alarm in such a case. If no alarm was triggered, this poses a potential hazard ； Still ignored? Of course, it can also be detected through proper inspections. Also, the fan has stopped working. Is the fan 12V, 24V, or 220V AC? Is it a fan failure? Or did one of the paths experience a power outage due to a redundant power supply failure? The fan power supply must not be used as the control power supply; otherwise, a short circuit in the fan could cause the control power supply to become overloaded and trip. Also, did the fan stopping from operating not trigger any alarm or be detected during inspections? The forum has been redesigned; after replying, there is no response when submitting, so I’m returning to the original post to try editing and adding something. The unit now operates for one shift per day, with inspections carried out in the afternoon; there are dedicated pages in the configuration for conducting these inspections: 1. System status check: cards, power supply (including safety barriers), online/redundant status of cards, CPU/cabinet fans, temperature (for Yokogawa original cabinets), etc., along with a sign-in function on those pages. 2. Switch positions related to control and interlocking: indicate the normal position and the current position. Sign in on the page at the same time. 3. Inspection log entries: inspection date, time, room temperature, and humidity. Status of cards, power supply, and fans in each cabinet.
1. Mainly, there are few staff members but many devices, making it impossible to conduct comprehensive inspections on a daily basis; inspections are carried out once every two days； 2. This power supply is not included in the system alarms ； 3. The fan operates at 220VAC and is integrated into the power module
The power supply units in industrial-grade devices are redundant; a shutdown due to power loss indicates that both of the redundant power supply modules have failed. There should be corresponding indicator lights and alarm signals in case of a power failure – perhaps there was a lapse in routine inspections, preventing the redundant power supplies from fulfilling their role
That is, there are very few personnel responsible for self-control; they can only conduct inspections once every two days. Moreover, this power module has been in use for 14 years now—it’s well past its service life. The company has no relevant regulations regarding this; the module is replaced only when problems arise. But the cost of doing so is simply too high!
Based on the leveraging approach used in your company’s system inspections, could you please provide information regarding the allocation of automation control personnel? Control systems have a long service life, and the cost of upgrading them is high; it’s not possible to upgrade them in a timely manner. Ah, working in this field is really difficult!
This post was last edited by jlshnlhj on 2022-9-2 at 08:06. There are over 10 people in one workgroup, responsible for maintaining 7-8 sets of equipment. Team leader, deputy team leader – no dedicated supervisor; 1–2 room administrators are assigned (responsible for different computer rooms), who are in charge of the maintenance and configuration of all the computer rooms, as well as DCS and ESD systems, and carry out inspections. If the room administrators are on leave, the deputy team leader or the team leader takes over the inspection duties. Otherwise, each person is basically responsible for the maintenance of one unit’s instruments (it’s also possible to have an apprentice working under them). We are also deeply aware of the shortage of staff and the uneven level of technical skills. The computer room is inspected once on rest days, carried out by the duty personnel.
Inadequate inspections are a way of shifting blame. Some things—staring at her won’t help at all. A reminder: attention should also be paid to the C300 power supply, as its fan tends to break easily.