It’s a case without a clear solution; I hope experts can help analyze it
Thread Content
One day, three crude oil pumps, B, C, and D, were in operation in the crude oil tank area; pumps B and C stopped almost simultaneously. There are no records in the comprehensive insurance database. The switch was pulled to the test position; it was opened immediately after being closed. It was found that the instrument interlock trip contact had activated (the instrument interlock contact was not connected to the comprehensive protection system). The SCADA electrical monitoring system shows that the switch tripping time for pumps B and C was at 10:14. The instrumentation team checked the DCS records and found that at 10:08, trip signals from the switches of pumps B and C were received, resulting in a significant drop in the flow rate at the outlets of pumps B and C. When the flow rate falls below a certain set value, interlock is activated to send a signal to trip the switch and shut down the system. Currently, there is a significant disagreement between the electrical engineering and instrumentation teams, with each claiming that the accident was not caused by their own mistakes. Reasons for the electrical engineering major: 1. The comprehensive protection system had no accident records, and the equipment operated normally thereafter. This indicates that there is no problem with the device. 2. The switch operation time recorded on SCADA is 10:14, a few minutes later than the 10:08 recorded by DCS. Both systems use GPS for time synchronization, with errors on the millisecond level. The switch opening signal received on the DCS may be a false signal. Furthermore, it was found that the DI for pumps A (which was not in operation at the time), B, and C shared the same card, while pump D used another card; yet pump D did not trip. Although it worked properly later on, which indicates that the component wasn’t damaged, it’s possible that it was interfered with at that time. 3. The switch for Pump B is powered by the busbar in Section A; Pump C is located in Section B, and the two switchgear cabinets are separated by more than 10 meters. The possibility of pumps B and C tripping and sending signals simultaneously is very low. At the same time, the chance of interference can be said to be none. Reasons for the instrumentation specialty: 1. The DCS did receive a switch tripping signal; even if it was a false signal, why would the outlet flow change suddenly at the same moment? 2. Currently, all pumps are operating normally, and there are no issues with the DCS cards. Everyone has their own reasons, and no one can persuade the other. Hehe. I would also ask the experts to help analyze it and see what the possible causes might be If it’s due to an electrical issue, why does the DCS receive the trip signal while SCADA does not? And there are no records of any protective actions either? If it is due to interference from the instrument signals, then why does the flow rate also change suddenly?
Why does the poster have to be so precise? It’s fine to just let it go without getting too detailed. We’ve seen many such incidents – in March alone we had 3 cases like this, similar to your situation: there was no alarm from the electrical system, no faults in the instruments, and no one adjusted the process or started/stopped the pump, yet the pump stopped on its own. No one would admit that there was a problem with them. We held numerous discussions, but nothing was resolved in the end. In my opinion, someone here must be hiding the truth in order to avoid taking responsibility and having to report it ； There are indeed some unexpected factors that cannot be detected by humans, just as the original poster mentioned, such as false signals from electrical systems and instruments ； There are also those connection points; for some reason, they might become loose and fall off temporarily only to be reconnected later (just like a light switch that works intermittently, sometimes on and sometimes off). I remember one manager telling us that accident analyses don’t necessarily aim to assign blame to certain people, but rather to determine the causes so as to prevent similar accidents from happening in the future. Therefore, for those ambiguous aspects, we need to apply fuzzy processing: in the case of DCS, we should schedule another time to re-calibrate the interlock system and check the wiring of the circuits as well as any issues related to corrosion or moisture ； For the electrical engineering field, check the circuits as well as the settings and operation status of the integrated protection systems ； In terms of equipment maintenance, if possible, the pumps can also be serviced and repaired; furthermore, stricter inspection procedures can be required for the operational aspects related to the process. Any area that might be related to the accident can be mentioned; carry out repairs where necessary, conduct inspections, and strengthen management where required. It’s perfectly fine to write an accident analysis report in this way as well.
This post was last edited by zhaohh3211 on 2010-6-2 at 11:10. The DCS received the signal and the interlock took effect immediately; the pump should not have been able to start. Why did it take until 14 minutes before the pump stopped working?
Our interlock has two settings: one in which it trips immediately upon the appearance of a signal, and the other in which it trips after the signal persists for a certain period of time (if the signal disappears during this time, it does not trip; timing starts over once the signal appears again). The appropriate setting is chosen based on the specific process requirements. I’m not sure if the original poster mentioned anything about delay. Let’s repeat the details of that incident we had: just like in your case, one pump stopped operating without any apparent reason; there were no records in the electrical or instrumentation systems either. Due to the delayed detection, the interlock system activated after 10 minutes, at which point it was discovered that the pump had stopped. After analyzing the cause of the accident, before the analysis was completed, another issue occurred: 2 units stopped operating at the same time. Moreover, once they stopped, the interlock system could not be reset, which prevented the pumps from starting. It was only after the third unit also stopped that the interlock system could be reset and the pumps were able to start again. By reading the interlock diagram, the setting should not be like this. Moreover, in the subsequent tests, the interlocking and on-site settings conformed to the interlocking diagram. It’s the period when the pump is stopped that’s abnormal. Later, the various teams met to analyze the cause, and the primary reason identified was that someone had touched things they shouldn’t have (such as on-site buttons, DCS cards, and the ‘drawer’ used to cut off electrical power, etc.), but no one admitted to doing such a thing ； Turning to the second reason: loose or detached wiring, as well as moisture-related issues in the electrical and instrumentation systems; various teams checked the wiring and found no problems ； Due to the high level of specialization in electrical and instrumentation matters, there is no one who understands both process engineering, instrumentation, and electrical systems to check all the settings and accident records; as a result, it is not possible to confirm that what each specialty claims is true. In the end, not even an accident investigation report was issued. But from a technical perspective, there is always a reason for something to happen. I consulted with the equipment specialists, who said that if the pump stops operating under normal flow conditions and it’s not due to human error, then there are basically no other reasons that could cause the pump to stop working under normal circumstances. There is no excess current, no overpressure, and no jamming, and the pump starts up normally again. Thus, it once again becomes an unjust case without a solution. Therefore, personally, I suggest that since each specialty adheres to its own views and there are no abnormalities, what we can do next is carry out some routine inspections and maintenance tasks, identify potential accident risks, and reduce the likelihood of such accidents occurring again.
I think the key question is whether the export traffic really decreased at that time Why has it decreased? But it’s probably impossible to find that out anyway... It doesn’t matter; there’s nothing to do about it!
Is there a control valve at the rear end of the pump outlet? It is recommended to check the curve diagram during tripping. Sometimes, certain specialties hide from each other some other critical factors that are not taken into account.
There is an electric valve at the pump outlet. During normal shutdown, the electric valve is turned off first, and then the motor is stopped.
The instrumentation team checked the DCS records and found that at 10:08, trip signals were received for pumps B and C, resulting in a significant drop in the flow rate at their outlets. When the flow rate falls below a certain set value, interlock is activated to send a signal to trip the switch and shut down the system. When an electric valve at the pump outlet closes due to misoperation or signal interference, the DCS determines that the pump flow is insufficient and sends a trip signal. Both systems use GPS for time synchronization. Sometimes the DCS time is adjusted, and sometimes the system itself speeds up the time by a few minutes (the reason for this is unknown); it is recommended that both parties check the time together.
This is because the GPS units of the two systems are installed in different locations, and both systems consist of one GPS unit with many sub-systems. Therefore, at that time we specifically compared the time of the electrical monitoring subsystem in that area with that of the DCS substation system, and found that there was little error, at least to the second.