I’ve learned a lot about DCS fault handling
Thread Content
I had previously seen discussions on HCBBS about how to deal with a DCS screen going black; I always thought it was something that happened very rarely. Yet just a few days ago, our DCS experienced such an issue – the screen stopped functioning and it was no longer possible to view the various parameters. This situation lasted for 30 minutes, until the instrumentation staff restarted the server, after which all indicators returned to normal. During this period, the main units come to a stop manually, resulting in a complete shutdown; in other words, they stop without being able to see all the parameters. Of course, the personnel on site can still transmit feedback information from the instrument readings to the control room. The entire distillation section was left untreated; only the product extraction was switched to the line for handling defective products. 30 minutes later, the liquid levels in all the main sections of the system were high, but the temperature was low. It took about half a day to a full day to address this high liquid level issue; fortunately, cold water was not injected into the system, otherwise it would likely have taken more than just a day or two to resolve the situation. Apart from one column whose temperature rose rapidly, there were basically no other changes in the system, with temperature, pressure, and liquid levels remaining stable. The next day, personnel from the DCS manufacturer came to conduct an inspection, but no issues were found. It’s unclear whether it was system instability that caused the server to crash, a problem with the connected hardware, or an error made by the operator – only heaven knows. Personal conclusion: One must be very familiar with the manufacturing process, to the point of understanding how any change in parameters can affect the system. One must have a holistic perspective; when dealing with individual components, it is necessary to consider the impact on other systems, especially utility systems such as water, electricity, and gas, which can directly lead to system failure. One must be very familiar with the operation manuals and emergency plans, so that in an emergency situation, no one will be at a loss. It is not only necessary to improve the skills of process engineers; it is also essential to raise the competence of those who work with instrumentation equipment. In this accident, instrumentation played a significant role in causing the problems. When issues arose in the system, the instrumentation technicians did not know how to handle the DCS system. Fortunately, foreign experts were available and were able to guide the emergency shutdown procedures despite the lack of instrument readings. If a similar situation occurs in the future, who will guide us, and do we have the capability to handle such problems on our own? Another issue is that the procedures for handling accidents are not clear; when such serious situations occur, there is no guidance on who should be approached to resolve them. We can rely on foreigners this time, but who will we turn to next time? ?
Thank you for providing a vivid commentary to one of our HaiChuan posts discussing the DCS screen going black! Thank you for sharing your experience
Thank you for sharing your experience: handshake
I was also shocked when I heard that: funk: As a process engineer, it’s indeed necessary to be familiar with emergency plans for dealing with various unexpected situations! Otherwise, without long-term considerations, there will surely be short-term troubles :)
No way – new processes should have corresponding emergency measures prepared before operation begins. To start operating without being familiar with the relevant processes and methods is really bold; those who do so learn from their mistakes, which is something worth taking as an example
It’s not a matter of lacking courage; there are emergency plans and procedures for emergency shutdowns. But what’s the use of those if, at critical moments, the leaders say, \"Wait, follow what the foreigners say.\" So what can be done? In fact, those shutdown plans only provide a framework; everyone understands the general approach. It’s just that without having experienced such situations before, no one dares to take on that responsibility. The cost of an emergency stop is that the entire system comes to a halt. If handled properly, it will take at least two days to return to normal operation; if not handled properly, it may take four days or even a week. Who is responsible for the losses incurred during this time? ? I think at this time, what the leader is thinking about the most is this. In a company like ours, stopping production for one day results in losses of over 2 million. If things can be managed well to avoid such stops, how much can that help reduce the company’s losses? Power outages and shutdowns experienced ； Shut down by stopping steam supply ； DCS is stuck and cannot display shutdown ； Forced partial shutdown after the pump runs dry for N hours ； Clogging of the flow meter caused some disruption in the system ； The emergency shut-off valve activated suddenly ； Forcing the valve to open by using brute force in order to reinstall the solenoid valve ； The solenoid valve activated unexpectedly, causing some of the system to stop operating ； The partial shutdown of the system for controlling excessive local temperatures in the hydrogenation reactor was experienced, as were various other emergency shutdowns caused by high temperatures, high liquid levels, low reflux ratios, and so on. I really don’t know what else might have happened during this drive. In fact, for beginners, having so many problems is a good thing; the more problems there are, the more experience you gain, and the more confident you will be when facing challenges in the future. Just like during those first few series of jumps, the operator was so panicked that he didn’t know what to do, but after a few attempts he was able to carry out the operations in an orderly manner and restore the system in a short time. This is also a form of growth. The power outage will last until the day after tomorrow before the system can be restarted; I’m not worried this time. Everyone has gone through a lot already, so we’re not afraid anymore……………
Yours seems to be fine with no problems, but ours is in a terrible state – the pipes have all burst, causing huge financial losses
Is it an ethylene plant? ! (Cooling water……) You really only realize the severity of DCS screen failures after experiencing them firsthand…… I remember that incident when a furnace was damaged beyond repair; the fire inside it burned throughout the night before it was extinguished… It resulted in losses of tens of millions! A valuable lesson! The summary is also very comprehensive...:L Calm and orderly manual parking is the only option; it’s important to be familiar with the manual control methods for the valves on site
It’s not ethylene; it’s phenol propylate. Fortunately, it’s only the DCS system that has a problem – the FSC system is fine. It’s possible to see whether the interlock has been activated from the control room, and manual emergency shutdown is also available. The foreigner also said later that no one had ever seen what an accident actually looked like, so they weren’t nervous. A foreigner in his 60s came here by taxi from the hotel in the middle of the night, extremely anxious. But in the end, there weren’t many problems, so I went back feeling relieved.
The more I think about it, the more scared I get. Our company is also going to install a set of DCS towers soon; I really don’t know what might happen. I’ll learn as I go and share this experience with everyone
It was very useful; thanks for sharing your experience
In fact, DCS not only requires quality-assured instruments to function properly, but it also needs us process engineers, instrument technicians, and others to maintain and operate it; otherwise, the situation becomes rather passive!
A DCS black-screening incident occurred in our heavy oil catalytic cracking unit in 2004; at that time, all five CRT workstations went black simultaneously. Since this was the first time such a situation had arisen, the operators on duty shut down the unit urgently as a precaution, activating the highest-level main air safety interlock system. There was no certainty at that time as to whether the interlock would actually function, so people were sent to verify it. As key parameters of the unit such as liquid level and pressure could not be displayed, others were sent to the site to operate based on the level gauges and pressure gauges. Until the instrument operator arrived.
Let me talk about the DCS systems I’ve seen, as well as the issues with PLCs. I’ve encountered several instances of the screen going black or freezing. The most serious case was when all the machines in a control room stopped working; we couldn’t operate the instruments there. My supervisor asked me to go and take a look, and I quickly restarted the machines (which were frozen). Two of them recovered right away, but one wouldn’t start. It turned out that its hard drive was damaged. Upon analysis, I found that there were as many as 42 system processes, with over 390 MB of memory usage. The hardware’s memory configuration was 256 MB (normally, industrial computers should use ECC memory rather than regular memory, but many industrial computers in China don’t use ECC memory). Analyzing the processes, I saw that many unnecessary programs were taking up resources, including some in the startup group. It seems that these programs were inadvertently included by the manufacturers during the development of the DCS systems, such as search tools (installed along with other software) and quick-start functions for PDF readers – processes that aren’t necessary for the DCS system or overall operation. How can such a system function without problems after operating under such heavy loads for several years? One more thing: from what I’ve checked, basically most DCS systems have bugs; anyone who knows how to do it can easily access them (that is, access the hard drives and read/write programs)
I really learned a lot. It’s really impressive to see all the cases related to DCS fault handling – you’re quite lucky, and of course your ability to handle emergencies is also excellent! :Lol, I guess it’s mainly the role of large-scale continuous production systems like DCS that makes them particularly important. In the workshop, production is semi-continuous and semi-batch, so the role of the DCS is relatively limited. Especially in the post-treatment processes, the DCS can only be used to monitor parameters, with very few operations being carried out on the DCS in the control room. :L
I’ve gained a lot of knowledge; thanks to the original poster. DCS is one of the key elements in chemical production
It seems that anything can happen; it’s essential to have plans in place for dealing with emergencies
The DCS is offline; one must not be greedy – stop operation when necessary. Ensuring the safety of the protection devices and the catalysts is of utmost importance. On-site, it is necessary to verify each interlock, and at the same time ensure that all equipment does not experience overheating or overpressure.
Emergency plans should specify the principles for handling such situations; generally, priority is given to ensuring human safety, followed by equipment safety, and finally process safety.
It’s really impressive; it’s an experience that can be useful for a lifetime!!! Also, I think it would be very helpful if every operator could perform manual operations skillfully on-site while carrying out their tasks!!! Of course, one needs to be very familiar with those devices and parameters as well!!! Hehe:handshake