I recall an experience of troubleshooting a communication issue, but I couldn’t figure it out!

Thread Content

On the day before yesterday, a DP network failure occurred in the company’s shampoo formulation process; several touchscreens were unable to communicate with their respective PLCs, and the host computer used for managing formula downloads had intermittent communication issues with those PLCs. At that time, upon hearing about the above situation, I thought it was a loose connection and asked the workshop mechanic to handle it, but after spending a long time he still couldn’t fix it. When I went to the site for work, the situation was as described in the previous paragraph. I first had the workshop operator describe what happened before the fault occurred; it seems that two frequency converters were replaced by an electrician, and the disassembly and assembly of the refining machine involved working on the motors and cables. After the frequency converters were replaced, the above-mentioned fault symptoms appeared once power was supplied again. System configuration: I observed the networking setup in that workshop. There are a total of six separate S7-300 PLC system cabinets, along with one Wincc host, which communicate via a DP network. In addition to these 6 devices, the DP network also includes 15 IND221 weighing instruments and 8 touchscreens associated with each PLC, resulting in a complex and chaotic DP network whose structure is difficult to discern. The DP cables between the cabinets run through the attic spaces and pipelines. The SF lights of each PLC are on while the BF lights flash; the data on some MP370 touchscreens is intermittent. The TP177 screen door remains at ## status since it does not recover automatically (unless powered back on). Cause analysis: System equipment was available in the periods from 2010, 2012 to 2016 and 2018; there was considerable corrosion and aging of the DP connectors. No problems were detected using a multimeter. Replacing the connectors and dealing with them properly could improve the situation. Alternatively, it might be possible to reduce the communication speed (this would need to be done for all networks, which is somewhat troublesome, especially since some programs lack backups; poor network performance makes uploading impossible). Another option is to replace the power supply. Damage to the DP cable in the interlayer or conduit caused the signal lines to ground, and an electrician took a quick look but didn’t detect any obvious damage. Fault handling: Since I only have 3 spare DP connectors available, I could only replace those that were visibly corroded. After replacing them, I turned on the terminals of some of those connectors, and the SF and BF lights of three PLC units went out, indicating that the problem lay in the remaining wiring. However, with other lines, it is only possible to clean the connectors and then turn the device on at the terminal; faults are reported on both the front and back sides, making it impossible to determine whether it is one device or several devices that are faulty. Three days have passed by now, and the workshop can’t wait any longer; it wants the original manufacturer’s technicians to handle the issue on site. They brought 10 DP connectors that were not from Siemens; after checking the conditions on site, they replaced all the DP connectors that appeared to be faulty, and then observed the situation again. As a result, the SF and BF lights on five sets of PLCs stayed illuminated constantly, while only one set of PLCs functioned properly since it was connected to a terminal in an operational state. Then all communication with the host was also cut off; in fact, one PLC stopped working altogether because the slave instrument had its terminal set to ON. This left them stunned, and we were also panicked – production could no longer proceed. They said a genuine DP connector was needed and asked us to go buy one; how could we get it so quickly? After reporting the situation to my superiors, I asked the manufacturer’s technical staff to replace it with the old connector. First, I found a PLC cabinet that was still showing BF lights (the communication with this cabinet was intermittent). I replaced the old DP connector, and after doing so, I noticed that the SF and BF lights were also on constantly; meanwhile, the communication with this cabinet was interrupted. They think the newly replaced connector is useless, and it’s likely impossible to restore things by putting the old connector back in. Ah, what bad luck – it was barely usable before, and now it’s completely useless! I suddenly thought that although it’s not the original version, it probably won’t be completely useless. Initially, when he replaced the CP5611 card connector, communication was still intermittent; now the SF and BF lights staying on constantly indicates a loss of signal. As long as the old components are replaced one by one, communication can be restored if a few of them work properly. Also, I wonder if the PLC detects the loss of connection of another master station – even though there is no DP configuration – and whether in that case the SF and BF indicators stay lit. I went over and turned off the PLC that was working normally with its terminal set to ON; damn it! The SF and BF lights of this PLC are not on, while the SF and BF lights of all the other PLCs are completely off! The host is also communicating normally now! Although this situation is unexplainable, the problem has been ruled out, and everyone can finally breathe a sigh of relief. (Comment: Could it be that this station was not originally a DP terminal, and after being manually set to ON, it prevented the subsequent DP stations from being detected, thus causing the system to report a bus error?) ) Debugging insights: The flashing BF light on the S7-300/400 PLC indicates an unstable network, while a lit BF light indicates that there is a master or slave station missing from the network. If network instability occurs due to such old networks with unclear connections, be sure to bring enough reliable DP connectors and DP cables to replace all of them; after all, sorting out the incoming and outgoing cables takes more time, and replacement is still necessary in any case, albeit in smaller quantities. If it is found to be a station drop, either the connector at that station is faulty or the station itself is damaged. It is also necessary to have DP connectors, interface modules, and a sufficient amount of DP cables available; having a professional wire crimper would be even better. Although there is no data communication between the various PLC master stations, the CP5611 controller is configured to communicate via DP with the older S7-300/400 PLCs; when the master station goes offline, it is estimated that the SF and BF lights on the PLC will remain lit (I have observed that when there is no host connected via Ethernet, the BF light on the PLC flashes). In the design, efforts are made to minimize the number of DP slaves and DP connectors within the same network. Each master station is equipped with dual DP ports (or PN/DP dual ports, or dual PN ports) in order to isolate the small networks containing slave DP devices from the larger networks connected to the master stations (of types 1 and 2). For example, for small DP networks, another DP network or a fiber optic PN network can be used when the distance is large; PN Ethernet is sufficient when the distance is short. This approach allows maintenance personnel to quickly locate the source of faults, while also minimizing the impact of any failures that occur. For this chaotic unified DP network of ours, it is planned to maintain the DP network within each main station cabinet going forward, while the main stations, touchscreens, and hosts will be upgraded using a large PN network with a centralized switch. Last but not least, in a DP network, all DP cables should have the addresses of the starting and ending DP stations marked, to facilitate problem diagnosis.