PACS troubleshooting tips and tricks series (part1): Network transmission.

Over many years of PACS consulting and training thousands of healthcare IT professionals, I think I have heard just about every issue they experience. This series lists the most common issues, how to diagnose them and what tools to use to resolve them.

Network transmission – The PACS components (workstations, archive, database, worklist manager, routers, VNA) talk among themselves and the digital acquisition modalities use the DICOM protocol to exchange images and related information. The DICOM protocol uses the TCP/IP communication protocol for a reliable information exchange. Reliable implies that it automatically adds a check digit, or cyclic redundancy check (CRC) to each packet transmitted, which is used to make sure that packets are delivered without any data loss or corruption.

The software will automatically retransmit a packet, in case it is corrupted, up to a certain number of times. The transmission and acknowledgement of these packets is asynchronous, which means that there could be several packets in the pipeline before an acknowledgement is expected. In many cases, the sender might be faster than the receiver, which means that the buffer at the receiver can get full, and the “window” of the sender is down to “0” for sending additional packets. This is part of the “sliding window” part of the TCP/IP protocol.

Issues related to network transmission include slow network performance, image transfers and subsequent retrievals taking a long time, and TCP/IP network time-out messages.

The causes could be:

·        Poor network components – In one particular case the wall connectors were sub-par (they purchased them from the now defunct Radio Shack store) causing a lot of retransmissions. After replacing them with high quality components, the problem was solved.

·        Slow network segments – A teleradiology system in Alaska was dealing with a lot of time-outs, not realizing that the link between the main hub in the village back to Anchorage uses a satellite connection with a considerable delay. We had to adjust the TCP/IP sliding window size at the network which solved the issue.

·        Someone “hogging” the network – At a small clinic with a limited WAN capacity, I found that the medical records department was streaming video using pretty much all of the bandwidth, causing a considerable delay for images to get through. Better IT policies took care of this. Note that a virus or malware can have the same effect.

·        Duplex settings – I also have seen issues with the duplex auto-negotiation, but unless if you have very old devices (and they are still around), these seem to be pretty much gone. This will manifest itself with very slow transmission and can easily be resolved with changing the setting at the router.

·        Application software issues – Even though it is technically speaking not a network issue, application software also could contribute to transmission performance issues. I have seen cases where a PACS application would literally take seconds (sometimes up to 10 or 20 seconds) to respond to a DICOM command. Note that DICOM implementations almost all are implemented in a “synchronous” mode, meaning that a sender will wait for a reply on a DICOM command before it will follow up with, for example, sending the next image in a series. Therefore, a delay in responding to a DICOM transaction will directly impact the transmission performance.

How to troubleshoot these types of issues?
Looking at log files will help, TCP/IP errors will typically be listed, however, a much better tool is the Wireshark network sniffer with a DICOM “plug-in,” which means that it will list the DICOM protocol sequences, so you can easily see how much time it takes between, for example, sending a DICOM command and its reply. In illustration 1 you can see the time it took for a simple DICOM “Echo”, i.e. 0.147517 seconds.

Network issues are somewhat tricky to troubleshoot, but with tools such as Wireshark the problem will become obvious. Note that if your IT department does not allow you to connect to Wireshark, they almost certainly have their own sniffers running at their routers and you can request the .cap file, which is a standard format for capture files, which you can upload it into your Wireshark and look at the DICOM commands.

For additional resources, see the “how to use Wireshark video,” the DICOM textbook (also in ebook version), or attend our PACS SA training class where we spend a lot of time troubleshooting and learning the details about using Wireshark.

Look for the next “troubleshooting tip” in this series on Addressing.