Many healthcare imaging and IT professionals have to deal with troubleshooting, testing, and validating connectivity and interoperability. There can be multiple objectives for this testing:
· A software engineer might test his or her newly developed software,
· An integration engineer needs to test connectivity between different devices,
· Application engineers test interoperability,
· Service and support people try to determine why something does not work or stopped working, and
· System administrators need to deal with finger pointing between different vendors and locate the problem source.
Another important test activity is acceptance testing by a user, often represented by a consultant, to determine whether the system works as specified and meets the initial requirements.
A common categorization of the different test tools is as follows: Test systems, simulators, validators, sniffing tools and test data sets. Many of them are available for free or as open source, some require a modest licensing fee. Test data is generated by either standards organizations or trade associations. Below outlines the characteristics of these tools – when they are used, what tools I recommend followed by a list of where to download them and find tutorials on how to use them.
1. Test systems:
Test systems are either a copy of the system to be diagnosed or a system with equivalent or very similar behavior. Specifically, if you have a PACS, you might have a “test-server,” which is another license of the same database as used for the production PACS. The test system could be running on a high-powered, stand-alone mini-server, which could possibly store images for a week or so. Most users purchase or negotiate a test system to be available as part of the new purchase. A recent OTech survey showed that about 40 percent of the PACS users have a test system. Another option, in case you don’t have a test system, is to use a free or open source PACS application such as Conquest or DCM4CHEE.
In addition to the PACS “back-bone,” one should always have at least one and preferably two additional test viewers from different vendors for displaying images. Examples of these viewers would be KPACS, ClearCanvas, Osirix for Mac’s and several others, all of these are freely available.
For EMR’s , I found that it is uncommon for users to have a test system available, which is kind of surprising. In many cases, a production server might be loaded with test data at system installation time, but as soon as the user training is complete and the system goes operational, this information is typically wiped to get ready for the production data. EMR’s are also quite different with regard to their functionality and interfaces; therefore a free or open source EMR might not be as useful as a test PACS system. But one could use the CPRS/VistA EMR, which is developed by the Department of Veterans Affairs and is available as open source.
The best-known interface engine that is available for free is Mirth. This device maps several different interface protocols, but is at its best when being used for HL7 Version 2 message mapping. I found it somewhat hard to use, but there is (paid) support available for someone who needs to configure the mapping rules.
One can use a test system to test new modality connections to a PACS, new interfaces (e.g. lab or pharmacy) to an EMR, or for reproducing certain errors. In the case of a new image acquisition modality connection, one could create test orders that would show up at a test worklist (the DCM4CHEE PACS has this capability), and query the worklist from the test system. This allows for the proper mapping to be tested from the orders to the DICOM worklist, and to tune any additional configuration needed to make sure that the worklist does not have too many or too few entries. The same applies for external interfaces, e.g. for lab or pharmacy to an EMR. It is usually better to test connectivity prior to actually going live.
There are those who use these test systems as a basis for their production, i.e. as their primary clinical system. For example, it is not inconceivable to use the VistA as an EMR, the Mirth interface engine as the HL7 router, DCM4CHEE as a PACS and modality worklist provider, and ClearCanvas for image viewing. However there are potential liability issues for using non-FDA approved and/or non-certified software for medical purposes, especially if used for primary diagnoses of humans. But, for veterinary use, these test PACS systems are relatively widespread in clinical use. I would not recommend using any of these in a production environment unless you have a strong IT background or can rely on a strong IT department or consultant.
A simulator is a hardware and/or software device that looks to the receiver to be identical to or similar to the device it is simulating. An example would be a modality simulator that issues a worklist query to a scheduler, such as provided by a RIS, and can send images to a PACS. If the simulator assumes the same addressing (AE-Title, port number, IP address) as the actual modality, such as a MRI, and sends a copy of the same images, the receiver would recognize the data the same as if the transaction came from the actual device. The same can be done for a lab simulator to an EMR, exchanging orders and results, and for a CPOE simulator sending orders and arrival messages. The advantage is that these simulators provide a “controlled“ environment, while providing extensive logging.
These simulators are typically used to test connectivity prior to having an actual operational system available in order to simulate and resolve error conditions and troubleshoot connectivity issues. They can also be used for stress testing and evaluating performance issues. One should note however, that a simulator does not exactly reproduce the behavior of the device it is intended to simulate. If there are timing related issues, or there are semi-random problems, one would try to keep the original configuration intact as much as possible and use sniffers instead to find out what is going on. I use an HL7 CPOE simulator and DICOM modality simulator, available from OTech. One could also use the various DVTK simulators, but these are not trivial to use and are therefore almost exclusively used by test and integration engineers. DVTK simulation tools also are programmable using a proprietary script, which makes them very useful for exception, performance and error testing and simulation.
A validator is a software application that validates a protocol or data messaging format against a standard set of requirements or good practices. These are extremely useful for testing by development and integration engineers, especially for new releases and new products.
I am amazed by how many errors I find when running a simple DICOM image against a validator. I personally believe that there is no excuse for these errors as these tools are available for free in the public domain. DICOM protocol and data formats can be validated using DVTK.
Another useful tool provided by DVTK is “file compare.” If there is any suspicion about data integrity, i.e. whether a vendor adds or removes information from a header, which could cause problems, one can simply compare the original and “processed” one to see the differences. In addition, this compare tool can be configured to filter certain attributes and highlight the ones that one is looking for. I have used this tool to determine any changes in software, which supposedly did not impact the data format by running this against the image before and after the upgrade.
The HL7 messaging workbench, developed by the late Pete Rontey of the VA, is a great test simulation and validation tool for HL7 Version 2 messaging.
For information exchanges between EMR’s, the CDA data format is emerging as the standard. This is an area where we might expect a lot of potential issues in the near future as these EMR’s are being rolled out. Its data format and compliance with the required templates can be verified on-line on the NIST website.
4. Sniffing software:
Sniffing software requires access to the information that is exchanged, which can be done by having the software installed at one of the devices that interacts with the connection to be monitored, which could be the device sending or receiving the information, or at a network switch, or connecting the sniffer to the link to be intercepted by a simple hub. This could be somewhat of an issue as many institutions clamp down on their networks and do not allow for a “listening” device to be connected as they are afraid that it compromises the network integrity. The de-facto standard for sniffing and analyzing DICOM connections is Wireshark, which used to be called Ethereal.
However, one could use Wireshark for the sniffing only, and asking the network engineer to provide you with the so-called .cap file, which can be captured on any of the available commercial sniffer and network management applications. The analysis can then be done separately using Wireshark. Sniffers to detect any semi-random and not easily reproduced errors, or to troubleshoot in situations where the error logs are either non-comprehensible or non-accessible, or to prove that changes are made in data before sending the information. A combination of a sniffer and validator is especially powerful, for example, one could upload a capture file into DVTK analyzer/validator and analyze both the protocol and data format.
Using a sniffer is often the last resort, but it is an essential tool for those hard to diagnose problems. As examples, I have been able to diagnose a device that randomly would issue an abort, which would cause part of the study to fail to be transferred, or to determine the errors as exchanged with the status code of the DICOM responses, and to find that query responses do not quite match the requests, and to resolve many other semi-random problems. One can easily configure the sniffer to capture all of the traffic from a certain source or destination, store it in a rotating buffer and when the problem occurs, start analyzing the information.
5. Test data
If a problem occurs with clinical data, it is often hard to determine whether the problem is caused by corrupt or incorrectly captured data, or whether it is a result of the communication and processing of the information. Therefore, having a “gold standard” of data is essential. Imagine a radiologist complaining that an image looks “flat” too “dark or light” or just does not have the characteristics he is used to seeing. In that case, being able to pull up a reference image is invaluable. There are not only sample images, there are also sample presentation states, structured reports and CDA documents available.
Most of the test data objects are created by IHE to test conformance with any of their profiles. For example, there are extensive data sets available to test proper display of all of the different positions indicators (and there are quite a few) on digital mammography images, together with correct mapping of CAD marks.
The same applies for testing the imaging pipeline, for which there are more than a hundred different test images which are encoded using almost every possible combination and permutation of pixel sizes, photometric interpretation, including presentation states. The nice thing is that the data is encoded such that the effect of the image display always is identical. For example, one might have an image for which the header says that the information should be inverted with the data to be inverted so that the ultimate end effect is that the image looks the same as the non-inverted test sample.
It is easy to load all of these images onto a workstation where you’ll see almost immediately for which image the pipeline is broken. This is a great test tool to be used when doing an acceptance test or to check after a new software upgrade is installed at your workstation, and you would be surprised how many systems do not render all of these correctly.
For verifying display and print consistency, the AAPM has created a set of recommendations and test images, both clinical and synthetic, which are invaluable to determine whether or not your display or printer supports the Grayscale Standard Display Function, also referred as “the DICOM curve,” and, if so, if it is properly calibrated according to that standard. A simple visual check will make sure that certain parts of the test pattern are visible and indicate compliance or the potential need for recalibration. Even if one uses non-medical grade displays, there is no reason NOT to calibrate a monitor or printer according to this standard (there are after-market devices and software available to do this) and to make sure they stay in calibration.
In conclusion, I am convinced that any connectivity issue can be visualized, located and resolved by using the right set of test, simulation, and validation tools using a wide variety of test data. It is just a matter of learning how to use these tools and applying them for the appropriate circumstance. In addition, they are also invaluable for acceptance testing and to prevent potential issues. Healthcare IT systems are not plug-and-play and never will be, and a healthcare imaging and IT professionals therefore will need to master these tools to ensure data integrity.
Where to find information and how to access the tools mentioned above: (all of them are free or open source unless mentioned otherwise)
- Test PACS systems: Conquest and DCM4CHEE
- Test viewers: ClearCanvas, KPACS, and Osirix
- Interface engine: Mirth
- Modality and CPOE simulator (requires license): OT-DICE and OT-Send
- Modality worklist simulator: OFFIS
- Test EMR: VistA
- Validation tools: DVTK for DICOM protocol and data formats, Messaging Workbench for HL7 validation and NIST for CDA formats
- Sniffer: Wireshark
- Test images: AAPM, IHE
- There are tutorials available at the OTech Youtube channel on how to install and use most of these tools.