Tuesday, December 31, 2013

Top 10 practical IT skills every PACS administrator (IIP) should have.

Most imaging and information professionals (IIP’s) who are taking care of PACS and EMR systems have some type of IT background or, if they started their careers in the clinical field, have taken courses in this area or learned on the job. The IT knowledge that is required in a particular job depends often on whether there is a strong IT department that supports the healthcare imaging and information systems to supplement the IIP skills, but regardless of the scope and strength of the internal IT support, it is always good to not have to rely on an external person who is often in another department, possibly outsourced or centralized and only available through yet another help-desk person. Therefore, being able to troubleshoot and diagnose fundamental problems is often invaluable, especially if there is “fingerpointing” going on such as, “it is not the network,” or “my system works fine.”

Here is my list of basic skills that every IIP should have based on discussions with the many PACS administrators attending our PACS training and derived from questions I have seen in the many PACS user forums:

Overview of the basic skills required:

1.       Network diagnostics: IIP’s should be able to isolate whether a problem is related to the network and its infrastructure (switches, routers, etc.), including bandwidth problems, or if it caused by an application or device. The most common networking problem is caused by cable cuts. Therefore, being able to find out if a connection is still live, is invaluable (ping). The second most common problem is that IP addresses might be re-assigned or expired, and therefore being able to check a device IP address is important (ipconfig). Performance and routing issues can be diagnosed with TraceRT, while netstat checks whether a port is still open or might have been closed by a well-intentioned IT person. And nslookup will allow you to troubleshoot any potential DNS issues. Knowing the command to be able to renew an IP address is also important.

2.       DICOM connectivity diagnostics: Assuming that the network is OK, the next step is to look for a connectivity issue with imaging devices using a DICOM protocol by analyzing any accessible log files. A typical DICOM connection always starts with connection (aka Association) negotiation, which is executed by an Association Request and corresponding Accept or Reject. A reject would definitely raise a red flag. The second part of the connection would be the actual DICOM command Request (Store, Find, etc.)  with the Response and, most important, its return status code (success or fail for whatever reason). You would want to look for any non-success reasons to be returned. Lastly, there should be a Release Request and Accept finishing the protocol. Being able to test negotiation of the connection, sending out a test commend (“Echo”) and successfully releasing the connection using DICOM Verification is invaluable. Most devices have this feature available from a service or tools menu, some even have this accessible from the GUI, look for “DICOM Echo,” “Verification,” “DICOM ping,” or simple “Test,” to perform this function.

3.       DICOM header knowledge and fixing capability: Assuming that the DICOM connection works, i.e. it passes the test under (2), there could be a problem with an image that is either misidentified, i.e. information is missing from its header or meta-data, or that certain information in the header potentially jeopardizes the data integrity of the receiving system because of duplicates or contradictions in the identifiers. This can cause images to be flatly rejected, or accepted by a receiver and ending up “unverified” or “broken.” Most of these cases are caused by incorrect data entry, for example, by using an already existing Accession Number, misspelling a patient ID, incorrect patient selection by a technologist, etc. Most Imaging and information systems have an elaborate set of tools to fix, merge, and split studies to solve these issues. However, there are some error cases that cannot be fixed with the commonly available tools. These can occur when trying to import an external study from a CD, or, which is increasingly more common, when migrating up to millions of records from one PACS system to another. An IIP should be able to recognize the problem (e.g. duplicate SOP Instance UID, or Institution Name exceeds maximum characters) and be able to use tools such as OT-DICE or DVTK-edit that are typically stand-alone and allow for fixing these headers. Note: It is recommended that you use a tool that is reliable and known to NOT create any corrupt DICOM headers, as some of them do, and only making changes that you feel comfortable with as some of those changes, notably the UID’s, could potentially impact the data integrity of the system.

4.       HL7 messaging protocol: A PACS system is fed by an order and sends back results, while often using arrival information as a trigger to add information on a worklist for a modality. These transactions are encoded using the HL7 protocol. In its most basic form, IIP’s should be able to determine if the HL7 feeds are alive and working, for example, by monitoring the incoming orders and outgoing reports. This is critical, as a one-hour downtime of HL7 feeds, for a busy department, typically causes 5 to 10 hours of fixing by a IIP on the PACS side because of all of the unverified studies caused by manual patient and order input, and its corresponding misspelling, missing accession numbers etc. at the modalities. Equally important is to determine whether the reports are going out, especially for STAT or emergency cases as physicians are waiting for those results to be available within 15-30 minutes. In many cases, the HL7 orders are received by a worklist provider, aka broker or connectivity manager, and the reports are sent from a Voice Recognition server. Knowing how to monitor these devices, and being able to restart them is essential. Sometimes, a queue might still be available at an interface engine and it simply requires restarting that queue for the orders or results to be resent.

5.       HL7 transaction knowledge: In addition to knowledge about the HL7 transactions, an IIP should also be able to know the details of the actual encoding of these messages. For example, most of the modality worklist information is generated by an HL7 order and it is not uncommon that after an update from the information system or interface engine, certain information is suddenly misplaced, absent or incorrectly encoded. The good news is that the HL7 transactions are all encoded as readable ASCII text and it is therefore relatively easy to find out from a HL7 message what could be missing or incorrect just by looking at the message in Notepad. If there are some questions about the encoding, one could use a reference book, or tool such as OT-SEND to parse the message and identify the problem to be resolved by the IT department or vendor.

6.       UNIX commands: There are only a handful of PACS systems that use UNIX or derivates (Linux, AIX, etc.) for their main viewing applications or other PACS workflow and archive components. However, there are several that base their core (database servers, archive servers) on UNIX. The good news is that these systems are so reliable that one needs to access this information very infrequently, however as a result, the knowledge on how to maneuver within this operating system might become stale. In any case, an IIP should know the most basic UNIX commands such as how to start and stop processes, show the process status, and perform basic file and directory management functions, as well as network administration (check and set IP address, etc.).

7.       Windows knowledge: Most workstations are using Microsoft Windows® operating systems and/or applications such as Explorer® and others as their core. An IIP should be able to check process status, kill and restart, be able to check and set firewalls, configure IP addresses and net masks, as well as proxy servers, and check system hardware and software configurations. Knowing how to restore a standard back-up “image” of the OS and relevant applications is critical as well.

8.       SQL queries: Almost all of the commercial databases use relational databases as the core of the Imaging and Information management systems ,and if not, for example, when they use the upcoming NoSQL databases, they still have a standard query interface based on SQL. Having access to the main database to perform custom queries is important for troubleshooting as well as for providing essential statistical information. For troubleshooting, imagine that images were sent to the PACS but they “disappeared,” meaning that they did not appear on a worklist for a radiologist and also did not appear in the “to be verified,” or “broken” study queue. This is an increasing problem as departments go paperless. In the past, there would be a piece of paper, i.e. requisition that would be given to a radiologist that he or she can use to find the study that was performed and has to be interpreted. If a department is paperless, there needs to be more checks and balances to make sure that nothing falls between the cracks as there is no visible evidence of that. Being able to access the database and find out using smart queries of what was added to the database is often invaluable. After being able to diagnose the issue, fixing it can be done either by the IIP, or by the vendor, or IT department, depending on whether the IIP has both read/write access. In addition to troubleshooting support, for statistics, it is very useful to be able to mine the database information for data analytics such as finding out the turn-around time of certain exams, how many images were created by certain modalities by certain technologists, how many studies were performed at certain modalities, etc.

9.       Monitor quality assessment: Monitors degrade because of their nature. Their light source degrades over time, which means that compliance with the DICOM calibration as well as meeting the common guidelines for maximum luminance has to be verified on a regular basis. Verification is often done automatically because the vendor has built-in sensors and calibration software that checks the performance regularly. Calibration has to be checked manually on a regular basis, depending on the application, for example, those used for digital mammography might require checking weekly and those used for other specialties monthly or even annually. A quick visual test can be done by using an appropriate test pattern, which would show any obvious issues with the proper mapping from the digital values onto the proper grayscale values on the screen. Being able to bring up this test pattern and interpret it is critically important, especially if there is a question about the image quality.

10.   Last but not least, there is one generic skill that is important to troubleshoot any problem, and that is being able to locate issues using simple logic and exclusion. One starts with identifying the area of concern and systematically excludes everything that seems to work to find the culprit. This skill is hard to teach and grows with experience but is essential to be able to diagnose any IOT related issues.


As a final note, I see many postings and questions in user groups about issues that could be easily diagnosed by using the right tools. Remember, it is all about visibility, for example, if a worklist does not work, try the tools as mentioned above to test the network, application and look in the log files. I am convinced that if every IIP would know how to use these tools and would be allowed by the IIP vendors access to the systems to use them, there would be less fingerpointing, less downtime, and better system support. Also, remember that by mastering these tools you empower both yourself and the organization you work for by visualizing issues instead of having to wait and rely on external experts either from the organization or your vendor. Lastly, you might want to consider getting certified to show to your employer and the outside world that you have learned these skills. Most of these are requirements included as the IT portion of the PARCA CPAS PACS administrator certification (see the OTech training schedule for more details about the schedule for CPAS certification).

2 comments:

  1. Hi, Your blog is really very informative and useful for me. Thanks for sharing this valuable blog.
    Regards..
    Unix Training Institutes in Chennai

    ReplyDelete
  2. The resume headline is for all those job seekers who have problem in writing resume. This is an increasing problem as departments go paperless.

    ReplyDelete