Summary: Claims Payments processing started processing slower and got behind after 3/17/21.
All times are in PST
Date/Time | Activity |
---|---|
2021-03-23 9:23 AM | ERAs & 277s from Trizetto not getting processed and tracked under the Quick View > EDI Received screen was reported via the RCM Team & Customer Support. Found files processed prior to 3/18 processed much faster. For example, a 4 KB file was taking 5 min when it used to take ~12s. |
2021-03-23 10:45 AM | Removed changes to Redis cache and the cloudwatch logging that were rolled out on 3/18 as we were concerned these were adding to the slow processing times. |
2021-03-23 1:42 PM | Setup NewRelic tracing for the cron job commands. Then found max_statement_time was spinning up db connections for each claim query in each file which was unnecessary. This took up 94% of the processing time. We put a patch in place to not set the max_statement_time=500 for Trizetto/Gateway. |
2021-03-29 5:45 PM | Put in hot fix to bypass and speed up a failure on initial matches to an appointment and provider, which were taking 1 min per each claim that failed and the time to try to write to the audit_log which would timeout on large files. |
2021-03-30 5:48 PM | Profiled the claims processing against staging. Set up parallel processing for Trizetto/Gateway with 100 workers. Got MONyog logs for MariaDB configuration changes and did not identify configuration changes that would have impacted query times and timeouts. |
2021-03-31 8:15 AM | Emdeon and Waystar were still processing too slowly or not at all. |
2021-03-31 9:23 AM | Moved Emdeon/Waystar exceptions back to inbox and restarted the parsing jobs. |
2021-03-31 2:45 PM | Monitored Trizetto’s backlog progress, and moved new files from the SFTP folder and exceptions back in for processing. |
2021-03-31 9:48 PM | Parallelized the Waystar and Emdeon processes as well. |
2021-03-31 9:48 PM | Patched max_statement_timeout for Waystar and Emdeon jobs as well. |
2021-04-01 2:33 PM | Split large Trizetto 837 files for 2021-03-24 and 2021-03-26. |
2021-04-05 11:23 AM | Developed a script & bot to monitor the status of the claims processing for internal use. |
2021-04-05 4:15 PM | Profiled the claims processing against the prod db/replicas and identified 99.5% of the time was spent on DB connections to a ‘Warehouse’ DB that had been removed from Rackspace and the Celery workers Django settings had CONN_MAX_AGE = 0 which creates and disconnects to the DB for every transaction. Removed the connection to the removed DB. |
2021-04-10 10:01 AM | Set the Celery worker max timeout to 90Ks (25 hours) to process large files (e.g. 900 KB). |
2021-04-12 5:52 PM | Put in hot fix for 277 parsing logic issues (temporal order issue). |
2021-04-12 5:52 PM | Put in fix for the RCM EDI Received page to show received and processed status dates. |
2021-04-13 1:38 PM | Reviewed and processed 277 and ERA exceptions. |
2021-04-19 5:56 PM | Put in hot fix for parsing Emdeon’s mrf files (version of the 277s) to check if ERA was already processed and if not, process 277. |
2021-04-22 1:38 PM | Moved older mrfs for Emdeon out. Continued processing the backlog of Emdoen mrf 277 files through 2021-04-22. |
2021-04-22 2:22 PM | Created a separate queue for files that timeout in parsing to manually call the parser for specific files to ensure we process them starting with the oldest files first through 2021-04-23. |
2021-04-22 2:22 PM | Research into duplicate 277/ERA files. Identified dups were sent by Trizetto/Gateway. For Emdeon, the files have updated statuses for the claims. |
2021-04-23 10:23 AM | Split large Trizetto 837 files for 2021-04-22 date over 7K KB to re-send to Trizetto/Gateway. |
2021-04-24 9:23 AM | Received MemoryError exceptions starting on 2021-04-23 at 3:12 AM due to too many claims being submitted at once. Added extra memory for the cron job to resolve this issue. |
2021-04-28 1:23 PM | Fixed Trizetto/Gateway 837 that ended part way through a claim for 03-31-2021 and re-sent to Trizetto/Gateway. |
2021-04-29 5:18 PM | Split large Trizetto/Gateway 837 files for 2021-03-29 and 2021-03-31 dates over 7K KB to re-send to Trizetto/Gateway. |
2021-04-30 10:21 AM | Developed & ran a script to re-bill claims that were stuck “in process at clearinghouse” claim status with a last EDI date greater than a week. |
2021-05-04 8:42 AM | Developed a script to handle duplicate records, to process the initial record first, then wait for that record to complete before processing the other record. |
2021-05-04 10:23 AM | Adding change to handle IntegrityError that occurs when there are duplicate claims sent by the clearinghouses for ERAs. |
2021-05-04 11:00 AM | Continuing development work to split claim files (ERAs, 277s) by claim to distribute across 100 workers for processing. |
2021-05-04 3:42 PM | Breaking out logging for each clearinghouse claims processing for better monitoring and troubleshooting of claims processing issues. |