DentalXchange experiencing errors and performance issues

Incident Report for DentalXChange Product

Postmortem

Eligibility and Claims Service Disruption

Summary

On September 1, 2026, clients using DentalXChange's eligibility and claims APIs were unable to reliably submit claims or check benefit eligibility. Client-facing errors began at approximately 1:58 AM PT and continued until 12:36 PM PT. Affected clients received connection errors, timeouts, or responses their software could not read. Claims submission was the worst affected path: at the peak of the disruption, roughly seven in ten claim submission attempts failed.

No claims data was lost. Claims that failed submission during the disruption were able to be resubmitted successfully once service stabilized. No protected health information was exposed, accessed, or improperly disclosed. This was not a security event, and no unauthorized access to any system occurred.

What happened

The cause was a single payer connected to our eligibility service that returned responses which did not conform to the standard eligibility response format. Where a normal response is a few dozen lines, invalid data expanded the affected responses to roughly 5,000 lines each. Our eligibility path attempted to parse each of these responses in full, which exhausted the response parser on the shared servers handling eligibility and claims traffic and left those servers unable to process requests of any kind.

Because those servers are shared across our eligibility and claims infrastructure, clients with no connection to the affected payer were affected as well. Two conditions extended the disruption. First, our application monitoring recorded the elevated error rate at 2:12 AM PT but did not page on-call staff; the incident was opened at 5:36 AM PT, and a client integration partner reported the problem to us eight minutes later. Second, because the symptoms presented as a general infrastructure failure, the team spent several hours restarting servers, which relieved symptoms temporarily while the underlying condition persisted. Once the affected payer's eligibility traffic was disabled, service stabilized for all other clients within minutes.

Root cause

The servers that handle real-time eligibility and claims requests are shared infrastructure. A single farm of eight application servers processes requests across many payers and many clients. When a client requests eligibility for a payer, our system calls that payer directly and processes the response it returns. One payer returned responses containing invalid data that did not conform to the standard eligibility response format. Instead of a response of a few dozen lines, invalid data expanded the affected responses to roughly 5,000 lines each. Our eligibility path attempted to parse each response in full, which exhausted the response parser on the servers handling the traffic. Those servers then stopped responding to any request, including requests that had nothing to do with the affected payer. Host-level resources were not the constraint: processor use stayed between 25 and 40 percent and memory use between 428 MB and 1.4 GB across all eight servers throughout the disruption.

Two conditions extended the disruption well beyond the time needed to fix it. Detection worked; notification did not. Our application monitoring recorded the error-rate breach at 2:12 AM PT. That alert was not routed to on-call staff, so no one was paged. The incident was opened at 5:36 AM PT, and a client integration partner reported the problem to us at 5:44 AM PT. Roughly three and a half hours of client impact passed with a correct signal recorded, and nobody notified. Separately, a monitoring sensor had been generating a high volume of false alerts overnight, and the team was working that queue when the real signal arrived in it.

Diagnosis was slow because the symptoms pointed away from the cause. Eight servers failing at once, recovering on restart, and failing again is the signature of an infrastructure or capacity problem rather than a single payer's data defect. The team measured and eliminated server resource exhaustion, database lock contention, network packet loss, and a first-of-month demand surge before turning to payer-specific data. Inbound request volume during the failure window was in fact 20 percent below the same window on the previous first of the month. Identification came by narrowing to a specific payer connection and disabling it as a test, after which the farm stabilized. The specific non-conforming response was captured and analyzed afterward, which confirmed invalid data segments expanding each response to roughly 5,000 lines.

We are addressing this at three levels. First, the affected payer's eligibility connection is disabled and will stay disabled until the data defect is corrected. Clients requesting eligibility for that payer receive an explicit payer-unavailable message. This is a containment measure and not a permanent fix. Second, we are building a response-validation and size guard in the eligibility path so that a malformed or oversized response from any payer is rejected safely without consuming shared server resources or affecting other clients' traffic. This is the permanent fix, and it is scoped for delivery in Q4-2026. Third, we have corrected the monitoring and alert-routing failure that let a detected condition go unnoticed for three and a half hours and added instrumentation on the application servers so that a resource-exhaustion condition can be confirmed or ruled out during an incident rather than after it.

How we restored service

  1. Measured and eliminated server resource exhaustion, database lock contention, network path degradation, and elevated inbound demand as explanations, using monitoring and network data rather than inference.
  2. Narrowed the source to a single payer's eligibility connection and disabled that connection as a test at 12:36 PM PT.
  3. Confirmed that service stabilized for all other eligibility and claims traffic once that connection was disabled, and restarted the affected application services to clear the residual condition.
  4. Configured an explicit payer-unavailable response for any eligibility request directed to the affected payer, so that it cannot be mistaken for an eligibility result.
  5. Captured the non-conforming response and shared the technical detail with the payer with a request for correction.
  6. Verified data integrity: no claims data was lost or corrupted, and claims that failed during the disruption were able to be resubmitted successfully.

What we are doing to prevent recurrence

  • Disable and hold the affected payer's eligibility connection until the underlying data defect is corrected. - Completed
  • Build a response-validation and size guard in the eligibility path so that a malformed or oversized response from any payer is rejected without consuming shared server resources or affecting other clients' traffic. - ETA Q4 2026
  • Work with the affected payer to correct the response defect; pursue an alternative resolution if the payer does not correct it. - In progress, ETA Sept 2026
  • Add memory and transaction-time instrumentation and alerting on the production application servers supporting eligibility and claims. - Completed
  • Review and right-size application server capacity limits against current traffic volumes. - In progress, ETA Sept 2026
  • Verify that application error-rate and response-time alerts are routed to on-call paging, and test end to end that a firing alert reaches a person. - Completed
  • Remediate the monitoring sensor generating high-volume false alerts, so that a real signal is not lost in an alert queue. - In progress, ETA Sept 2026
  • Add a diagnostic step to the incident runbook: when a restart relieves symptoms and the condition recurs, treat the recurrence as evidence of a persistent underlying condition and change investigative approach. - In progress, ETA 2026

Questions

For questions about this incident or to request a technical briefing for your team, please contact your DentalXChange account representative or reach our support team through the usual channels.

Posted Sep 09, 2026 - 15:40 PDT

Resolved

This incident has been resolved.
Posted Sep 01, 2026 - 19:40 PDT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Sep 01, 2026 - 13:33 PDT

Update

We are continuing to investigate this issue.
Posted Sep 01, 2026 - 12:03 PDT

Investigating

We are currently investigating this issue.
Posted Sep 01, 2026 - 11:06 PDT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Sep 01, 2026 - 06:35 PDT

Investigating

We are informing all clients that we are currently experiencing issues with Claims Services, Eligibility, and other services. Our team is engaged and addressing the issue. Please note this is not a cybersecurity Incident.
Posted Sep 01, 2026 - 05:52 PDT
This incident affected: ClaimConnect (ClaimConnect /DCI, ClaimConnect /CRED, ClaimConnect PreLive) and Eligibility AI.