By Dan Wayland, Used with permission from Ellumen, Inc.
The Agile movement hasn’t just changed Federal government software procurement and development – it has undoubtedly whet the most influential Federal Government DoD and VA leaders’ appetites for more Silicon Valley innovation. But what else is out there – what is the next big thing to emerge? Beyond Agile, another technique called dark launch has been used in the VA’s DoD FHIR service for sharing healthcare information between the agencies’ enormous healthcare systems. It is just one of the ways that Ellumen is bringing Silicon Valley’s proven and successful testing methods to Federal Health IT customers.
Why We Need Dark Launch
The overall trajectory for quality assurance and testing of VA/DoD health record sharing initiatives has been positive for years. However, one of the most significant challenges has been that despite being tested thoroughly, issues frequently manage to remain hidden until projects release a software delivery into production. These latent issues happen because production load is typically much heavier, and production data is typically more varied, and more exotic than the de-identified data loaded into test environments. The problems that arise when the system/application is exposed to the stress of real production load and data translate to delays and difficulty for clinicians who provide and document care to our service members, Veterans, and their beneficiaries. Once issues occur they are typically detected and corrected after a new version of software is deployed to production. However, users have already experienced delays, errors, and possibly even down time.
The good news is that this challenge is not unique to the VA, DoD, health IT, or the federal government. Many complex web applications such as Facebook, eBay, Google Maps, and others have extensive databases with a huge variety of data and are under incredibly heavy loads. These large scale commercial web applications frequently update without any glitches. How do these commercial web applications address this same issue? How does Facebook, for example, verify that an update will work under those difficult conditions?
The answer is dark launch.
How It Works
Dark launch is a testing method that allows a system update or version to be exposed to only a subset of users. For example, perhaps some QA users and willing members of the real user community who volunteer to be beta testers and provide feedback. The rest of the users see no change, but these users see the new version under the stress of production load dealing with the variety of production data. The benefit of dark launch is that bugs and areas of the infrastructure that need attention are exposed before deployment. Dark launching saves engineers from having to firefight issues post-deployment and allows new features and updates to be activated immediately for millions of users without the adverse side effects. When Facebook deployed its “Facebook Chat” feature in 2008 and the “username” feature in 2009, both times they relied heavily on dark launches to seamlessly make the updates without users noticing any errors, delays, or downtime.
Would Dark Launch Actually Work in the Federal Space?
Ellumen Inc. has already performed a dark launch on an interagency DoD/VA system handling clinical data exchange. Users in various locations throughout the nation were able to obtain a much more accurate assessment of how the software will behave for all production users than could ever have been determined in any existing test environment. It worked extremely well. The dark launch brought to light issues that otherwise would have left clinician users scratching their heads and frustrated because of errors. The adoption of dark launch for VA and DoD health information sharing initiatives is one example of how government agencies are successfully adopting Silicon Valley patterns for success to improve healthcare for Veterans and members of the armed services.
About the Author: As the Vice President of IT Solutions for Ellumen, Dan Wayland has 17 years of experience in the fields of health IT, system architecture and design, legacy system analysis, and expert project management. Mr. Wayland has served as a technical lead and scrum master on multiple major development, upgrade, and sustainment efforts for VA, DoD, and IHS projects and commercial hospital medical systems integrations. His expansive knowledge of healthcare’s most relevant emerging technologies is honed from years of insights gained while guiding DoD and VA healthcare organizations on the delivery of smarter, better solutions.
