Well, we just killed off that myth! Good mission critical systems come from good mission critical systems people – not political hacks or pseudo-experts.
The ACA website cost more than the Costa Concordia
With today’s Congressional testimony of Kathleen Sebelius, the Secretary of Health and Human Services, we are finally seeing the importance of good systems people and how they can impact the success – or failure – of a major real-time mission critical application. Many people take for granted that software engineering is no big thing and the less you really know about systems, the more you can gloss over major problems without understanding their impact or their resolution.
This would be my advice to her and to the Congressmen trying to talk like they know systems: Just as one course in first-aid doesn’t make you a brain surgeon, one course in technology doesn’t make you an expert in real-time systems.
It is clearly evident Secretary Sebelius is NOT a systems person. Some of her answers to hard questions were laughable at best, and pretty sad at worst. She should not be the one hung out to dry. Who is the top “systems person” who was in charge to oversee the rollout and implementation? Was there one, or was that role not even defined or deemed necessary?
It is also evident that many, if not all, the members of the House Energy and Commerce Committee don’t have a clue about rigorous design standards and mission critical applications. Congressman Waxman said, “The Affordable Care Act is working. I would urge my colleagues to stop hyperventilating.” He said the problems would be fixed.
Hyperventilating? Wake up Waxman, the system doesn’t work! The system has security issues and most of all, there has been no plan to system test the application from end-to-end. Please Mr. Waxman – Yield your time to someone who knows real-time systems cold and is not a “pseudo-expert”.
LOOKING INTO THE SPOTLIGHT
With all of this on the table, this is not a one or two-week fix. With the magnitude of this system and all the interworking it needs to have with other data bases, it is not a one or two month fix either.
Plus, and this is the HUGE indication that they are systems-clueless in Washington DC, the fact that they are not going to shut it down until it is fixed is the wrong direction to take to correct the defects and do no further damage. The government cannot hobble along with a system that is insecure and also inaccurate. There is a bakery listed as a enrollment site within the depth of the website. How did that get there? How did it already get 150 people trying to call in and enroll into the program?
Seriously, try to save face by turning off public access to it until you can guarantee accuracy as well as security. Yes, that could take months but it is much better to take it slow and set the course right, than continue to run “full speed ahead” in the wrong direction with a system that has so many security and accuracy holes in it, it is more like a sieve than a solid ship for healthcare solutions.
I see all of this having a great positive impact that is finally coming into the national spotlight: Real-time mission critical applications are complex and need to have close and extensive testing before being released for public use. You do not entrust this to amateurs. And, you do not gloss over real problems.
What is really scary is to hear people who work in IT commenting “it’s no big deal. We release bad software all the time”. If that is the case, you should not be working in IT. You have no focus on quality and no focus on a sense-of-urgency within the profession.
All this solidifies what I said in an earlier column:
“This was not a small systems project, nor one that did not have impact on every family in America. Evidently, Washington DC does not have the caliber of systems people in charge that it thinks it has. Probably a lot of people with big egos and small skill sets claiming to be “experts” in software engineering and mission critical applications. Well, if you’re an expert, I’m a god.”
CARLINI-ISM: To quote Harold Geneen, former Chairman of ITT and I hope Mr. Waxman reads this –
“Words are words,
Explanations are explanations,
Promises are promises, but
only Performance is Reality.”
Copyright 2013 – James Carlini
Watch this Video on A Better Way To Fight Harmful Free Radicals
Jeffrey Zients needs to be careful that he is not simply using his punch list method to avoid actually finding and solving the root cause problems in the ACA web page. This is what happens when inexpert management selects a Big Bang delivery without expert help and then further sabotages the effort with political interventions all along the way.
Instead of the Jeffrey Zients punch list recovery method, HHS should consider the following ten steps:
1. Use of ACA website by customers seeking healthcare insurance should be terminated.
2. Existing customer profile data, personal data, and decision data should be quarantined.
3. The ACA website requirements foundation and technical architecture should be reviewed, assessed, and audited by a team of experienced industry experts.
4. The management, engineering, and process practices employed on the project should be reviewed, assessed, and audited by a team of experienced industry experts.
5. The accumulated Technical Debt on the project should be reviewed, assessed, and audited by a team of experienced industry experts.
6. A professional team should be charged with assembling factual analytics associated with assurance metrics, compliance metrics, noncompliance metrics, product engineering metrics, project management metrics, and process metrics.
7. A full scale program review should be conducted to assess requirements, architecture, practices, and metrics. The review team should record its findings and consequences and provide recommendations and rationale for carrying the project forward.
8. A professional team should be charged with assessing Cyber Security vulnerabilities in accordance with the NIST Cyber Framework.
9. A professional team should be charged with assessing privacy and civil liberties vulnerabilities in accordance with the NIST Cyber Framework.
10. The completion date for these activities should be established as December 16, 2013.
Don –
Thank you for your comments. Unfortunately, your recommendation comes late and to the wrong entity. It is clear that who they hired had little – if any- mission critical applications experience.
My first question would be – where is the Test Plan? It doesn’t seem like they had any type of structured approach to test the software that they put together and still have not demonstrated any real systems approach to clearing all these problems.
A punchlist? That sounds more like something written up for a house, not a website. Where are the module testing? The sub-system testing? The integration testing and the Stress testing for the entire website?
The worst thing they could do is bring in more politically-clouted “experts” from the Washington DC area. What they need are some independents outside of the DC Metro area that have no political affiliations that could “cloud” their findings and their judgments.
Who on Sebelius’s staff was supposed to be in charge of this? Certainly not Sebelius as she clearly has no systems background. This ACA website will continue to limp along until someone realizes you need “real” experts and not the pseudo-experts they have at many nearby consulting firms and ones that have political connections.
Oh look another one today: — http://www.tradersmagazine.com/news/nyse-experiences-data-malfunction-as-twitters-listing-looms-111715-1.html
Good insights, lets hope folks get the point soon for these real mission critical apps that don’t have operational work around alternatives available.
See Friday’s story –> Nasdaq Error Shuts Options Market for Almost Entire Day
http://www.bloomberg.com/news/2013-11-01/nasdaq-shuts-options-market-for-almost-entire-day-on-malfunction.html
American –
Read these short articles on timing and universal clocks —
Cloud Transaction Synchronicity
http://cloudcomputing.sys-con.com/node/2136070
Cloud Computing and Precision Time Protocol (PTP)
http://cloudcomputing.sys-con.com/node/2164323
American,
I am familiar with HFT servers and have actually written several articles about them. The problem is – all the transactions are being shot out every several micro-seconds. NONE of the servers are taking timing from the same timing source (a Universal Clock). Until that happens, things will ALWAYS be mucked up.
AND – as to the “best and brightest”, when it comes to real-time mission critical networks, you have to go back to Bell Telephone Labs. We addressed “digital slippage” between digital central offices back around 1980. The solution was you HAD to have a universal timing source so everything in the network would stay synchronized. (the digital phone network takes its timing from the Atomic Clock)
Too many people think they are “experts” today and we see the results. Until the SEC mandates a universal timing source, trades will continually get messed up AND when it comes to 1000 point drops like what happened on the DOW in 2010, the SEC will be unable to dissect and analyze the problem because they do NOT have the sophistication – either in personnel AND equipment – to go into a trading sequence and put it all together again. If HFT servers are trading in MIcro-seconds, the SEC needs to have analyzers that can get down to the nano-second range.
Think of the old days where traders would make trades and runners would go “timestamp” the trade. If you had several timestamps on the floor and knew that they were off by 10-15 seconds between several, you could actually manipulate trades just by going to a different clock on the floor. They cleaned that up by having a universal timing clock where ALL timestamps are pulling time off of one Master clock. 911 Centers (where they record your call) do exactly the same thing. Everything – call tapes, dispatch calls, etc. are all time-stamped off of one clock.
Just explaining that concept and solution – is MORE than what the SEC knows. Experts? If they’re “experts”, I’m a god.
Incessive details indeed
Harbans –
incessive?? that isn’t in the dictionary. What did you mean?
The exact word is ‘incisive’ meaning insightful. Sorry for use of wrong word.
Your blog is really insightful.
The government doesn’t pay good help what it is worth. They’ll pay contractors plenty based on that eco-system. I wouldn’t think that the Federal government would learn anything from Mass. as is obvious. As humans, we are reactionary. They’ll learn something now that the scrutiny is there. The point of the retort was not to be partisan perse’ but to acknowledge that in my humble opinion, it was to be expected and all the hoopla over it will fade away just as it did in Mass. The technology not initially working is a common thing when the work is funneled across so many vendors and effectively sabotaged in many cases, amongst other things. I work in high frequency trading and if you’ve been attentive to the “glitches” that have been occurring with players in that space over the past 6 to 8 weeks, e.g., NASDAQ, Goldman, NYSE, DTCC, (Where’s Knight Capital now?) etc.., These Fiserv groups do have the best and brightest on board, blowing past real-time to the micro-second now for messaging & transactions yet look at the SEC’s latest mandate to the Exchanges, i.e., “Get your IT together!” – To your point, it’s not easy.
American-
Based on your comments, you would have thought the federal government would have learned from Mass.’s mistakes instead of repeating them.
This isn’t about partisanship or the great Rep vs. Dem. infighting. This article is about the cluelessness of people who take real-time systems and mission critical applications for granted that you can just get anyone to do the software and it will come out flawless. It isn’t that easy.
Romneycare’s launch wasn’t successful either
Posted by Tim Louis Macaluso on Wed, Oct 30, 2013 at 1:51 PM
After listening to conservatives’ hysteria over the Affordable Care Act for the last two weeks, it was fascinating to hear Massachusetts Governor Deval Patrick talk about his state’s health care program, or Romneycare. The program, a micro version of the Affordable Care Act, served as the ACA’s blueprint.
Though the program is wildly successful today, it launched with many of the same problems as the ACA, Patrick said on MSNBC’s Up with Chris Hayes. Massachusetts has 97 percent of its residents signed into the program. The website, which is much smaller than the federal government’s ACA website, experienced numerous problems that had to be fixed in the days and weeks after the launch.
And even though there was evidence that many people were visiting the site, most of the enrollment activity took place as the deadline date drew closer.
Patrick was asked why Massachusetts’ plan has been so successful. His response? Democrats and Republicans worked together to make the program operate correctly. Problems were identified and fixed quickly.
Much of the catastrophe that opponents of the ACA predicted hasn’t materialized in Massachusetts, but that shouldn’t surprise anyone.
http://www.rochestercitynewspaper.com/NewsBlog/archives/2013/10/30/romneycares-launch-wasnt-successful-either