![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
(no subject)
Feb. 11th, 2005 11:38 pmThere are ways to get concessions from a QA staff. There are also ways to get the QA folk to absolutely and without question put on their "No Fucking Way" hat. This is a story of the second order.
I have a release which is going out Tuesday night, and, for which, until some point this afternoon, the web.config files were coming out of our build process broken. Not broken beyond my capability to fix, but requiring intervention before they were functional. We've also been chronically short-staffed, and were missing one of our number the final three days of the week and another two, so i've had a lot of disjunct scrambling about.
All this builds up to 4:40 today, when i'm summoned (as the senior tester on site) to an emergency discussion about fixing a broken production issue. Yes, 4:40 on a Friday afternoon.
Apparently, something in the last release (on the 3rd) caused something to go slightly wonky in one of the marketing tools, and they didn't report it to us until after four. The developer who owned that area talked with the database admin who wrote most of the stored procedures, and together, they weren't positive about what broke what, but reasoned that since it seemed to work before, they could just roll back the code, recompile the relevant libraries, quick-fix a pair of stored procedures, we could test it quickly and shove it out.
Doesn't sound like a huge deal, or does it? The sprocs- definitely no problem. We handled a fix made necessary by user error the day before, and all in all it took about fifteen minutes for everyone involved. So, i ask, what's the relevant library? Aye, there's the rub. It's the main framework library in which the change are going to be made, and out comes the hat.
We make changes all the time to that library as the site is extended, but the potential liability of breaking something by pulling something out of it carries the potential risk of taking the entire site (and with it the revenue stream it represents) down. Not to mention the time. We have an agreement with the guy who does our deployments- we tell him on the day of a deployment no later than 5:30 that we have something he'll need to post, and then when we're ready, he does it. But by this time, it's five minutes to five. In order to release, we'd need to compile the library, deploy it to a test environment, verify that it fixed what was broken- which wasn't guaranteed, since it hadn't gotten pinned down definitively to a particular group of lines in a specific code-behind, and verify that making that change didn't break anything else. It would have required at least a full sanity pass for all viewer types on the entire site- and, since it was Friday, we'd need to have a definitive go/no-go by 5:30.
So i put on the "No Fucking Way" hat. If we'd known about this, say at noon, we'd have had some wiggle time to verify that the particular code-behind was the real cause, and not just a likely suspect; and still been able to smoke-test at least the immediately visible revenue channels if not the majority of functionality. At twenty to five, with no one going to be directly effected by the break until the next business day, while already scrambling around to whip my own release into shape and trying to figure out why our integration hadn't gone as smoothly as it should have, much less why we weren't having any success in getting overlay builds so that our six active projects would be able to coexist in our three test environments, i put my foot down. The potential risk was just too great for the return.
It's nice to occasionally 'win' a battle at work. I wish we were in a position where we didn't have to fight them; but if i'm going to be making wishes like that, i'd just wish for a 1:1 test:dev ratio and a workflow cycle that made logical sense rahter than being something Kafkaesque.
OK, no more work talk for the next two days.![[livejournal.com profile]](https://www.dreamwidth.org/img/external/lj-userinfo.gif)
damashita finally revealed her true Discordian nature by talking about eating a hot dog without the bun- and on a Friday no less. Fnord, and Hail Eris and all that good stuff. I've also got a date tomorrow night which i've been looking forward to since at least Wednesday, so i should have topics of conversation, at very least.
I have a release which is going out Tuesday night, and, for which, until some point this afternoon, the web.config files were coming out of our build process broken. Not broken beyond my capability to fix, but requiring intervention before they were functional. We've also been chronically short-staffed, and were missing one of our number the final three days of the week and another two, so i've had a lot of disjunct scrambling about.
All this builds up to 4:40 today, when i'm summoned (as the senior tester on site) to an emergency discussion about fixing a broken production issue. Yes, 4:40 on a Friday afternoon.
Apparently, something in the last release (on the 3rd) caused something to go slightly wonky in one of the marketing tools, and they didn't report it to us until after four. The developer who owned that area talked with the database admin who wrote most of the stored procedures, and together, they weren't positive about what broke what, but reasoned that since it seemed to work before, they could just roll back the code, recompile the relevant libraries, quick-fix a pair of stored procedures, we could test it quickly and shove it out.
Doesn't sound like a huge deal, or does it? The sprocs- definitely no problem. We handled a fix made necessary by user error the day before, and all in all it took about fifteen minutes for everyone involved. So, i ask, what's the relevant library? Aye, there's the rub. It's the main framework library in which the change are going to be made, and out comes the hat.
We make changes all the time to that library as the site is extended, but the potential liability of breaking something by pulling something out of it carries the potential risk of taking the entire site (and with it the revenue stream it represents) down. Not to mention the time. We have an agreement with the guy who does our deployments- we tell him on the day of a deployment no later than 5:30 that we have something he'll need to post, and then when we're ready, he does it. But by this time, it's five minutes to five. In order to release, we'd need to compile the library, deploy it to a test environment, verify that it fixed what was broken- which wasn't guaranteed, since it hadn't gotten pinned down definitively to a particular group of lines in a specific code-behind, and verify that making that change didn't break anything else. It would have required at least a full sanity pass for all viewer types on the entire site- and, since it was Friday, we'd need to have a definitive go/no-go by 5:30.
So i put on the "No Fucking Way" hat. If we'd known about this, say at noon, we'd have had some wiggle time to verify that the particular code-behind was the real cause, and not just a likely suspect; and still been able to smoke-test at least the immediately visible revenue channels if not the majority of functionality. At twenty to five, with no one going to be directly effected by the break until the next business day, while already scrambling around to whip my own release into shape and trying to figure out why our integration hadn't gone as smoothly as it should have, much less why we weren't having any success in getting overlay builds so that our six active projects would be able to coexist in our three test environments, i put my foot down. The potential risk was just too great for the return.
It's nice to occasionally 'win' a battle at work. I wish we were in a position where we didn't have to fight them; but if i'm going to be making wishes like that, i'd just wish for a 1:1 test:dev ratio and a workflow cycle that made logical sense rahter than being something Kafkaesque.
OK, no more work talk for the next two days.
damashita finally revealed her true Discordian nature by talking about eating a hot dog without the bun- and on a Friday no less. Fnord, and Hail Eris and all that good stuff. I've also got a date tomorrow night which i've been looking forward to since at least Wednesday, so i should have topics of conversation, at very least.
