![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
ravencallscrowsOn top of the earlier bitch du jour, here's another one. If you don't want to deal with me venting, just scroll right on past it.
We're coming up on review time here. That's a good thing and a bad thing. The good part is that with reviews come raises and performance bonuses. The bad thing is that management starts looking for ways to evaluate performance, and there aren't really many good metrics for doing so.
What they fall back on for testers is bug count. If you find more bugs than other people, you must be doing your job well, right? Ok, so there's more to it than that. They also evaluate by fix percentages, to have some way of attempting to evaluate to be sure that testers are entering good bugs. Again, that makes sense to some extent. Otherwise it's be possible to just enter duplicate bugs and have your bug numbers look good, but without having really done anything.
Unfortunately, the general rule of thumb with things is something like this:
If you think you've found a bug, try to look it up in the database.
Circulate mail to the PM and Dev owners of the particular issue to see if it's a known issue.
When in doubt, bug it. It's better to have duplicate bugs than unreported ones.
Beginning to see a problem with this? Here's one dichotomy- If you enter a bug which the PM or developer who owns the feature thinks is a duplicate, you'll usually get tagged with a dupe. This can happen even if one code change causes several different things to be fixed- the one the dev was fixing will get tagged fixed, the others are all dupes simply because he wasn't specifically fixing them. This can suck.
I have the lowest 'fixed' percentage on my team. It's about 15% lower than optimal. I also have nearly three times as many bugs logged as other testers on my team, but because the fixed percentage is low, my boss is "concerned" and passing this on.
OK, let's look at why my fixed percentage is lower. Yes, there are some duplicates. Eighteen, in fact, of 85 total. Of these, eleven are issues which were fixed by other code changes, but are classified as duplicates regardless following the reasoning above. Of the remaining seven, four are outright duplicates. Hey, it happens. Not everyone writes bugs in the same manner, and when digging through the over 140,000 bugs logged, it's possible to miss something. The other three are questionable duplicates- they're similar behaviours, but in different parts of applications, and not necessarily the sort of thing that would be a known correlation to someone who wasn't a tester on that specific feature.
There are a few other classes of non-fixed bugs. There are "By Design" bugs- things that aren't deemed to be real bugs. I have 16 of these. Of them, i'd argue fourteen are valid bugs, ranging from things which should be fixed, but the developers in charge are too fucking lazy to fix (there are some hideously bad grammatical errors which are bugged severity 4- minor- just because they look bad and are embarrassing, for example), and there are some which are the result of simple inconsistencies in the specifications for the feature in question. Regardless, this is another dichotomy- "When in doubt, bug it" vs. "No, that's the behaviour that's expected, you fucking moron." Most of my "By Design" resolutions fall into the latter classification- spec inconsistencies, and most are from when i owned testing for the implementation of digital rights management in Word. It should be noted that during the time i owned testing for that, the feature went through three different project managers, is now on its fourth, and is still in flux (although it was a first milestone feature, it's still not settled now, after the end of milestone three). I still have eleven bugs open against the SPEC for this, because it's so unclear what the expected behaviours are in different places.
Then there are Wont Fix bugs. I've got nine of these. These differ from "By Design" in that they're acknowledged bugs in the product, but are deemed either too inconsequential to fix or too labour intensive to bother with. Counting these against a tester is sheer bullshit- "Hey, you found a problem, but we're too fucking lazy to fix it. We'll admit that it's a problem, though."- but they're still not resolved as fixed, which lowers that critical percentage.
There are two more classes of bugs. These are Postponed- "Hey this is a problem, but the feature which is affected by them is going to be cut for this release, so we'll fix it when the feature is included." (I have a few of these, nothing can really be done about them), and Won't Repro- "Bad tester, no donut"- which just indicates that the developer who investigated the problem couldn't reproduce it. These occasionally happen because things have been fixed by the time devs get around to looking at them (only one of these, and i still repro it on two of my test machines, but no one else seems to be able to).
So, what does this mean? It means that if things are judged strictly by the metrics, come review time, it's going to look as if i suck at doing my job. Never mind that i have more responsibilities than anyone else on my team as far as feature ownership goes, or that i own all the automation for all of the Development and Release and Deployment and Roaming teams on the International side; my numbers suck. If they try to ding me on this on my review, they're getting the sanitized (read: not profanity-laden) version of this rant. I so much want to get the fuck out of this department. I hate my fucking job.
We're coming up on review time here. That's a good thing and a bad thing. The good part is that with reviews come raises and performance bonuses. The bad thing is that management starts looking for ways to evaluate performance, and there aren't really many good metrics for doing so.
What they fall back on for testers is bug count. If you find more bugs than other people, you must be doing your job well, right? Ok, so there's more to it than that. They also evaluate by fix percentages, to have some way of attempting to evaluate to be sure that testers are entering good bugs. Again, that makes sense to some extent. Otherwise it's be possible to just enter duplicate bugs and have your bug numbers look good, but without having really done anything.
Unfortunately, the general rule of thumb with things is something like this:
If you think you've found a bug, try to look it up in the database.
Circulate mail to the PM and Dev owners of the particular issue to see if it's a known issue.
When in doubt, bug it. It's better to have duplicate bugs than unreported ones.
Beginning to see a problem with this? Here's one dichotomy- If you enter a bug which the PM or developer who owns the feature thinks is a duplicate, you'll usually get tagged with a dupe. This can happen even if one code change causes several different things to be fixed- the one the dev was fixing will get tagged fixed, the others are all dupes simply because he wasn't specifically fixing them. This can suck.
I have the lowest 'fixed' percentage on my team. It's about 15% lower than optimal. I also have nearly three times as many bugs logged as other testers on my team, but because the fixed percentage is low, my boss is "concerned" and passing this on.
OK, let's look at why my fixed percentage is lower. Yes, there are some duplicates. Eighteen, in fact, of 85 total. Of these, eleven are issues which were fixed by other code changes, but are classified as duplicates regardless following the reasoning above. Of the remaining seven, four are outright duplicates. Hey, it happens. Not everyone writes bugs in the same manner, and when digging through the over 140,000 bugs logged, it's possible to miss something. The other three are questionable duplicates- they're similar behaviours, but in different parts of applications, and not necessarily the sort of thing that would be a known correlation to someone who wasn't a tester on that specific feature.
There are a few other classes of non-fixed bugs. There are "By Design" bugs- things that aren't deemed to be real bugs. I have 16 of these. Of them, i'd argue fourteen are valid bugs, ranging from things which should be fixed, but the developers in charge are too fucking lazy to fix (there are some hideously bad grammatical errors which are bugged severity 4- minor- just because they look bad and are embarrassing, for example), and there are some which are the result of simple inconsistencies in the specifications for the feature in question. Regardless, this is another dichotomy- "When in doubt, bug it" vs. "No, that's the behaviour that's expected, you fucking moron." Most of my "By Design" resolutions fall into the latter classification- spec inconsistencies, and most are from when i owned testing for the implementation of digital rights management in Word. It should be noted that during the time i owned testing for that, the feature went through three different project managers, is now on its fourth, and is still in flux (although it was a first milestone feature, it's still not settled now, after the end of milestone three). I still have eleven bugs open against the SPEC for this, because it's so unclear what the expected behaviours are in different places.
Then there are Wont Fix bugs. I've got nine of these. These differ from "By Design" in that they're acknowledged bugs in the product, but are deemed either too inconsequential to fix or too labour intensive to bother with. Counting these against a tester is sheer bullshit- "Hey, you found a problem, but we're too fucking lazy to fix it. We'll admit that it's a problem, though."- but they're still not resolved as fixed, which lowers that critical percentage.
There are two more classes of bugs. These are Postponed- "Hey this is a problem, but the feature which is affected by them is going to be cut for this release, so we'll fix it when the feature is included." (I have a few of these, nothing can really be done about them), and Won't Repro- "Bad tester, no donut"- which just indicates that the developer who investigated the problem couldn't reproduce it. These occasionally happen because things have been fixed by the time devs get around to looking at them (only one of these, and i still repro it on two of my test machines, but no one else seems to be able to).
So, what does this mean? It means that if things are judged strictly by the metrics, come review time, it's going to look as if i suck at doing my job. Never mind that i have more responsibilities than anyone else on my team as far as feature ownership goes, or that i own all the automation for all of the Development and Release and Deployment and Roaming teams on the International side; my numbers suck. If they try to ding me on this on my review, they're getting the sanitized (read: not profanity-laden) version of this rant. I so much want to get the fuck out of this department. I hate my fucking job.
no subject
Date: 2002-05-30 12:53 pm (UTC)I hear you. Course, I'm on the other end of these. I spent the last year as the developer of the most trafficked part of our product. My usual course of action is to try not to ever close bugs as "By Design," "Won't Fix" or "Not Repro."
- By design ones always get assigned to PM as they are usually spec flaws. THe PM can then incorporate a spec change into the next release and close the bug as "Fixed."
- Occasionally, I'll close something as "Won't Fix" but by and large I'll just postpone these. If there is a real bug, it's gonna get fixed eventually.
- I only close as "Not Repro" if it is clearing something that can't be reproed. I'll spend days working with a tester to get a good repro case for something they saw once but we can't get the conditions again. I figure the testers aren't ever lieing. On occasion conditions change enough that it can't be repro'd, or there is a case where the tester was slacking some. But that's rare. I did have one where our entire back end wasn't working due to a configuration error that the tester didn't know about. He went through a test case that should have reported an error. But it reported that the back end was down. So he filed a bug that said "The wrong error message is displayed in case X." Keeping up with his email on our servers would have saved that bug from getting filed. I closed it "Not Repro."
But in general I try to get my fix percent as high as possible. I hate the animosity between dev and test.no subject
Date: 2002-05-30 01:30 pm (UTC)I hate the animosity between dev and test.
Amen to that.
From what i've seen, the larger the number of people working on a project, the worse it is, too.
Animosity between dev and test when i was at Cavedog was usually whichever side won the last game of pick-up basketball- dev and test were really good at communicating what was going on to each other, and i know that there were times when i was the lead that the dev lead and i'd spend as long as it took with an individual tester to track down what was going on. Course, there i was also doing all the builds, and out lead dev was great, so it wasn't uncommon for us to check in three lines of code, make a special debug release and watch exactly what was happening, only to repeat whenever necessary.
I've been testing long enough that i don't try working with my head up my ass too often. Usually if i'm taking the time to write a bug, it's a good bug- meaning that it's repro'ed at least twice, that it's been run to ground so it's isolated if it's a localization issue (i work in Int'l Office- can't recall if that was mentioned before) or not, and more often than not, i've got a pretty fair idea of exactly what the problem is. (hell, in the Cavedog days, my lead dev and i'd play a game- he'd make checkins, tell me what the functionality change was, and i'd tell him what parts of the source code he changed, and ran about an 80% accuracy rate, which isn't too damn bad for someone who only can read code).
There are just too damn many people involved in Office, and between the chaos of not having stable, concrete spec (and it's been like this since day 1 of the project- lots of stuff on the PM/design side contingent on other people's decisions, and now over 12 months into development and still no hard decisions on some stuff which common sense would say has to be pretty foundational), and just hating the stuff i work on, working as a friggin' barrista at Starbucks is beginning to sound more and more attractive every day.
Today was just so NOT a day i needed to get dumped on by anyone. Fortunately, venting about stuff is usually pretty theraputic.