The psychology of Radar is broken

by

Xcode 4 has been a mixed blessing for quite a few Mac OS X & iOS developers and recently Daniel Pasco posted a good blurb asking people to stop whining and file bug reports already and why.  It’s a good post.  I’ve actually thought about Xcode 4 and the challenge of producing an IDE/Dev tools in the current climate and I need to write a post on that.  I’ve worked on an IDE way back when, and also on at least one very large project – so I know some of the challenges of both.  However, this post is going to be about Radar because if Radar worked better we outside developers could support Apple’s Xcode & Developer Tools team better and that would be good for all of us.

Radar has a lot of problems and both Daniel and now Martin Pilkington in his recent post have filed bug reports and made suggestions to fix Radar.  I think they both make good points, but I don’t think their general “make radar searchable by 3rd party devs” is going to fly.  I’ve had this conversation with Apple people over the years and some have been “yah, I wish!  I’ve tried, they’ll never go for that…”  Further, I’ve certainly had clients for whom completely readable bug reports would have been a problem.

First, let’s take a quick look at what’s wrong with Radar as is:  to me the biggest issue is that there is a strong Negative Reinforcement dynamic with the current system.  Negative reinforcement is a strong behavioral modifier and is used to decrease the occurrences of a behavior (think punishing a misbehaving child or pet).  As such, it’s a serious problem that Radar provides continual negative reinforcement for filing bug reports.  Every “duplicate” email I get is a strong negative reinforcement for having wasted the time making a good bug report:  I always think and/or feel “All that work and it was already in their system!  DARN!  WHAT A WASTE!”  ok, that’s the PG version :).  I know, I know, it’s not a total waste because Apple uses the duplicate filing as some kind of primitive “voting system”, but that’s too sub-par to feel valuable.

I think both Daniel & Martin are identifying this same dynamic when they point to lack of searchability as the key problem to fix in Radar.

This “file a duplicate to vote up the bug” seems like a terrible idea to me and  here’s why:  not only does it create the negative reinforcement dynamic, it also creates a ton of unnecessary work for both Apple staff and outside developers.  On Apple’s side someone (many people I’d imagine) has to go through all the filed bug reports doing presumably manual searches to try and determine if a newly filed bug report is a duplicate or not.  (Hey Apple! We developers would be willing, no, we want to do those searches for you, for free! ) 

But the bulk of the extra work is shouldered by the outside developers who are writing the unnecessary duplicate bug reports.

Like any developer I hate receiving a badly written bug report.  As a result, it’s difficult for me to file a bug report that hasn’t been made reproducible, doesn’t have the simplest sample code necessary to demonstrate the bug, and lacks a good description of the circumstances under which it does and doesn’t happen.  (The only exception is a bug that I have a crash log for but can’t reproduce; these I’ll write up a bug report if I can add value to the crash report indicating what I was doing and what I saw.  But I’ll still spend time trying to make it reproducible).  For me this means that writing up a “proper” bug report takes between 1 and 4 hours (sometimes more).

Now it seems to me that Apple would really rather have their system full of well written bug reports and not clogged up with a bunch of badly written incomplete bug reports.  So beyond saving them staff time, they’ll also end up with higher quality bug reports (we’re more motivated, or less de-motivated more likely).

Now I’ve gone through phases of bug report writing.  Sometimes I write a lot, but generally I find it too depressing to get the “duplicate!” emails and so I don’t do very many.  I was wondering if I was exaggerating the strength of the negative reinforcement embodied by Radar as is, so I went back and looked through the bug reports I’ve filed with Radar in the last 10 years (apparently I didn’t use Radar before that or they were deleted…?).  Roughly speaking, 90% of the bug reports I filed were marked duplicates, 5% were marked as closed (good bugs, worth filing) and 5% were closed as ‘not going to fix’ or ‘as designed’ or something else I mostly disagreed with :).

This means that for every 20 bug reports I file in Radar I’m only filing 1 that is a “keeper” (not a duplicate and something that gets fixed).  Since each bug report takes 1-4 hours this means that for every “good” bug report I file I’ve spent somewhere between 20 and 80 hours to get that one good bug report.  Yikes!  Let’s call it 50 hours average.  Now most Mac/iOS developers (in the USA anyway) make something like $50+/hr salary (which costs their employer roughly $100/hr with taxes, benefits, etc) or $100+/hr contract rate.  This means that one good bug report effectively costs $5000 or more to file, on average.

Now I work primarily as a contractor for software publishers or hardware manufacturers and for them this is not a cost they want to absorb and here’s why:  even if the bug gets fixed it won’t get fixed immediately and that means the client also has to pay to have a work-around figured out and coded up no matter what (and this isn’t always trivial or even possible).  So they end up paying twice.  Ouch!

So we have a second strong negative reinforcement for filing bug reports – someone has to pay:  either the client pays $ and it impacts the schedule or it comes out of the developer’s personal time.  Bummer.  Bigger bummer when 95% of them don’t end up as new useful bug reports.  *Very* demotivating, to say the least.

Now some people are copying bug reports to open radar and that’s very generous of them, but honestly it’s hard not to feel like this is a completely unnecessary waste of time because of a system that already takes too much time.

Ok.  So, using Radar as it’s currently implemented is clearly a bummer (to me and many others if twitter and how few of my colleagues write bugs is any indication).  I’m not just writing to whine, so what’s the fix?  Daniel created four bug reports on radar itself which he posted on open radar for all to read and duplicate (thanks, but “ugh” – see above :)).  See his post for details.  Martin also created some bug reports as referenced in his article.

I totally agree we as developers need the ability to search Radar for existing bug reports.  However, I don’t think the proposal as written will fly and here’s why:  Look at all the leaks on MacRumors and such about stuff that’s covered under developer NDA.  Sad to say it, but there are dishonest developers among us (I wish they’d stop – the result of their blabbing can’t be anything other than Apple sharing less and less with developers which is definitely bad for us).  So my proposal for Radar goes like this:

  1. Make the summary of all 3rd party bug reports searchable by logged in radar users.
  2. Make the balance of the bug report default to not-readable/not-searchable but have a button that lets a developer mark a bug report as viewable by all logged in radar users.  This let’s the person authoring the bug indicate the level of secrecy they need.  They are the only ones who know, so this makes sense.
  3. Allow other registered developers to “vote up” a bug to indicate that a given bug is also a problem for them.
  4. Allow developers to add their email address (or their radar account) to an existing bug report to be automatically kept informed about changes to the bug (when it’s fixed, new info added, etc).
  5. Allow developers & DTS or whomever to add notes on work-arounds for bugs.
  6. When the bug is fixed have Apple engineering mark what version of the relevant OS or component the fix is in (this might involve a slight change of workflow at Apple).
  7. Allow other registered developers to attach additional crash reports, sample code, etc to any existing bug report to aid Apple engineering in finding and fixing the root problem.
  8. I also like Martin’s “ability to create drafts of bug reports” suggestion.  I’ve been bitten by Radar’s “session timeout” problem enough times (and ended up skipped complex bug reports as a result) that this would be welcome.

So, what’s the result?  Well, developers are motivated to file quality bug reports because they’ll know when they “have a live one” on their hands.  So Radar will have more high quality bug reports, fewer duplicates (nearly none), and there will be more information about each bug that IS in Radar.  All this will result in Apple being able to devote fewer staff to finding duplicates and being more efficient and effective and managing and fixing bugs. And that means less buggy software shipping to customers and developers.

I think everyone wins and I most developers I’ve spoken with agree that this is where the majority of the frustration comes from – if everyone wins, why hasn’t Apple fixed this yet?  I can only conclude that we are missing some information and/or there is some lawyer at Apple that’s blocking it.  I’ve suggested to Apple people that we get a bunch of long-time Mac/iOS developers together with the appropriate Apple people and have a working conversation about this.  Developers are, by and large, intelligent and capable problem solvers, why not take advantage of that and the power of a broad spectrum of experience and backgrounds to solve this problem already?!  I’ve never gotten an answer, but I’m intentionally not “high profile” so I guess that’s not entirely surprising.

My bug report referencing this blog post is: rdar://10995739  so feel free to duplicate it.  Here’s what it says:

Radar has some serious issues which I’ve outlined here:

https://geekanddad.wordpress.com/2012/03/06/the-psychology-of-radar-is-broken-13/
along with some constructive suggestions for improving it.

Thanks.

 

Advertisements

7 Responses to “The psychology of Radar is broken”

  1. thinbits Says:

    Number 6 will never fly. Apple NEVER comments on future releases.

    And just to keep things interesting, whenever a new OS release comes out and I’ve got open bugs in Radar, I get a slew of boilerplate messages from Apple suggesting I try to reproduce my bug in the latest release. The impression this leaves me with is “go see if we fixed something for you, because we’re not going to check”.

  2. Dad Says:

    Heard from one Apple guy that searching for duplicates is actually quite simple and doesn’t take as much effort as I imply above. good news!

  3. Dad Says:

    I realized that part of the problem is that Apple is silent on the issues developers have with radar and that makes it seem like they don’t care and aren’t thinking about it. Both of these are clearly false if my interactions with Apple developers is any indication.

    So, is there some way Apple & outside devs could have a constructive conversation about this? I’d like to think so and I hope it will happen.

  4. Sam Deane Says:

    I’ve been blogging about this stuff recently too, and I agree with you and Pillky

    http://www.bornsleepy.com/bornsleepy/radar-why-closed-bug-reporting-sucks
    http://www.bornsleepy.com/bornsleepy/radar-gtfo

    • Dad Says:

      I don’t agree with Pilky’s tone or choice of words, particularly, and I think the solutions he proposes are not do-able as written. I have tried to be more specific and think through things from Apple’s side of it, but I’m sure there’s more that we, being outside Apple, don’t understand. That’s why, in the end, we need Apple to open up a dialog on this so we can understand all aspects of the problem and help them solve it in a way that works better for us outside developers.

  5. Andy W Says:

    Duplicated, in my own words, as rdar://11018490.

    Backlash on this seems to be growing; it’s clear that Radar needs some love…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: