Watchdog 30...

I use the Garden Watchdog feature here a lot, especially right now while I'm frantically ordering seeds to quell my cabin fever. Today I noticed something disheartening about the Watchdog 30 though-
4 of the companies have ratings below 76 and 6 companies have ratings in the mid-low 80's. I understand to be in the "30" the company needs a volume of ratings and it is an average of all them. But why are we rewarding comanies who 1 out of 4 times make people mad/unimpressed? I got punished for C's in school, and B's weren't much better. Isn't that really their grade with 75%?

I want to be able to click on any company in the Watchdog 30 and find a reputable company that will deliver every time. That list should not contain hit or misses. Sorry for the rant, but does anyone else agree?

I agree, the top 30 ought to be trustworthy companies. One thing to do though is read the negative comments that are there--I've seen some negative feedback left for some companies where it was very clear that the person who left the comment was the one being unreasonable and the company had done everything they should have done. I'm not sure if "false negative" comments like that are factored in at all when the top 30 are chosen.

My suspicion is that companies have to have a certain number of ratings total in order to crack the top 30, and there are tons and tons of companies in the database that probably don't have the required number, so the pool of eligible companies is probably relatively small, and that's probably why you end up with some in the 75-80% positive range that still make the list. Honestly though if I see a company that has a ton of ratings and 80% are positive, I still feel pretty comfortable placing an order.

My experience with GW in general is that the ratings are biased negative. People complain (and rate negatively) way too easily and demand too much. So that does make the high positives really impressive to me. I had nothing but good experiences from a company that has a 47 rating!! Had I read that first (and before I got a better sense of GW), I never would have tried them.

I also think a lot of times people are more likely to take the time to complain about a bad experience rather than praise a good one. The "regulars" here probably do a good job of rating all their purchases good or bad, but there are a lot of people who sign up just to use the Watchdog once or twice, and they are probably more likely to have come with a complaint. Same thing happens with the 800 number at the company I work for--there are always a few people who call to say how much they like a product, but the phone rings off the hook if there's anything wrong with it!

True.

You can click on any company's score to see how it was calculated. I think that will probably answer a lot of your questions, and you'll see that the none of the Top 30 companies have a 1-in-4 record with their customers ;o)

I completely agree with that--that people remember the negative experiences and will more than likely comment on those rather than positive experiences. And a lot of the time neutrals and negatives are like, come on people have some grace for human error/nature and let down your expectations for perishable goods just a bit. But I feel like the number of negatives it takes to receive a 73 are more than just some people being touchy.

I didn't mean that any of them have 1 in 4 rating, I said that 1 in 4 was a bad experience which is what 75/100 represents to me. Which I guess means the 3 in 4 were good at the same time.

I've never clicked on the score to see how any of them were figured, I just assumed that 100/150 positives gave a 67/100 score, for example. Shows what I know. I think it's a good idea that the newest positives and negatives count more than older ones. But to me it seems like good, small customer service-centered companies should benefit more from our system than big, unreliable companies. How are the small, fantastic companies supposed to make it when we only reward volume?

You have to say, there's something about the 100 percent companies that is amazing. They must take the time to make every order perfect. They must communicate, offer fair prices and make even the bad orders good...right? There's no way they've bribed their customers to only have good things to say. I feel like the top 30 that we as Dave Gardeners tout to be the best should make A's; not necessarily 100's, but anything over 90 is good to me. I wouldn't mind having a secondary list of honorable mentions -- the B's and C's. I'm just suggesting a new way to list them so that we can be super sure of our company choices.

I'd almost rather find a small company with 90-100% than risk a 73% on the top 30. That said, I do take risks for cheap prices, knowing full well what the company rating is. I just want to talk about it, not cause a ruckus.

You've made some good points.

It is a truism that people are more likely to complain than praise about most things. However, for as long as the Watchdog (and its predecessor, the PBM-FAQ) have been around, the comments submitted have run roughly 70% positive, 10% neutral and 20% negative.

Because the score is weighted, it's important to remember that doesn't mean a company with a score of 75/100 has a ratio of 75/25 positive/negative comments. In fact, a company may receive a positive comments 9 times out of 10 and still have a score of 75/100.

This algorithm prevents companies from being rewarded strictly on volume. (In the early days, the calculation was a simple average, with positives given the same weight (but opposite) weight as negatives. We quickly realized that allowed a company with a lot of feedback to rise to the top due to sheer volume, and it didn't matter if half their feedback was negative.

Whether reviews are too harsh or not isn't something we can judge or censor. Each reviewer is responsible for their choice of words and rating, and each vendor is notified when feedback is left, and can react as they see fit to address it either publicly or privately.

If anyone has ideas for a different approach to scoring or an idea on how we can do anything else better, we'd love to hear your suggestions ;o)

Now I'm confused, Terry. You did verify my 'feeling' that there was a 'negative bias', though I attributed that only to the members tendencies, not to the algorithm.

I don't get how a 'simple average' method, where each positive carries the same weight as each negative, can skew the results based on volume if the split was 50 / 50. To me, simple average implies percentage alone. So 50% negative should result in the same rating whether it's 20 reviews or 500.

Victor, let's say you're a company with 500 reviews: 251 of them are positive, and 249 of them are negative.

With a simple average, your 500 reviews would push you into the top echelon, even though you have only a 51% rating with your customers.

Thanks. So why not give weight to the straight percentages a bit to emphasize recent reviews and possibly assign a designation reflecting volume of reviews. Some multi-tiered system? I know that sounds like it's too complicated and people may just want one number, but I think a company that gets 90% positive ratings, yet only gets a score of 75 is getting short-changed. Surely that must scare off a few potential customers.

It is easy to fix - if more people did what I did and give details to everyplace they order from - good or bad - this would not be a big issue. Getting more people to get out there and post the good and the bad we would have less issues. I also look to see people I know and what they said - if I see NNNNN from my favorite forum is on there and says no way then I would never order, if it is a bunch of people who I have no idea who they are then I dont take their word so quickly.

If I understand this correctly, we basically get to comment on a vendor once. That comment weighs less (regardless of what it was) as time goes on, and the fact that we love any given company enough to re-order and re-order with all positive interactions, we can add to our original comment, but it means nothing? It seems that to be pleased enough to keep going back to somebody, placing a new comment about a new transaction should count for something then. The goal is for people to come back because they are happy, right?

I was just peeking back to Buggy Crazy, one of my favorite vendors of all time. I see her new rating is 87%, based on one negative (if you can call it that) from May. It's tough to look at all those beautiful, positive , glowing comments and this one negative and believe she is at 87%.

I have had the utmost respect for what the Watchdog does. It's what brought me into DG. It's been interesting to see now how it is calculated. It seems like victor has some nice ideas on how things could be weighed differently.

This message was edited Jan 4, 2008 10:15 AM

chuckle...will the mathematicians (or statisticians, to be more precise) please raise your hands? We would welcome your input. (What sounds easy in theory becomes pretty tangled when we try to execute it in a formula.) It's also important that the formula is easy enough for the casual reader to "get" at a glance.

magnolialover, your point is well-taken that customers who are happy year-over-year are being discounted. Again, if anyone has suggestions on how to counteract that unintended effect, we're all ears ;o)

I am not a statistician and don't know the 'optimum' formula. But all I know is that a company that has 40 positives and one negative should be rated much higher than 87!! 40 out of 41 is 97.56%, so that's quite an effect from one person. Maybe less weight should be given to recent ratings (how much less? I don't know.)

One possibility is to expand the three options (neg, neut, pos) to a five star system or something like it, as many other shopping sites have.

I really appreciate you listening, Terry. I am no statistician and really appreciate all the work that has gone into the current formula, but I hope that someone will chime in here to help us out.

So I understand it to be correct in saying that a new interaction with an old vendor and positive comments will never weigh the same as a totally new comment? I know I am not giving out numbers, but what if you said comments under say, a six month time period would all count under the same review, maybe a year? Then any comment posted during x time period could be counted in full as a full evaluation?

I really like when you guys added the number of reviews under the person's screen name. Is it possible to show how often they rated negative, positive or neutral? I would give less credence to a person's negative rating if he or she had mostly negative reviews.

magnolialover, if you access any company's entry and then click on their score (it's a hyperlink), you'll see how the comments are "aged".

victorgardener, if you click on the member's name, you can see on their home page a link to their GWD reviews, and it will show you who they rated, and whether they gave them a negative, neutral or positive.

I guess I don't like that a negative comment rates 25, while a glowing positive rates a mere 5 in the same time period. I understand the formula has it's reasons, as I read above, it just doesn't seem right.

Thanks Terry. I meant a way to see it right there under their name, just like the number of reviews. I have clicked to see how people have rated, but I think most people would not go through that so I was wondering if it could be made easier.
We can do two things to make it somewhat better. Rate more often - especially when things are good, not just when they're bad. And also state how many times we ordered from them. My own self-imposed policy is not to rate unless I have ordered multiple times - I try for at least three. One time is just not representative to me.

victor, we've had requests to add subscriber (or non-subscriber) status, how long they've been a member, etc. to that area and to try to cram everything in would create a jumble. I think we'd all agree it's good to put a comment in context, but that may mean doing a little virtual "digging" (errr, clicking) to see how much weight you personally want to give someone's review.

I agree wholeheartedly with you: we all need to rate the vendors we deal with - especially the good experiences, which are often taken for granted. When you post a review, think about what information will be most helpful to someone reading it (the vendor or a would-be customer): how many times you've purchased, the specifics that made the transaction go smoothly, did the plants or seeds or whatever meet or exceed your expectations? Would you order from them again?

If your experience was negative and you weren't able to get it resolved by working with the vendor (that should always be the first step), then make sure your review is fair, objective (not inflammatory or a personal attack), and credible: when did you place your order, and describe what specifically went wrong with the transaction. If you were partly to blame, take responsibility for what you did or didn't do.

If the vendor comes back and makes things right, take the time to change your rating and add a footnote to explain why you're changing it.

All good points, Terry! I believe many people don't give the vendor the chance to correct the situation before posting - especially if there are other negative ratings.

There are no statistics that can be applied to the Watchdog ratings. Why? People come to post a rating of their own volition. It is not a poll by an independant interviewer who randomly selects subjects to post a rating. Then, the rater is given three choices (pos., neut, neg.) to identify their overall rating. Their choice is subjective, based on their particular opinion. Another rater might have the exact same experience but give a different rating (for example, positive vs neutral).

The case of buggycrazy is an excellent example of why the rating/weighting system is, well, meaningless in some circumstances. Here we have 41 results, one being negative. And this person wrote ONE sentence and didn't even attempt to contact the vendor! And for this buggycrazy loses 13%!

So, what to do?? I say get rid of the "Company Profile Score Bar" altogether and the so-called "statistic" weighting associated with it. There is no statistic or weighting that can be applied to the subjective input of raters. Unfortunately there are probably people who look at nothing but that rating and make their decision. Get rid of it and keep the "Recent Review" box, which is much more relevant.

Surely we as consumers must take the responsibility to analyze the information provided by our fellow gardeners. Is it not already intuitive that older ratings should have less weight than recent ones? I for one look at the Recent Reviews box before anything else because that tells me something about the number, type, and timeliness of reviews.

My 2 cents worth for now.

pardalinum (and others), thank you for your thoughtful input and suggestions today. You'll be happy (I think ;o) to note that we've acted upon your suggestion and removed the score bar. Hopefully that will cause readers to look beyond the numerical assessment and read the actual feedback as they form an opinion about any particular company.

Since January is the start of the annual mail order frenzy, I hope everyone will take the time to share your opinions with the Garden Watchdog companies as you place and receive your orders.

Some food for thought:

* There are over 35,000 reviews published, but only about half of the 6,100 companies have been rated at all - that means there are ~3,000 companies that might as well have a big question mark next to their name. My guess is some of them are not unknowns to some of you; won't you share what you know?

* For those companies that have been rated, remember the helpful comments you read today also need a steady stream of input (from all of us) to remain up-to-date and relevant for those who will discover the GWD in years to come.

Thank you Terry for listening to input and suggestions ;)

And thank you art-n-garden for starting this thread ;)

Thanks Terry and art.

Thanks Terry and Dave! I think it is a sensible improvement and much fairer to the vendors.

What a coincidence--we were just discussing this in the Annuals forum today: MaineMan brought up some good points in this post, responding to me being nervous about ordering from a poorly rated company: http://davesgarden.com/community/forums/p.php?pid=4360725

Looking at the two companies that I saw him mention there, another thing to keep in mind is that some companies sell plants in multiple forms (plants, seeds, bulbs, etc) and they may do a good job with some things but a bad job with others. With Burpee and Park I know they both sell plants as well as seeds, and while I don't know this for a fact, I would suspect that they have higher numbers of complaints on their plants than on their seeds. Whenever I'm thinking of ordering from a company that has multiple things like that, I always read the comments specifically looking for how many positive/negative comments there were relating to the type of thing I was purchasing. I've run into a few where I would have felt perfectly comfortable ordering seeds for example but wouldn't have touched their plants with a 10 foot pole.

Interesting--I just noticed that the ratings scales are gone from Garden Watchdog!

KyWoods, we explained the removal of the score here: http://davesgarden.com/community/forums/p.php?pid=4361082

I am glad to hear the rating bar has disappeared. I watched as my favorite company went down the scale. It was sad to watch as I figured it would hinder people from ordering from them and missing out on a good variety of seeds.
The company shall remain nameless, but I did notice that most of the negatives were complaints about things that could have been avoided had the person carefully read the directions on how to order.
So blaming a company because you carelessly disregarded important instructions, is really not the companies fault. LOL

Ingenious! Thanks, Terry!

I read art-n-garden's comments in the first post which intrigued me, so I went to GW to see what she was talking about. I didn't find any ratings, so I thought all of you were talking through your collective hats till I got down to Dave's comment about removing the rating bar...lol. I don't remember the thing, so can't say whether I'm happy it's gone or not. I try to comment as soon as I receive an order, and if the order isn't correct, then after it's been resolved one way or the other.

I don't expect companies to have perfect records, but I do expect good customer service when I buy something. Some of these vendors shine, while others skate by. I would echo Terry's comment hoping that everybody would post a comment when they order from any of the vendors listed on GW, and to add to your original comment if you have something further to say. GW is AWESOME!

I wasn't on at all yesterday and how great to come back to a thriving conversation! You all made wonderful points victor, Pardilinum, Maineman, ecrane, magnolialover, Judith and Terry--things I never would have thought of. Dg is such a democracy and I thank dave and Terry for acting fast and appropriately. This is why I love Dave's!

I will go through my stuff tonight and make sure I have done all the rating I can, positive or not. I agree that's the only way to keep the system honest. In the end I think the GW has invaluable information, even if we have to sift through some exaggeration and/or negative bias.

With this change, will the Watchdog 30 still be figured the same way, just not publically (for lack of a better word) ? And just a question of curiosity- but how often does the lineup in the 30 change?

You're very welcome, and we really do appreciate the feedback and input from our members. We were already mentally wrestling with the scoring, and the suggestions here helped us with our decision.

Yes, the Top 30 are still selected based on the same criteria (which was in place long before we made some tweaks to how each company's score was displayed to members. It's been tested over time, and seems the fairest way to represent the "best" based on all the reviews.

The Top 30 is real-time and dynamic, so when any company receives a new rating (or an existing rating is changed by the member), the system automatically moves them up or down the rank, which can move them into or out of the Top 30.

Thank you for explaining that. I think it's interesting that the top 30 can and do change, but how cool is the program Dave wrote for that! :)

I'm glad I went looking for this thread-I thought I was going nuts or you'd changed the code and firefox was blocking it when the bar disappeared.

I think removing the bar is an improvement, as we're all agreeing that it's not really that accurate, but I'm not sure it solves the problem of how a site gets rated and into the top 30. I know there are a couple of companies that pop in and out pretty regularly. If they're that volatile, are they really that stellar? I've found lots of companies that have higher ratings that, as far as I know, have never cracked the top 30. I'd still like to see a new algorithm that could fix some of these things so that the really, truly best of the best get the recognition they deserve. After all, whether the bar is still there or not, the scores are still calculated the same way, it's just less transparent.

And, another comment on Buggy Crazy-how many other companies here have as many repeat, glowing comments as she gets? It should really say something about a vendor when repeat customers go out of their way over multiple years to continue to add to their reviews-that's pretty extraordinary.

When I post a rating, I mostly post positive ones. I think those companies deserve the attention and kudos and sales over ones that didn't do me so well. However, I always very, very carefully read the negative ones others post, and I also find the company responses very interesting. I also look at the actual numbers of positive vs negative, and how recent they are. IF a new company had a bunch of negs two years ago, but have had all positives over the last year, then I'd be more likely to take a chance on someone with more negatives than a company that is now getting negatives after lots of positives.

Just because you see a company slip in and out of the top 30 doesn't mean they necessarily did anything wrong. Let's say that company #30 and company #31 are very close in score--then one day someone writes a positive review for #31. That could be enough to make their score a touch higher than the company that was at #30, so the new company pops onto the list and the old one gets bumped off (even though nothing changed in terms of their performance). Then the old #30 gets some more positive reviews, which push it back onto the list again. Both companies are still doing exactly the right things for their customers, and depending on how many people are writing reviews, they could bounce back and forth multiple times.

And the companies you know of that have high ratings but aren't in the top 30 probably don't have enough comments yet. There are probably some wonderful smaller companies that don't sell enough plants and get enough reviews to crack the top 30, but without having some volume of comments how do you know whether the company really has a good track record, or if you just haven't heard from a lot of their customers yet. Statistically speaking, you can feel more confident in a company with 90% positive comments if there are 1000 comments total vs if there are only 10, and the purpose of the top 30 is to list companies that you really can't go wrong ordering from so you want the statistics on your side.

Terry,

Thank you for eliminating the scoring for the companies and using just the actual numbers. I think the straight numbers are more fair to all the companies.

I had no idea that there was a "formula" involved in the ratings -- silly me -- I thought they were based on a percentage of negative vs. positive. To be honest, I read your explanation of how that formula works, but didn't completely understand it. However, what I did understand of it didn't seem fair or logical. So, since the Top 30 will still be determined by the "old" system, I'll just ignore it in the future.

Susybell.

I agree with your summation in the last three sentences. That is exactly how we should be using the WD ratings. Of course, if a company has 58 negs and 11 positives, all bets are off. LOL.


Karen