Agencies should get ready to get rated as there’s now a Yelp for marketing services.

The CMO Club, a group of 700 heads of marketing, has launched what it dubs a private “vendor rating program” for the purpose of allowing marketers to share recommendations on vendors across 18 product and service categories, which includes everything from creative and media agencies to mobile and analytics firms.

According to CMO Club founder and president Peter Krainik, he often witnessed chief marketers swapping recommendations informally at the club’s dinners and events. “This is us helping people behind closed doors,” Mr. Krainik said of the new program. “We’re connecting people, that’s what the club is all about.”

Pete Krainik, founder, CMO Club

Since the inception of the program at the start of 2013, between five and 10 CMOs per month are adding recommendations. Mr. Krainik says the listings are growing, with multiple reviews for some vendors, as well as reviews for some bigger agencies. The group last week began the program promoting the with a PR push.

Evan Greene, CMO for the Grammys, says he’s a “big fan” of the vendor-rating program and has both submitted reviews and used the service for recommendations. “It’s really helpful to see what people in a similar role are doing. … It’s an open exchange of ideas and recommendations,” Mr. Greene said of the program.

But will busy marketers — who are worrying about everything from product development to distribution channels to consumer awareness, and not just their agency relationships — have time to regularly log on to submit their recommendations? And moreover, will they actually be willing to spill the beans? After all, most marketers view the work done by their agencies as a competitive advantage.

The review process is not “laborious” Mr. Krainik says, though it certainly is thorough. Each entry has 15 questions, with a mix of drop-down menus and open-ended questions, in addition to basic information like reviewer name and vendor name. CMOs can rank vendors on quality of service, on-time delivery, price and customer service. Mr. Krainik says that reviews can’t be anonymous, and notes that he has personally approved each member of the club.

“I don’t feel any hesitation to share any recommendations of people who do good work,” Mr. Greene said. He noted that to date, he’s not observed CMOs using the platform to warn their peers about vendors to steer clear of.

Ad Age spoke to several executives at the 4A’s Transformation Conference in New Orleans — which gathers creative and media agency leaders for three days each year — about this new CMO Club initiative to get their take.

Jack Klues, the soon-to-retire CEO of Publicis Groupe’s Vivaki, one of the speakers at the conference, had this to say: “A confidential sharing of experiences within this group seems no different than that provided by Glass Door, which allows employees a place to anonymously share their experiences with current past employers. I welcome constructive, credible criticism. Rants and complaints without context or substantiation help no one.”

Most agency executives at the event told Ad Age they believed that radical transparency is where this industry is heading anyway, so this initiative is merely part and parcel of such a movement.

“Given the tenure of CMOs, they are under intense pressure to make immediate and impactful successes for brands and the overall business,” said Deutsch CEO Val DiFebo. “Will this help them? I’m curious about the topics CMOs will be rating, to determine if this type of service will be valuable or have utility … Will it be useful in determining how an agency’s chemistry works with a particular client? Will it monetize the success of ideas an agency generates for brands?”

“I think it is a good idea and if implemented the right way with proper guardrails it could be useful,” said Horizon CEO Bill Koenigsberg. However, he added, “I’m not sure the crossover in whom all the CMOs are rating is broad enough to have a large enough sample.”

Bob Liodice, head of the Association of National Advertisers, told Ad Age the idea of CMOs banding together in one place to share feedback on the companies they work with definitely has merit. What concerns him about program, however, is the “ratings” component, which can he fears can prompt overly subjective responses and unintentionally tar the reputations of very good agencies and other companies that work with marketers.

The ANA itself in the past tried a pilot program with what it called the Marketing Knowledge Center, which had a ratings mechanism about various topics of marketing information but didn’t permit the ratings of individual companies, nor was it shared broadly with the members. But it has since pulled the program because it didn’t yield what the ANA deemed reputable opinions.

Where nearly all of the folks we spoke to for this piece did take issue with the new initiative is in its branding. By using the term “vendor,” it casts an unpleasant feeling upon the value of agencies to clients, several people noted. It’s a term that doesn’t signal strategic advice that agencies are called upon, or the ways in which agencies help improve marketers’ bottom lines.

Still, if it keeps heading in the current direction, the program could turn out to be a good thing for shops. While some may bristle at the idea of CMOs sharing information in such a structured way, Mr. Krainik says, so far, most of the reviews are positive.

“Because CMOs are so busy, they’re writing about what they recommend,” and not using it as a forum to air grievances he said, noting 19 of 20 reviews are positive. He also added that the information listed in the vendor-rating program won’t be available publicly. “The plan is not to open this up for everyone to take a look at. This is not-a-for-sale thing.”

~~ Contributing: Alexandra Bruell, Rupal Parekh