CommunityMetrics

From Community Leadership Summit Wiki

Workshop on Metrics that Matter

Organizer:

  • Mark

Attending:

  • Pat Patterson, Sun Microsystems

(Lots of others!)

Groundrules:

  • Keep it positive: granted that "every current metric is wrong and meaningless," but still we need numbers
  • Looking for "Vital Stats": signs of health

Framing:

  • Understand there are different types of communities
  • "What do these types look like in real life?"
  • What are the things we can measure that indicate health (or lack thereof), and what do they really mean?

Types of communities

Proposal:

  • Customer support: get answers, get things fixed
  • Customer suggestion: get ideas into the suggestion box
  • Community of purpose: specific goal, agreed action is the purpose
  • Community of passion: activity or interest is the reason for meeting

Onno: a major obstacle is that the purpose of the comunity is often not agreed upon

Lesson Learned #1: very important to define goals, nature, purpose up front Lesson Learned #2: let your metrics and goals evolve: pick the right noes for now, do later for later

Simon:

  • Don't measure anything if you possibly can avoid it (always get gamed)
  • Beware of stupid metrics, and stupid metrics requesters

Dave Nielsen: get stakeholders involved in choosing metrics, so they learn which ones are stupid, what the others mean or don't.

Knut: Design your metrics to stop stupid suggestions, faulty assumptions, misunderstandings.

  • Measure number of contributors so funding doesn't get yanked
  • Measure tactical success: if you have a program to get "it" into schools, count school installations

Don: marketers often very familiar with junk-mail statistics, which are elaborate, which makes anything we can do in community metrics look weak

Karsten: Since one goal/claim of community is "better ideas come from the community," measure success of product ideas that were forced from inside, vs. success of product ideas that came from outside. Provide such metrics, close to mission, in advance, to deflect requests for meaningless metrics.

Mark: some war stories

  • Number of posts per login (to a TV media community)
    • You'd expect to want high posts-per-login
      • Saw the ratio tightly tied to whether show is in-season or out-season
      • Discussion in off season was often off topic: build community, but had nothing to do with show

Karsten: Need to spend a lot of time mining the data before showing to stakeholders

Mark: pick any data source you like, but stick with it: no two sources of data will ever agree on anything; what's important is trends, patterns, not values or equivalence

Onno: Be careful not to measure merely what's easy: refuse to measure something without supporting information that tells what it means.

Patti: Distinguish what you measure from what you report: measure, understand, explore, correlate, confirm, track, describe, then present.

Mark: Seed for future discussion: consider churches: how do you compare health of churches? Much less numbers than spirituality, but how do you measure that?

Simon:

  • Never gather a metric that can be driven by the organization.
    • Simple numbers (number of users, downloads, whatevers) are toxic
    • First/second derivatives are safer
    • Compounds, ratios, comparisons are safer

Patti: example of sharing metrics back to the community. SalesForce.com had an appearance of unreliability, established TrustForce.com, where they published up-time stats to the community, greatly improved community perception.

Ilan: Another kind of community: marketing appendix

Simon: Management might perceive a community as "people who might buy support," hence large numbers are good. But driving large non-contributing membership dilutes, disenfranchises, discourages actual developers.

Mark: anecdote: a community whose goal was to inspire people about something. Metric: scan discussion for keywords that sound newly inspired (not specifically on topic). Treat as trend to watch.

Ray: Key is to engage the metric-consumer in dialog over what to measure, help them find good metrics instead of hip-shot toxicities.

Ray: Measure many things, analyze, track them yourself, only show what's telling.

Perine: Co-design metrics with stakeholders; drag them away from concrete-but-meaningless, into graphics, trends, comparisons.

Karsten: Provide some feedback on this session in the google group

Perine: please pick one thing, track for three months, and tell us your experience (here in wiki or on the group)



-- jrep 23:03, 19 July 2009 (UTC)

Retrieved from "http://www.communityleadershipsummit.com/wiki/index.php/CommunityMetrics"