The Guardian was the first news organization to really harness the power of crowdsourced data with its MP expenses database.

Now it has the lofty goal of distributing its technology worldwide to expose wrongdoing.

The man who has the job of opening up The Guardian‘s data technology told an audience at the Mesh technology conference in Toronto this morning: “The plan is to make the API (Application Programming Interface) open source.”

“We’re also exploring the possibility of making it a data clearinghouse,” in essence, allowing other people to upload their own data.

Chris Thorpe, The Guardian’s Developer Advocate for its Open Platform, set out the grand goal of  helping the oppressed and the impoverished worldwide. “We want to heal the world,” he said. “Openness is the friend of good.”

In June 2009, The Guardian hired a programmer for a week and built a portal to distribute more than 400,000 government documents of MPs’ expenses. The project prompted thousands of people to join The Guardian’s reporting team in pouring through claims and highlighting ones for further investigation. It was a huge step forward in newsgathering.

The Guardian API aims “to shine light on corruption,” Thorpe said. ”Let’s be a world liberal voice.”

The audience pressed Thorpe to state The Guardian’s stance on the current trend toward news paywalls. To that, he said, ”The people putting up paywalls will really reduce their influence in the world.”

“We’re seeing good advertising revenue from being open,” he added, but refused to talk numbers.

He said The Guardian is starting to realize “We have to behave more like a newspaper” by increasing audience engagement with more in-depth content.

“What we’re finding is the more pages people view, the more people are likely to click on adverts,” he said.

The Halifax Chronicle Herald signalled today it’s going to beef up its commenting system to improve the level of discourse on its site. Good to hear. Much of what passes for comment on news websites continues to be the lowest possible level of name-calling and uninformed bravado.

The Herald’s director of news content Dan Leger states:

“So we’re working on ways to end abuse, partly through registration and software to encourage use of real names and identify abuse. It won’t be perfect, but perhaps we can temper the over-the-top attacks.”

The issue of weak commenting systems was highlighted nationally last Wednesday when Halifax weekly The Coast said it would surrender the IP addresses of commenters following a Nova Scotia Supreme Court order. It’s the first time a news site has done that in Canada. Two senior fire officials at the Halifax Regional Municipality had sought to unmask the identities of these commenters, alleging they defamed them by painting their behaviour as racist and incompetent.

The move resulted in many people online and on local radio imploring news organizations to adopt a policy requiring real names for commenters as a means of raising the quality of online conversation.

They wanted to hear from someone like Howard Owens, who argues a real names policy is practical and enforceable at his New York State news site The Batavian.

I check public databases for names that match in the zip code provided. If no match, the user is asked to provide either by fax, e-mail or in person a copy of a picture ID.

Owens, himself, calls his policy “a ‘best effort’ practice,” siding with the news executives Richard Pérez-Peña cited in his New York Times article last week who “say that merely making the demand for a name and an e-mail address would weed out much of the most offensive commentary.” He may be right.

But while this may work at smaller news organizations, it is likely to be unworkable at sites that get tens or hundreds of new registrations a day. Pérez-Peña says as much in the Times story, which details the growing unease of news organizations with anonymous comments.

Real names strikes me as an offline solution to an online problem.

I argued in an interview on CBC Radio’s Mainstreet Nova Scotia last week that the big problem with the comments areas on most news stories is they seem to be abandoned by the news site itself. The result is the so-called “broken windows” problem — it becomes a wasteland vacated by moderates who have been long-since shouted down by blow-hards.

Author/blogger Scott Rosenberg states: ”Show me a newspaper website without a comments host or moderation plan and I’ll show you a nasty flamepit that no unenforceable ‘use your real name’ policy can save.”

I’m much more intrigued by Gawker’s approach. It implemented a more robust software system last year that gave its staff and community a way to promote the “funniest, thoughtful, intelligent, well-argued” comments. It divided its community into two tiers:

• a small community of “starred” commenters “who have proven themselves to be engaged, intelligent, humorous, fair-minded, thoughtful, rational.” These people can promote “well-written, thought out, intelligent and/or otherwise notable comment” below stories.
• The rest, whose comments will be obscured behind a “Show all comments” link

A key aspect is that the discussion is guided by an engaged community with an interest in creating a place people want to be. In addition, the community is fluid, with new commenters rising to become stars, and people who abuse their star power falling to become relative nobodies.

Gawker Media CTO Tom Plunkett suggested in this graph last week, that the site recovered — and thrived —  from an initial drop in comments that resulted from stripping people’s status based on the number of followers they had — and making many comments more difficult to view. He concludes: “purging commenter accounts is not a solution for the out-of-control commenter community. Nor is a large moderation staff.”

Will this completely stop defamatory comments? Probably not. Editor moderation is likely the only way to do that. But it’s a bottom-up — not top-down— prescription that seems a better fit with the medium.

It live blogged 8 events this past summer during Nova Scotia’s provincial election. Seven were political events — party platform releases or debates — the other was a discussion of the Herald’s use of social media. The Herald used CoverItLive, the leading (free) live-blogging service based in Toronto.

Rick Conrad, editor of thechronicleherald.ca, calls it a “vital” means of engaging readers and its effect is evident in the quality of conversation: “You get a more defined audience – almost a more educated one – because it’s something they are interested in.” He acknowledges the events attracted the occasional “crank” but “99% of the people were there for the right reasons and wanted to have a real discussion.”

Some tips from the Herald’s experience:

• Promote, promote, promote: People need advance notice to schedule a live blog into their day. Conrad promoted them on the site, in print, and on Twitter. The result was some success in expanding the site’s audience: “For the social media blog, I got the distinct sense it attracted people who wouldn’t normally think of picking us up every day or coming to us online.”
• It takes two: If you’re covering an event, you generally need at least two people – one to report and the other to engage the audience. Says Conrad: “With the platform releases, [the audience] would come with a lot interesting — and tough — questions like “What does this mean?” When you’re listening to what the leader is saying and trying to go through the documents, it’s difficult to make a snap analysis.” Conrad suggests bringing a reporter along, if you can. He concedes it’s difficult because a reporter frequently has to file a story as well. But he adds, “Readers really like to talk to a person directly involved in the story.”
• Let the audience lead: When to drive the conversation and when not to? It depends on the topic, says Conrad. If the audience is getting information from other sources it often works best to let them lead the conversation. “With the election night blog, we essentially took the approach that ‘this is their night’ and we would pop in with the occasional comment on the results … Whereas with the platform releases, I found there was more interaction with me because I was their eyes and ears.”
• … but keep it civil: Conrad says he approved the vast majority of comments, but rejected a few: “For me, the reason for doing the live blogs for the election was to give as many people as possible the chance to comment.” However, a few were “nasty” and wildly speculative. “We didn’t think that was appropriate. You want [the conversation] to be substantial and you don’t want it to spiral down.”
• Use it to assess your own efforts: People are especially engaged when they feel they can make a difference. The Herald received almost two comments for every person who attended the session in which it asked how it could make better use of social media. While Conrad acknowledges the difficulty in drawing conclusions from raw numbers, he says the tone of the conversation was spirited. “For me, the people who were involved in that blog were really into it.”
• Keep it available for replay: The number of people who replayed the Herald’s live blog the day following election night was almost twice the number who participated live. Some people don’t have time to follow a live stream — or aren’t available when it’s scheduled. But many are interested in seeing which topics resonated with the audience.

Conrad says he would like to live blog a provincial budget release and he hopes to experiment with the tool at a concert or a live arts event.

Google Fast Flip interface for mobile.

So this is it.

In April, Sharon Waxman reported a conversation with Google CEO Eric Schmidt in which he told her:

“In about six months, the company will roll out a system that will bring high-quality news content to users without them actively looking for it” … one in which users “will be automatically served the kind of news that interests them” … and which Google “will be able to sell premium ads against premium content.”

Google Fast Flip appears to be all that.

The company claims the visual news finding tool “personalizes the experience for you, by taking cues from selections you make to show you more content from sources, topics and journalists that you seem to like.”

Make no mistake — this is an application built from the ground up for mobile. (In addition to being available in Google Labs on the web, Google offers it as a mobile site for iPhone and Android devices.)

Reading stories from multiple sources isn’t easy to do on a mobile device right now. Neither is search. Done right, personalized, visual news could truly be a killer app.

As Scott Karp says, it’s “a new UI (user interface) for news.” Further, it’s Google re-asserting its claim to the browser, not the app, as the viewing space for mobile content.

Martin Nisenholtz, senior vice president of digital operations at the New York Times told Ken Doctor, “We’re trying to find out how users relate to a visual interface.” There’s certainly much to learn — and there are quirks with Fast Flip. But, the interface, with a firm nod to Apple’s Cover-Flow, is an excellent fit for mobile devices.

That’s great for consumers. And, as Nisenholtz points out, it’s the brand presence news publishers have wanted. But it also gives Google the upper hand in yet another news viewing space.

Colour-coded lines show traffic congestion in Google Maps for Mobile.

“It’s stop and go on the bridge.” “Watch out for that deer on the 102.”

The voices are a staple of local radio — listeners calling in to report traffic conditions. On-air updates build audiences, involve listeners and bring advertisers to morning and afternoon shows.

But Google believes it can deliver traffic reports faster and with more accuracy by harnessing the power of mobile users.

Google announced today it was bringing the power of crowd-sourced data to its Traffic layer in Google Maps, currently available only in the U.S.

Users of Google Maps for Mobile have to allow Google to see their location first. But when they do, they contribute traffic data anonymously and effortlessly as they travel. Their GPS-enabled mobile phone contributes their location and aggregates it with others to form a live picture of traffic in the area. Its announcement states:

When we combine your speed with the speed of other phones on the road, across thousands of phones moving around a city at any given time, we can get a pretty good picture of live traffic conditions. We continuously combine this data and send it back to you for free in the Google Maps traffic layers.

Google also announced it was expanding its Traffic layer to cover all U.S. highways and arterial roads.

Google has the leading mobile map app. While it’s competing with standalone GPS providers such as TomTom and Garmin in providing driving assistance, the division between smartphones and standalone devices is already starting to fall. TomTom released its turn-by-turn maps as an iPhone app 10 days ago — the first company to do so.

Google’s move is yet another grab at services traditionally provided by news media — services that bring revenues to support local newsrooms. While newspapers have had their audiences slashed by aggregators of news and classified listings, radio has remained relatively unscathed by advances in online technology.

The move puts radio producers on notice that they too need to find other ways of strengthening listener interaction.

Source: Google News

Give Associated Press President Tom Curley credit for grabbing people’s attention.

A New York Times story today quotes him stating that online references to AP stories containing as little as a headline and a link require a licensing agreement.

The interview follows AP’s announcement Thursday that it will begin a two-pronged approach to tracking sites that re-use AP content. The first initiative will be to include new metadata in news stories to identify key attributes of the story such as the author, the placeline and usage rights attached to it. The second will be to maintain a registry of stories and use software to track the content across the Internet. The goal is “to be paid for any use.”

Calling the registry a “declaration of war,” Erik Sherman calls the move “so stupid, so clearly self-damaging that you have to wonder whether someone inside the corporation is trying to torpedo it.”

Salon co-founder Scott Rosenberg chimes in, calling the effort “foolish and self-defeating.”

Most observers have focused on Curley’s comment concerning licensing for short chunks of text. But, in the Times article, Curley was coy as to how AP would go about enforcing its claims, adding, “We’re not picking the legal remedy today … Let’s define the scope of the problem.”

The prospect of suing people en masse for refusing to send money is indeed crazy. Curley’s claim would seem to ignore provisions of fair use in the American copyright act, not to mention the nature of the Internet. His intimation of enforcement would suggest he hasn’t learned much from the American recording industry’s efforts.

A story by the Nieman Lab’s Zachary M. Seward states that the New York Times’ top lawyer himself doubts aggregation is a copyright issue. Even if it is, as Google deftly pointed out to testy European publishers last week these companies can easily remove themselves from most aggregators’ reaches. Rosenberg argues in an excellent post that the real danger is the issue will end up in court where a judicial ruling could narrow the definition of American fair use provisions in its copyright act.

Too bad AP botched its PR on this.

There is actually much to admire in AP’s efforts. It’s the first major news organization to make a serious effort at tracking the use of its stories on the Internet. AP says it is using copyright-protection software made by Attributor Inc., which offers a Google Analytics-type interface for “finding copies of your content in near real-time.”

The company states, interestingly, that 30-40% of the excerpts it tracks fail to contain a referring link. That in itself indicates value in tracking egregious copyright violations and raising public awareness.

Update: Comment from AP on all the coverage

The phrase “mobile usability” is pretty much an oxymoron. It’s neither easy nor pleasant to use the Web on mobile devices. Observing user suffering during our sessions reminded us of the very first usability studies we did with traditional websites in 1994. It was that bad.

Sure, Nielsen is a curmudgeon on this stuff. But his analysis is generally bang on. The worst aspect, he states, is half of the problems with the devices are structural and are unlikely to be substantially improved.  The primary issues include:

• Small screens. They force users to remember more because less information is displayed at any one time. They also make interactions difficult because the work area is smaller.
• Awkward input. Doing anything – entering text, clicking a link or scrolling – is slow and prone to error.
• Download delays. Wireless service is still slower than a landline.
• Poorly designed sites. Most sites are designed for the desktop, not mobile. In fact, the study showed that the success rate for people who used sites that were designed specifically for mobile devices was 21% higher than that for the same sites desktop users see.

One of the key findings was that mobile users spent 38% more time on tasks such as finding the weather and TV listings than they did in 2000, when users had fewer choices. Why?

Today’s mobile users are highly search-dominant. When we don’t specify which site they should use (and often even when we do), they turn first to their favorite search engine. Again, this means plenty of typing, which is slow, awkward, and error-prone on mobile devices.

What’s needed, says Nielsen (not surprisingly), are more sites designed specifically for mobile devices.

The study is unclear as to whether smartphone applications provide substantial benefit over browser-based sites tailored for mobile. That’s the argument Google Engineering Vice-President Vic Gundotra highlighted on Friday with his comment, “the browser, for economic reasons almost, will become the platform that matters.”

Still, on that note, Nielsen offers a telling anecdote:

In our current study, one user did really well — an iPhone user who had a weather application installed on the phone and used it to get the weather forecast in only 18 seconds (1/3 of the fastest speed from 2000).

Photo: Extra Ketchup/Flickr

Interest in the report authored by U.K. teen Matthew Robson says a lot about how much we fret over how young people consume media.

London-based market research firm Morgan Stanley published the report, authored by their 15-year-old intern, last week. As the Globe and Mail reports today, the paper, “How Teenagers Consume Media,” has since become a minor sensation. Much has been made about the excess of anecdote and lack of research in the report, but it is clear Matthew is a remarkably articulate 15-year-old with an unusual ability to synthesize the activities of his peers.

Why the interest? My guess is a general fear that young people understand something about the Internet that we oldsters don’t.

Matthew’s comment about Twitter use among his peers is the most quoted observation. He states that while most teens are heavy Facebook users, “Teenagers do not use Twitter … They realise that no one is viewing their profile, so their ‘tweets” are pointless.” The comment follows barely a month after Nielsen’s announcement that Twitter was the fastest-growing Web brand this spring.

Huh? How do we reconcile that?

Well, 16-year-old Daniel Brusilovsky makes a fine argument as to why it’s important we not read too much into Matthew’s paper.

Twitter is a different type of social network than Facebook. Facebook is about connecting people, and sharing information with each other … With Twitter, it’s the exact opposite. Anyone can follow your status updates. It’s a completely open network that makes teenagers feel “unsafe” about posting their content there.

Teens are all about socializing with close friends — an activity well suited to closed networks such Facebook. Twitter is about spreading information far and wide to people you don’t even know.

As Suw Charman-Anderson points out, the most interesting thing is how little seems to be understood about this age group. Every generation struggles to understand young people, of course. But there is a fair bit of research about them (see John Palfrey’s Born Digital: Understanding the First Generation of Digital Natives). Charman-Anderson cites danah boyd’s research and suggests that there’s still a strong disconnect between academics and analysts — not to mention the general public — on the issue.

The fact that they haven’t ever had a clear insight into the teen demographic would seem to imply that their existing researchers and analysts aren’t doing their jobs properly. The information is out there, a lot of it is freely available, and all that remains is for someone to read it and write the report.

It would seem there is much work for journalists here too.

It brought rigour to HTML, encouraging cleaner markup, broader standards and generally a loftier vision for structuring content on the web. The journalism students who cared about such things saw the logic and clarity it brought to the medium through its tight integration with XML, the “super language” of the Internet.

XHTML 2 promised even more. As the World Wide Web Consortium states, the markup standard would have offered a clean and robust foundation for creating content on future devices. Specifically, it would have banished presentation elements entirely to CSS, and offered greater accessibility and internationalization.

Alas, the W3C ceased work on XHTML 2 last week. In fact, it was all but dead months ago.

Developers had grown weary of the W3C’s focus on standards at the expense of new features. Most had thrown their support behind HTML 5 — an extension of HTML 4.01, which was last updated in 1999.

Bruce Lawson, an HTML 5 advocate, told CNET: “XHTML 2 was a beautiful specification of philosophical purity that had absolutely no resemblance to the real world.” Developers — the folks who actually build the web — wanted more tools for interactive content.

A major flaw was that XHTML 2 wasn’t backwards compatible. XHTML 1 code would have to rewritten it to be valid in the new standard. Browsers themselves would have to be substantially expanded.

As Scott Gilbertson points out, every major browser update this year included at least some elements of HTML 5, but none touched XHTML 2.0:

Where HTML 5 has loads of new stuff for developers to use — native audio and video embeds, multi-column layout tools, offline data storage, native vector graphics — XHTML 2.0 didn’t offer anything of the sort.

Ultimately it seems the standards folks couldn’t see the trees for the forest. Their vision was simply too grand.

Oh, I know there’s an XML serialization of HTML 5. But there’s something a little sad about the death of such a well-considered plan for the future of the web.

Seriously … there’s some quite good, basic reporting advice in the new YouTube Reporters’ Center. My favourites of the bunch are the Associated Press’s: How to Pitch a News Story — 2:11 of jam-packed practical advice — and NPR’s Scott Simon breezily delivering the fundamentals of audio presentation in How to Tell a Story.

Still, it’s full evidence of the tortured state of the news industry. Six days ago Dow Jones Chief Executive Les Hinton called Google a “digital vampire.” Two months ago Associated Press chairman Dean Singleton vowed the company would pursue “legal and legislative remedies” against those he said were unfairly “misappropriating” its content. And he bluntly suggested Google was the main enabler. Now, here are some of the best minds of mainstream journalism using a Google product to advise amateurs on how to do it for free.

Google also today invited news publishers to host their videos on YouTube and participate in an advertising revenue share program via its YouTube Partner Program. CEO Eric Schmidt has promised to release more tools to help the ailing news industry and this move appears to build on comments he made in a speech to the Newspaper Association of America on April 7, 2009:

The fundamental issue is that the Internet distribution model doesn’t work on scarcity — it works on ubiquity. What we have to do is find models that involve very broad distribution and that you (news organizations) make money all along the way. We of course are in the advertising business. And we think that money will be there. We think there is a way to do it with even more targeting and even more immersive kinds of advertising models.

So Google continues to dangle a bigger and bigger carrot in front of news publishers. Take the leap, it says, free your content — the real money is in targeted advertising.

I believe Google is right about that. But it might be years before we find out.