These are my notes from the session Building a Participation Platform: Yahoo! Web Services Past, Present, and Future by Jeffrey McManus

This was a talk about the Yahoo! Developer Network. Over the past year, Yahoo's efforts to harness the creativity of the developer community has lead to the creation of healthy developer ecosystem with tens of thousands of developers in it. They've built their ecosystem by providing web APIs, technical support for developers and diseminating information to the developer community via http://developer.yahoo.com. Over the past year they have released a wide variety of APIs for search, travel and mapping (AJAX, Flash and REST-based). They have also provided language specific support for JavaScript and PHP developers by offering custom libraries (JavaScript APIs for cross-browser AJAX, drag & drop, eventing and more) as well as output formats other than XML for their services (JSON and serialized PHP). They plan to provide specific support for other languages including Flash, VB.NET and C#.

The Yahoo! APIs are available for both commercial and non-commercial use. Jeffrey McManus then showed demos of various Yahoo! Maps applications from hobbyist developers and businesses.

Providing APIs to their services fits in with Yahoo!'s plan to enable users to Find, Use, Share and Expand all knowledge. Their APIs will form the basis of a 'participation platform' by allowing users to interact with Yahoo!'s services on their own terms. They then announced a number of new API offerings

• Browser-based authentication: This is a mechanism to allow mashups to authenticate a Yahoo! user then call APIs on the user's behalf without having the mashup author store the username and password. Whenever the mashup wants to authenticate the user, they redirect the user to a Yahoo! login page and once the user signs in they are redirected back to the mashup page with a token in the HTTP header that the mashup can use for authentication when making API calls. This is pretty much how Microsoft Passport works. I pointed this out to Jeffrey McManus but he disagreed, I assume this is because he didn't realize the technical details of Passort authentication. . The application is given permission to act on behalf of the user for two weeks at a time after which the user has to sign-in again. The user can also choose to withdraw permission from an application as well.

• Yahoo! Shopping API v2.0: This API will allow people to make narrow searches such as "Find X in size 9 men's shoes". Currently the API doesn't let you get as granular as Shoes->Men's Shoes->Size 9. There will also be an affiliate program for the Yahoo! Shopping API so people who drive purchases via the API can get money for it.

• My Web API: This is an API for the Yahoo!'s bookmarking service called MyWeb.

• Yahoo! Photos API: This will be a read/write API for the world's most popular photo sharing site.

• Yahoo! Calendar API: A read/write API for interacting with a user's calendar

Most of the announced APIs will be released shortly and will be dependent on the browser-based authentication mechanism. This means they will not be able to be called by applications that aren't Web-based.

In addition, they announced http://gallery.yahoo.com which aims to be a unified gallery to showcase applications built with Yahoo! APIs but focused at end users instead of developers.

.Jeffrey McManus then went on to note that APIs are important to Yahoo! and may explain why a lot of the startups they've bought recently such as del.icio.us, blo.gs, Flickr, Dialpad, Upcoming and Konfabulator all have APIs.

As usual, I'm impressed by Yahoo!

 

These are my notes from the session Musical Myware by Felix Miller

This was a presentation about Last.fm which is a social music site. The value proposition of Last.fm is that it uses people's attention data (their musical interests) to make their use of the product better.

The first two questions people tend to ask about the Last.fm model are

1. Why would people spy on themselves?
2. Why would they give up their attention data to a company?

Last.fm gets around of having to answer these questions by not explicitly asking users for their attention data (i.e. their musical interests). Instead, all the music they listen to on the site is recorded and used to build up a music profile for the user. Only songs the user listens to in their entirety are considered valid submissions so as not to count songs the user skips through as something they like. The service currently gets about 8 million submissions a day and got over 1 billion submissions last year. One problem with submissions is that a lot of their songs have bad metadata, he showed examples of several misspellings of Britney Spears which exist in their song catalog today. For this reason, they only use the metadata from 8 million out of their 25 million songs for their recommendation engine.

The recommendation engine encourages users to explore new artists they would like as well as find other users with similar tastes. The site also has social networking features but they were not discussed in detail since that was not the focus of the talk. However the social networking features do show users one of the benefits of building up a music profile (i.e. hence giving up their attention data) since they can find new people with similar tastes. Another feature of the site is that since they have popularity rankings of artists and individual songs, they can recommend songs by obscurity or popularity. Appealing to the music snob in users by recommending obscure songs to them has been a cool feature.

The site does allow people to delete their music profile and extract it as an XML file as well.

 

These are my notes from the session Who Is the Dick on My Site? by Dick Hardt

This was a talk about the services provided by Sxip Identity which identifies itself as an 'Identity 2.0' company. Dick Hardt started the talk by telling us his name 'Dick' and then showing us images of lots of other people named 'Dick' such as Dick Cheney, Dick Grayson, Dick Dastardly and a bunch of others. The question then is how to differentiate Dick Hardt from all the other Dicks out there on the Web.

In addition, Dick Hardt raised the point that people may have different personas they want to adopt online. He used women as a classic example of multiple persona syndrome given that they constantly change personalities as they change their clothes and hair. He used Madonna as a specific example of a woman who showed multiple personalities. I personally found this part of the presentation quite sexist and it colored my impression of the speaker for the rest of the talk.

So how does one tell a Web site who one is today? This usually involves the use of shared secrets such as username/password combinations but these are vulnerable to a wide variety of attacks such as phishing.

Besides telling sites who I am it would be nice to also have a way to also tell them about me so I can move from site to site and just by logging in they know my music tastes, favorite books, and so on. However this could lead to privacy issues reminiscent of scenes from Franz Kafka's The Trial or George Orwell's 1984. There should be a way to solve this problem without having to deal with the ensuing privacy or security issues. I can't help but note that at this point I felt like I had time warped into a sales pitch for Microsoft Hailstorm. The presentation seemed quite similar to Hailstorm presentations I saw back in 2001.

Dick Hardt then talked about various eras of identity technology on the Web

• Identity 1.0 - directory services and X.500
• Identity 1.5 - SAML and other technologies that enable business partners to assert identity information about individuals. They require trust between the identity provider and the relying party
• Identity 2.0 - user-centric identity models such as InfoCard

Sxip Identity has shipped v1.0 of their technology but has gotten feedback that its customers would like it to be a standard. They have now begun to investigate what it would mean to standardize their solution. One of their customers is Ning who used their technology to add identity management to their site in 12 minutes.

 

These are my notes from the Artificial, Artificial Intelligence: What It Is and What It Means for the Web by L. F. (Felipe) Cabrera, Ph.D.

This was a talk about Amazon's Mechanical Turk. The session began with the original story of the mechanical turk which was a hoax perpertrated in 1769 by Wolfgang von Kempelen. The original mechanical turk was a chess playing automaton that turned out to be a powered by a dimunitive human chess master as opposed to 'artificial intelligence'. In a sense it was artificial artificial intelligence.

There are lots of questions computers can answer for Amazon's customers such as "where is my order?" or "what would you recommend for me based on my music/book tastes?" but there are others it cannot. However there are other questions a computer can't answer well today such as "is this a picture of a chair or a table?". Amazon's Mechanical Turk provides a set of web APIs that enable developers harness human intelligence to answer questions that cannot be answered by computers. The service has grown moderately popular and now has about 23,000 people who answer questions asked via the API.

The service offers benefits to developers by giving them a chance to enhance their applications with knowledge computers can't provide, to businesses by offering them new ways to solve business problems and to end users who can make money by answering questions asked via the API.

Examples of businesses which have used the API include a translation service that uses the API to check the accuracy of translations, polling companies testing opinion polls and purchasers of search advertising testing which search keywords best match their ads.

 

These are my notes from the session The Future of Interfaces Is Multi-Touch by Jeff Han.

These was mainly a demo of touch screen computing with a screen that supported multiple points of contact. Touchscreens and applications today only support a single point of contact (i.e. the current location of the mouse pointer). There is a lot of potential that can be explored when a touchscreen that supports multiple points of contact at once is used. For example, truly collaborative computer usage between multiple people is possible.

Most of the content of the talk can be gleaned from the MultiTouch Interaction Research video on YouTube. Since I had already seen the video, I zoned out during most of the talk.

 

These are my notes from the Simple Bridge-building session by Ray Ozzie.

Ray started of by talking about how he sees RSS as having the potential to be the connective tissue between web sites. The increased usage of RSS and mashups are part of a trend of building 'composite applications' on the Web. Although Microsoft and other tools vendors have done a good job selling tools to enterprises for building composite applications, it has been RSS and mashups that have brought these trends to power users and hobbyist developers. Ray believes the next step is bringing the power of composite applications to end users. He explained that this idea isn't far fetched given that UNIX pipes are a manifestation of composite applications surfaced at the end user level.

The Web today is primarily a bunch of web sites which act as data silos. Although there has been some progress made on the data interchange front with XML and microformats, we don't have something as simple and powerful as the clipboard for the Web. So Ray talked to his Concept Development team at Microsoft and asked them to implement a clipboard for the Web that was cross-browser and secure. After a few weeks they came up with a JavaScript-based solution which worked in major browsers like Internet Explorer and Firefox. In addition, the solution was XML-based and harnessed microformats. They christened the technology Live Clipboard.

So how does it work? Basically when a user right-clicks on a special Live Clipboard icon on a website they see the regular right-click menu containing Cut, Copy, Paste, and Select All. However when the information is copied, it is copied to the clipboard as an XML document containing a rich payload which could be a microformat. This enables a bunch of useful scenarios such as cutting and pasting rich data between websites (e.g. events from Eventful to my calendar in Windows Live Mail) or between websites and desktop applications (e.g. events from Eventful to my calendar in Microsoft Outlook).

Ray then proceeded to give a series of demos of the Live Clipboard technology. The Concept Development team has created a screencast of a Live Clipboard demo , and a simple web page-based demo that you can test.

More information about Live Clipboard can be obtained from Ray Ozzie's blog post entitled Wiring the Web.

 

These are my notes on the Scaling Fast and Cheap - How We Built Flickr session by Cal Henderson.

This was an 8 hour tutorial session which I didn't attend. However I did get the summary of the slide deck in my swag bag. Below are summaries of the slide deck Cal presented at the tutorial.

Overview and Environments
Flickr is a photo sharing application that started off as a massively multiplayer online game called Game Never Ending (GNE). It has 2 million users and over 100 million photos. The site was acquired by Yahoo! in May 2005.

A key lesson they learned is that premature optimization is the root of all evil. Some general rules they've stuck to is

1. buy commodity hardware
2. use off-the-shelf software instead of building custom code

When trying to scale the site there were a number of factors that needed to be considered. When buying hardware these factors included availability, lead times, reliability of vendors, and shipping times. Other factors that affected purchase decisions included rack space, power usage, bandwidth, and available network ports in the data center.

Load balancing adds another decision point to the mix. One can purchase an expensive dedicated device such as a Cisco 'Director' or a Netscale device or go with a cheap software solution such as Zebra. However the software solution will still require hardware to run on. One can apply several load balancing strategies both at the layer 4 network level such as round robin, least connections and least load and at the layer 7 network layer by using URL hashes. Sites also have to investigate using GSLB, AkaDNS and LB Trees for dealing with load balancing at a large scale. Finally, there are non-Web related load balancing issues that need to be managed as well such as database or mail server load balancing.

The Flickr team made the folowing software choices

• PHP 4 (specifically PHP 4.3.10)
• Linux (2.4 kernel on x86_64 and 2.6 kernel on i386)
• MySQL 4/4.1 with InnoDB and Character sets
• Apache 2 with mod_php and prefork MPM

There were 3 rules in their software process

1. Use source control
2. Have a one step build process
3. Use a bug tracker

Everything goes into source control from code and documentation to configuration files and build tools.

For development platforms they chose an approach that supports rapid iteration but enforces some rigor. They suggest having a minimum of 3 platforms

• Development: Working copy of the site which is currently being worked on
• Staging: Almost live version of the site where changes to the live site are tested before deployment
• Production: The customer facing site

Release management consists of staging the application, testing the app on the staging site then deploying the application to the production servers after successful test passes.

Everything should be tracked using the bug tracker including bugs, feature requests, support cases and ops related work items. The main metadata for the bug should be title, notes, status, owner and assigning party. Bug tracking software ranges from simple and non-free applications like FogBugz to complex, open source applications like Bugzilla.

Consistent coding standards are more valuable than choosing the right coding standards. Set standards for file names, DB table names, function names, variable names, comments, indentation, etc. Consistency is good.

Testing web applications is hard. They use unit testing for discrete/complex functions and automate as much as they can such as the public APIs. The WWW::Mechanize library has been useful in testing Flickr

Data and Protocols
Unicode is important for internationalization of a site. UTF-8 is an encoding [not a character set] which is compatible with ASCII. Making a UTF-8 web application is tricky due to inconsistent support in various layers of a web application; HTML, XML, JavaScript, PHP, MySQL, and Email all have to be made to support Unicode. For the most part this was straightforward except for PHP which needed custom functions added and filtering out characters below 0x20 from XML files [except for normalizing carriage returns]. A data integrity policy is needed as well as processes for filtering out garbage input from the various layers of the system.

Filtering bad input doesn't just refer to unicode. One also has to filter user input to prevent SQL injection and Cross Site Scripting (XSS) attacks.

The ability to receive email has been very useful to Flickr in a number of scenarios such as enabling mobile 'blogging' and support tracking. Their advice for supporting email is to leverage existing technology and not write an SMTP server from scratch. However you may need to handle parsing MIME yourself because support is weak in some platforms. For Flickr, PEAR's Mail::mimeDecode was satisfactory although deficient. You will also have to worry about uuencoded text and Transport Neutral Encapsulation Format (TNEF) which is only used by Microsoft Outlook. Finally, you may also have to special case mail sent from mobile phones due to idiosyncracies of wireless carriers.

When communicating with other services, XML is a good format to use to ensure interoperability. It is fairly simple unless namespaces are involved. The Flickr team had to hack on PEAR's XML::Parser to make it meet their needs. In situations when XML is not performant enough they use UNIX sockets.

When building services one should always assume the service call will fail. Defensive programming is key. As a consequence, one should endeavor to make service calls asynchronous since they may take a long time to process and it makes callers more redundant to failure.

Developing and Fixing
Discovering bottlenecks is an important aspect of development for web applications. Approaches include

• CPU usage - rarely happens unless processing images/video, usually fixed by adding RAM
• Code profiling - rarely causes problems unless doing crypto
• Query profiling - usually fixed by denormalizing DB tables, adding indexes and DB caching
• Disk IO - usually fixed by adding more spindles
• Memory/Swap - usually fixed by adding RAM

Scaling
Scalability is about handling platform growth, dataset growth and maintainability. There are two broad approaches to scaling; Vertical scaling and Horizontal scaling. Vertical scaling is about buying a big servers, to scale one buys even bigger servers. Horizontal scaling is about buying one server and to scale one buys more of the same kind of server. In todays world, Web applications have embraced horizontal scaling. The issues facing services that adopt horizontal scaling are

• increased setup/admin cost
• complexity
• datacenter issues - power / space / available network ports
• 
  
• underutilized hardware - CPU/Disks/Mem may not be used to full capacity

Services need to scale once they hit performance issues. When scaling MySQL one has to worry about

• Choosing the right backend - MyISAM, BDB, InnoDB, etc
• Replication
• Partitioning/Clustering
• Federation

One big lesson learned about database scalability is that 3^rd normal form tends to cause performance problems in large database. Denormalizing data can give huge performance wins.

The rest of the slides go on about case studies specific to Flickr which are interesting but I don't feel like summarising here. :)
 

Sometimes reading blogs makes you feel like you are in high school. People sometimes chase popularity and wear their immaturity on their sleeve in ways you haven't seen since you were struggling with puberty. One such example is the hubbub around the MashupCamp vs. BarCamp started by Ryan King in his post MashupCamp jumped the shark. Basically some folks who got upset because they weren't invited to Tim O'Reilly's FooCamp came up with a knockoff conference called BarCamp then got upset when another conference called MashupCamp started getting press for the knocking off the same concept. Amazed? I can barely believe it either.

Although the debate [if one can call it that] has been pretty pointless, I did find one quote from Ryan King that I thought was worth highlighting. In his post Live by the Snark, die by the Snark Ryan writes

Wait, no it’s not just me.

David Berlind busted his ass to put together something not vendor-controlled that unavoidably involved vendors because vendor APIs were what was being mashed up.

I see no reason why vendors have to be involved because their services are being mash-up’ed. Aren’t the people writing the mash-ups actually more important here?

The above argument seems weird to me. If I was attending a conference about building Java applications, I'd expect to see Java vendors like Sun and IBM there. If I was attending a conference on building Windows applications, I'd want to see Microsoft developers there. So, if there is a conference about building applications based on data and services from various service providers, why is wouldn't you expect to see the data/service providers at this conference? I think sometimes people take the big companies are bad meme to ridiculous extremes. This is one of those examples.

Personally, I think one of the problems with the discussions around Mashups I've seen at conferences and in blogs is the lack of high level discussion between providers of services/data and developers who use these APIs and data. This is one of the things I miss from when I worked on XML APIs for the .NET Framework. Back then it was clear what developers were interested in our APIs and what they wanted us to do next. The uncertainties were around prioritizing various developer requests and when we could deliver solutions to their problems. On the other hand, now that I've been in discussions around various aspects of the Windows Live platform I've found it hard to figure out who our customers were and what APIs they'd like us to provide. What we need is more dialog between developers building mashups and vendors that provide APIs & data not an us vs. them mentality that builds unnecessary animosity. 

 

A few days ago, someone at work asked about tips on getting traffic to their newly launched team blog. One of the responses suggested that the team should pick blogging at blogs.msdn.com instead of on MSN Spaces because the MSDN blogs get more traffic. This turns out to an example of the myopic view folks have of the blogging when they view it out of the lens of the blogs I read. I'm sure there are lots of bloggers who have never read a blog on MSN Spaces but read dozens on blogs.msdn.com, similarly there are millions of people who read blogs on MSN Spaces that don't even know blogs.msdn.com exists. The main advice the team got was to provide good content and traffic would follow.

However the discussion get me interested in the relative popularity of blogs on MSN Spaces compared to other blogging services. Below are the blogs hosted on MSN Spaces that are on the list of top 100 most linked blogs according to Technorati.

6.   spaces.msn.com/after1s - 27,380 links from 8,916 sites

7.    spaces.msn.com/lin28379801400 - 30,841 links from 8,506 sites

18.  spaces.msn.com/l1n - 16,733 links from 5,960 sites

21.  spaces.msn.com/lovingyo - 16,922 links from 5,215 sites

22.  Herramientas para Blogs - 14,396 links from 5,207 sites

25.  spaces.msn.com/members/MSN SA - 14,021 links from 4,802 sites

37.  The Space Craft - 11,235 links from 4,119 sites (the MSN Spaces team blog)

There two surprises here for me. The first is that two blogs hosted on MSN Spaces are in the top 10 most linked blogs tracked by Technorati and the second is that the MSN Spaces team blog is in the top 50. I'll be quite interested in seeing how these statistics change once Technorati figures out how to add MySpace blogs to their index.

Update: Added Herramientas para Blogs which I missed the first time around.

 

In the past I've mentioned that I don't like the phrase Web 2.0 because it is vacuous and ill-defined making people who use it poor communicators. However, some of the neologisms used by computer geeks are much worse because they just plain dumb. One such neologisms is blogosphere which started off as a joke but is now taken seriously by various pundits.

The blog post that has gotten my goat is Steve Ruble's post The Center of Gravity is Shifting where he writes

One of the themes I kept hitting over and over is that the blogosphere is not where all the action is going to be in the months ahead. Yes, you read that right. Don't adjust your set.

For sure the b'sphere will continue to remain the largest galaxy in the social media universe in the short term. It's a major center of gravity that pulls people toward it. However, over the last few months a number other social media galaxies have rapidly risen to prominence. Take YouTube, digg and MySpace. These are just three examples, but they are drawing huge audiences. Richard Edelman is gushing over a fourth - StupidVideos.com.

The first thing that confuses me about this post is that it implies MySpace isn't part of the blogosphere. Why not? Is it because Technorati's coverage of MySpace is sorely lacking as Steve Rubel claims? Do the media talking heads really think that Technorati covers all the blogs in the world? After all services like MySpace, MSN Spaces and Xanga each have more than the 30 million blogs that Technorati claims to cover. This isn't the first time I've seen someone assume Technorati's numbers actually measure the total number of blogs out there.

The thought that one can lump all the blogs in the world into a lump category called the blogosphere and generalize about them seems pretty silly to me. We don't make similar generalizations about people who use other social applications like email (the mailosphere), instant messaging (the IM-osphere) or photo sharing sites (the photosphere). So what is it about blogs that makes such a ridiculous word continue to be widely used?