I spent the last few days hacking on a side project that I thought some of my readers might find interesting; you can find it at http://hottieornottie.cloudapp.net 

I had several goals when embarking on this project

• As my day job is working on activity feeds in Windows Live, I wanted to prototype various search experiences around searching a feed of our friends' activities.
• I wanted to build an application hosted on Windows Azure to learn more about the platform. 
• I wanted to learn more about the Twitter's various APIs.

After a few days of hacking I'm glad to say I've achieved every goal I wanted to get out of this experiment. I'd like to thank Matt Cutts for the initial idea on how to implement this and Kevin Mark's for saving me from having to write a Twitter crawler by reminding me of Google's Social Graph API.

What it does and how it works

The search experiment provides four kinds of searches

1. The search functionality with no options checked is exactly the same as search.twitter.com

   • Example query: search for "stackoverflow" on Twitter

2. Checking "Search Near Me" finds all tweets posted by people who are within 30 miles of your geographical location (requires JavaScript). Your geographical location is determined from your IP address while the geographical location of the tweets is determined from the location fields of the Twitter profiles of the authors. Nice way to find out what people in your area are thinking about local news.

   • Example query: search for "stackoverflow" mentioned by Twitter users who live within 30 miles of Renton,WA

3. Checking 'Sort By Follower Count' is my attempt to jump on the authority based Twitter search bandwagon. I don't think it's very useful but it was easy to code. Follower counts are obtained via the Google Social Graph API.

   • Example query: search for "stackoverflow" on Twitter sorted by number of followers of the author

4. Checking 'Limit to People I Follow' requires you to also specify your user name and then all search results are filtered to only return results from people you follow (requires JavaScript). This feature only works for a small subset of Twitter users that have been encountered by a crawler I wrote. The application is crawling Twitter friend lists as you read this and anyone I follow should already have their friend list crawled. If it doesn't work for you, check back in a few days. It's been slow going since Twitter puts a 100 request per hour cap on crawlers.

   • Example query: search for "stackoverflow" on Twitter filtered to people followed by Carnage4Life

Developing on Windows Azure: Likes

After building a small scale application with Windows Azure, there are definitely a number of things I like about the experience. The number one thing I loved was the integrated deployment story with Visual Studio. I can build a regular ASP.NET application on my local machine that either used cloud or local storage resources and all it takes is a few mouse clicks to go from my code running on my machine to my code running on computers in Microsoft's data center either in a staging environment or in production. The fact that the data access APIs are all RESTful makes it super easy to go from pointing the app running on your machine to cloud storage or local storage on your machine simply by changing some base URIs in a configuration file. 

Another aspect of Windows Azure that I thought was great is how easy it is to create background processing tasks. It was very straightforward to create a Web crawler that crawled Twitter to build a copy of its social graph by simply adding a "Worker Role" to my project. I've criticized Google App Engine in the past for not supporting the ability to create background tasks so it is nice to see this feature in Microsoft's platform as a service offering. 

Developing on Windows Azure: Dislikes

The majority of my negative experiences were related to teething problems I'd associate with this being a technology preview that still needs polishing. I hit a rather frustrating bug where half the time I tried to run my application it would end up hanging and I'd have to try again after several minutes. There were also issues with the Visual Studio integration where removing or renaming parts of the project from the Visual Studio UI didn't modify all of the related configuration files so the app was in a broken state until I mended it by hand. Documentation was another place where there is still a lot of work to do. My favorite head scratching moment is that there is a x-ms-Metadata-ApproximateMessagesCount HTTP header which returns the approximate number of messages in the a queue. It is unclear whether "approximate" here refers to the fact that messages in the queue have an "invisibility period" between when they are popped from the queue but before they are deleted where they can't be accessed or whether it refers to some other heuristic that determines the size of the queue. Then there's the fact that the documentation says you need to have a partition key and row key for each entry you place in a table but doesn't really explain why or how you are supposed to pick these keys. In fact, the documentation currently makes it seem like the notion of partition keys is an example of unnecessarily surfacing implementation details of Windows Azure to developers in a way that leads to confusion and cargo cult programming.

One missing piece is the lack of good tools for debugging your application once it is running in the cloud. When it is running on your local machine there is a nice viewer to keep an eye on the log output from your application but once it is in the cloud, your only option is to have the logs dropped to some directory in the cloud and then run one of the code samples to access those logs from your local machine. Since this is a technology preview, it is expected that the tooling shouldn't be all there but it is a cumbersome process as it exists today. Besides accessing your debug output there is also seeing what data your application is actually creating, retrieving and otherwise manipulating in storage. You can use SQL Server Management Studio to look at your data in Table Storage on your local machine but there isn't a similar experience in the cloud. Neither blob nor queue storage have any off-the-shelf tools for inspecting their contents locally or in the cloud so developers have to write custom code by hand. Perhaps this is somewhere the developer community can step up with some Open Source tools (e.g. David Aiken's Windows Azure Online Log Reader) or perhaps some commercial vendors will do step in as they have in the case of Amazon's Web Services (e.g. RightScale)?

Outside of the polish issues and bugs, there was only one aspect of Windows Azure development I disliked; the structured data/relational schema development process. Windows Azure has a Table Storage API which provides a RESTful interface to a row-based data store similar in concept to Google's BigTable. Trying to program locally against this API is rather convoluted and requires writing your classes first then running some object<->relational translation tools on your assemblies. This is probably a consequence of not being a big believer the use of ORM tools so having to first write objects before I can access my DB seems backwards to me. This gripe may just be a matter of preference since a lot of folks who use Rails, Django and various other ORM technologies seem fine with having primarily an object facade over their databases.  

Update: Early on in my testing I got a The requested operation is not implemented on the specified resource error when trying out a batch query and incorrectly concluded that the Table Storage API did not support complex OR queries. It turns out that the problem was that I was doing a $filter query using the tolower function. Once I took out the tolower() it was straightforward to construct queries with a bunch of OR clauses so I could request for multiple row keys at once.

I'll file this under "documentation issues" since there is a list of unsupported LINQ query operators and unsupported LINQ comparison operators but not a list of unsupported query expression functions in the Table Storage API documentation. Sorry about any confusion and thanks to Jamie Thomson for asking about this so I could clarify.

Besides the ORM issue, I felt that I was missing some storage capabilities when trying to build my application. One of the features I started building before going with the Google Social Graph API was a quick way to provide the follower counts for a batch of users. For example, I'd get 100 search results from the Twitter API and would then need to look up the follower counts of each user that showed up in the results for use in sorting. However there was no straightforward way to implement this lookup service in Windows Azure. Traditionally, I'd have used one of the following options

1. Create a table of {user_id, follower_count} in a SQL database and then use batches of ugly select statements like SELECT FROM follower_tbl WHERE id=xxxx OR id=yyyy OR id=zzzz OR ….
2. Create tuples of {user_id, follower_count} in an in-memory hash table like memcached and then do a bunch of fast hash table lookups to get the follower counts from each user

Neither of these options is possible given the three data structures that Windows Azure gives you. It could be that these missing pieces are intended to be provided by SQL Data Services which I haven't taken a look at yet. If not, the lack of these pieces of functionality will be sticking point in the craw of developers making the switch from traditional Web development platforms.

Now Playing: Geto Boys - Gangsta (Put Me Down)

This morning, Jeff Atwood wrote a blog post about software piracy entitled My Software Is Being Pirated where he talks about how companies can deal with the fact that the piracy rate among their users could be as high as 90%. He writes

Short of ..

1. selling custom hardware that is required to run your software, like the Playstation 3 or Wii
2. writing a completely server-side application like World of Warcraft or Mint

.. you have no recourse. Software piracy is a fact of life, and there's very little you can do about it. The more DRM and anti-piracy devices you pile on, the more likely you are to harm and alienate your paying customers. Use a common third party protection system and it'll probably be cracked along with all the other customers of that system. Nobody wants to leave the front door to their house open, of course, but you should err on the side of simple protection whenever possible. Bear in mind that a certain percentage of the audience simply can't be reached; they'll never pay for your software at any price. Don't penalize the honest people to punish the incorrigible. As my friend Nathan Bowers so aptly noted:

Every time DRM prevents legitimate playback, a pirate gets his wings.

In fact, the most effective anti-piracy software development strategy is the simplest one of all:

1. Have a great freaking product.
2. Charge a fair price for it.

(Or, more radically, choose an open source business model where piracy is no longer a problem but a benefit -- the world's most efficient and viral software distribution network. But that's a topic for a very different blog post.)

It is interesting to note that Jeff's recommendation for an effective anti-piracy solution is actually contradicted by the example game from his post; World of Goo. The game is an excellent product and is available for ~$15 yet it is still seeing a 90% piracy rate. In fact, the most effective anti-piracy strategy is simply to route around the problem as Jeff originally stated. Specifically

• target custom hardware platforms such as the iPhone or XBox 360 which don't have a piracy problem
• build Web-based software

However if you do decide to go down the shrinkwrapped software route, I'd suggest casting a critical eye on any claims that highlight the benefits of the "Open Source business model" to shrinkwrapped software developers. Open Source software companies have been around for over a decade (e.g. RedHat was founded in 1995) and we now have experience as an industry with regards to what works and what doesn't work as a business model for Open Source software.

There are basically three business models for companies that make money from Open Source software, they are

1. Selling support, consulting and related services for the "free" software (aka the professional open source business model ) – RedHat
2. Dual license the code and then sell traditional software licenses to enterprise customers who are scared of the GPL – MySQL AB
3. Build a proprietary Web application powered by Open Source software – Google

As you scan this list, it should be clear that none of these business models actually involves making money directly from selling only the software. This is problematic for developers of shrinkwrapped, consumer software such as games because none of the aforementioned business models actually works well for them.

For developers of shrinkwrapped software, Open Source only turns piracy from a problem into a benefit if you're willing to forego building consumer software and you have software that is either too complicated to use without handholding OR you can scare a large percentage of your customers into buying traditional software licenses by using the GPL instead of the BSDL.

Either way, the developers of World of Goo are still screwed.

Now Playing: The Notorious B.I.G. - Mo Money Mo Problems (feat. Mase & Puff Daddy)

UPDATE: It seems the problem was hardware related. Disc unreadable errors with perfectly good discs is one of the symptoms of an impending red ring of death.

My wife bought me Gears of War 2 for Christmas from Amazon. So far I've been unable to get past the start screen of the game due to a This disc is unreadable error and attempting to install the game to the hard drive fails at 40% complete. This is the only game in my library that exhibits this problem. I also got Grand Theft Auto IV and have been able to play that fine with no problems.

Searching online indicates that this is a widespread problem.

• Several threads on Yahoo! Answers
• Multiple threads on EuroGamer.Net
• Several threads on the GameSpot forums
• A thread on the Epic Games forum
• There's even been discussions about this on Yelp 

The only consistent thing about all the threads is that I have seen no confirmed solution to the problem. The best suggestion seems to be to return the disk to the retailer and ask for an exchange for a different Gears of War 2 disc. With this option there's still a chance of getting another bum disc and it seems this would be a difficult option to pursue with an online retailer like Amazon.  There's also the option of sending my XBox in for repairs although given that the problem only exists with this one game, it is unclear that this is a problem with my actual hardware.

Before deciding to chalk up the $60 + tax my wife spent on this game to bad luck, I thought I'd throw up this blog post just in case one of my readers has encountered this problem and found a work around.

Merry Xmas!

Now Playing: The Notorious B.I.G. - You're Nobody (Til Somebody Kills You)

Reading depressing layoff related blog posts like George Oates's Not quite what I had in mind and the Valley Wag's Laid-off Yahoos packing heat for Jerry Yang? reminded me that I've been meaning to post about open positions on our team for a while.

The team I work for is responsible for the "social graph", "news feed" and online presence platforms that power various Windows Live experiences. You can see some of our recent efforts in action by downloading Windows Live Essentials (beta) or visiting my profile on Windows Live and browsing around. If you are interested in building world class software that is used by hundreds of millions of people and the following job descriptions interest you then send me your resume 

Software Design Engineer (Developer)

The Windows Live Messenger service is the backbone of one of world’s leading instant messaging services. The service enables hundreds of millions of users to communicate efficiently using text, voice, video and real-time status updates. This high-profile business is growing to accommodate mobile devices, social networking, web applications and other new areas.
We are seeking a developer with a fondness and talent for working on large-scale fault-tolerant distributed systems. The job involves working on the back-end components that maintain user state and route messages and notifications. In addition to improving the systems performance and resiliency, our team will tackle hard new problems such as
- Supporting new ways of addressing users (personas or aliases)
- Extending user state to support offline presence and presence views
- Creating a generic notification service
- implementing effective request throttling and load-balancing across datacenters.

Software Design Engineer/Test (Tester)

Looking for your next big challenge? How about building the next version of the world’s largest IM and social network platform?
We are looking for a great SDET with solid design, problem solving skills and exceptional track record to help build the next version of Windows Live Messenger and Social Network platform. The Messenger network is already one of the largest social networks on the planet, delivering BILLIONS of messages a day for HUNDREDS of MILLIONS of users world-wide.
The SDET role involves working on these next-generation services and proving they can be delivered to the massive scale required, with the quality our users have come to expect. Particular focus areas for this role are scalability, performance and reliability:
Scalability - building software systems to take each piece of hardware to its limits, identifying bottlenecks, removing them and pushing harder; while also, proving the system can grow linearly, as hardware is added. (…think 1,000s of machines).
Performance - ensuring consistently fast response times across the system by smoothly managing peak traffic -- which averages in the 10s of millions of simultaneous online connections.
Reliability - building online services that remain reliable under stress which the operations team is able to easily monitor, troubleshoot, and repair; enabling the aggressive up time requirements we aim for.

Email your resume to dareo@msft.com (replace msft with microsoft) if the above job descriptions sound like they are a good fit for you. If you have any questions about what working here is like, you can send me an email and I'll either follow up via email or my blog to answer any questions of general interest [within reason].

Now Playing: Rascal Flatts - Fast Cars and Freedom

In a post entitled CS Broke Joe Gregorio writes

Computer Science, at a deep and fundamental level, is broken, and that applies not only to software but to hardware. One of the reasons that I have this feeling is that after programming for the past 25 years the field hasn't really changed. The conversations aren't any different. You could substitute 'Windows API' or 'Borland CGI' for 'HTML and CSS' and you'd be having the same exact conversations I had 15 or 20 years ago. We still struggle with leaks, be it memory, or file handles, or threads, or whatever. We still have race conditions. We still struggle with software that grows linearly in features but exponentially in complexity.

Two things came to mind as I read this

1. If your definition of computer science includes HTML & CSS or Win32 then you're doing it wrong. In the words of Edsger Dijkstra, computer science is no more about computers than astronomy is about telescopes.

2. Even if you limit the discussion to computer programming, does it also mean that civil engineering is broken because people today still have to discuss and solve the same kinds problems faced by the builders of the great wall of China or the Roman aqueducts?

Now Playing: Ice Cube - Go to Church

A few days ago, the top news story on Techmeme was the fact that Google launched Google Friend Connect and Facebook announced the final version of Facebook Connect within minutes of each other. Reading up on both announcements it seems interesting to note how most of the coverage is about who will win the race to dominate the Web as opposed to what end user value is actually being created by these technologies.

It was somewhat depressing until I read Dave Winer's brilliant post Soon it will be time to start over, again which contains the following excerpt

We're now reaching the end of a cycle, we're seeing feature wars. That's what's going on between Facebook and Google, both perfectly timing the rollouts of their developer proposition to coincide with the others' -- on the very same day! I don't even have to look at them and I am sure that they're too complicated. Because I've been around this loop so many times. The solution to the problem these guys are supposedly working on won't come in this generation, it can only come when people start over. They are too mired in the complexities of the past to solve this one. Both companies are getting ready to shrink. It's the last gasp of this generation of technology. 

But the next one can't be far away now. It will be exhilirating!!

Remember how great Google was when it first appeared?

Remember how great Netscape was, and before that Apple, and I know you guys won't like this, but Microsoft offered us some great new places to play. I remember finding out that their OS address space in 1981 was 640K. That was a lot to guy who was spending huge amounts of time trying to cram a 256K app into 48K.

The trick in each cycle is to fight complexity, so the growth can keep going. But you can't keep it out, engineers like complexity, not just because it provides them job security, also because they really just like it. But once the stack gets too arcane, the next generation throws their hands up and says "We're not going to deal with that mess." 

We're almost there now.

The value of Facebook Connect to Facebook is obvious. They get to become a centralized identity provider for the Web including the benefit of tracking every single time one of their users logs-in on a partner which lets them build an even better advertising profile of their users. Similarly the value to the customers of the sites adopting it seem clear at first. Below are the claimed benefits of Facebook Connect to users from my initial perusal

1. One login, multiple sites. No need to create a new account on partner sites.
2. Account information such as profile picture, location and other fields on the partner site can be prepopulated from Facebook
3. Bring your social graph with you to partner sites. 
4. Let your friends on Facebook know what you are doing on partner sites. Updates show up on your profile but do not go in your friends' news feeds (they go in their live feed instead). 

Where things get interesting is that none of these benefits require a proprietary and centralized approach like Facebook has done. If Facebook implemented OpenID and OpenID attribute exchange, they could have given their users the benefits of #1 and #2 using widely adopted industry standards.  For #3, there is the burgeoning Portable Contacts effort to define a set of standard APIs for accessing the social graph that supports the key data portability principles around this information. As for broadcasting your updates from one site to another, FriendFeed has shown how that can be done using standard technologies like RSS, Atom and XMPP. 

Ignoring the fact that Facebook Connect is a proprietary and centralized approach instead of being based on open standards, there are still other points worthy of debate. When trying out sites like CitySearch beta with Facebook Connect, the experience is that I am connected with all of my Facebook friends who also use CitySearch. There is the genuine question of whether users really want to use one friends' list across every site regardless of context (e.g. interacting with the exact same people on LinkedIn, MySpace and XBox Live) or whether they want to have universal access to any of their friends lists and bridge them when necessary?

Yesterday on Twitter, I mentioned that Facebook Connect is the wrong direction to go on the Web for the reasons above. I also called Google Friend Connect a misguided "me too" effort for trying to copy Facebook's strategy and glomming an AJAX widget play on top of it. Kevin Marks, an evangelist at Google challenged by statement with the following response

@Carnage4Life the problem for users is re-entering data and restating friends for each app. For developers its becoming social without this

If that is truly the problem, how does the technology in the video below solve the problem any better than the combination of OpenID and Portable Contacts?

As with OpenSocial, Google has fallen in love with its role as a spoiler when it comes to Facebook's platform efforts without stopping to think whether it actually makes sense to be aping Facebook's strategies in the first place. Monkey see, monkey do.

This will be the death of them if they aren't careful. Once you become a follower and define yourself by reacting to others actions, it is hard to step back into a leadership role both in the industry and even within one's corporate culture.

Now Playing: Britney Spears - Circus