One important lesson I've learned about designing software is that sometimes it pays to smother one's perfectionist engineer instincts and be less ambitious about the problems one is trying to solve. Put more succintly, a technology doesn't have to solve every problem just enough problems to be useful. Two examples come to mind which hammered this home to me; Tim Berners-Lee's World Wide Web and collaborative filtering which sites like Amazon use.

1. The World Wide Web: Almost every history of the World Wide Web you find online mentions how Tim Berners-Lee was inspired by Ted Nelson's Xanadu. The current Web is a pale imitation of the what Ted Nelson described over forty years ago of what a rich hypertext system should be capable of doing. However you're reading these words of mine over Tim Berners-Lee's Web not Ted Nelson's. Why is this?

   If you read the descriptions of the Xandadu model you'll notice it has certain lofty goals. Some of these include the ability to create bi-directional links, links that do not break, and built-in version management. To me it doesn't seem feasible to implement all these features without ending up building a closed system. It seems Tim Berners-Lee came to a similar conclusion and greatly simplified Ted Nelson's dream thus making it feasible to implement and adopt on a global scale. Tim Berners-Lee's Web punts on all the hard problems. How does the system ensure that documents once placed on the Web are always retrievable? It doesn't. Instead you get 404 pages and broken links. How does the Web ensure that I can find all the pages that link to another page? It doesn't. Does the Web enable me to view old versions of a Web page and compare revisions of it side by side? Nope.

   Despite these limitations Tim Berners-Lee's Web sparked a global information revolution. Even more interestingly over time various services have shown up online that have attempted to add the missing functionality of the Web such as The Internet Archive, Technorati and the Google Cache.

2. Collaborative Filtering on Amazon: The first place I ever bought CDs online was CDNow.com (now owned by Amazon). One feature of the site that blew my mind was the fact the ability to get a list of recommended CDs to buy based on your purchase history and the ratings you gave various albums. The suggestions were always quite accurate and many times it suggested CDs I already owned and liked a lot.

   This feature always seemed like magic to me. I imagined how difficult it must have been to come up with a categorization and ranking systems for music CDs that could accurately match people up with music based on their tastes. It wasn't until Amazon debuted this feature that I realize the magic algorithms were simply 'people who purchased X aldo purchased Y'. My magic algorithms were just a bunch of not very interesting SQL queries.  

   There are limitations to this approach ,you need a large enough user base and enough purchases of certain albums to make them statistically significant, but the system works for the most part.

Every once in a while I am part of endless discussions about how we need to complicate a technology to satisfy every use case when in truth we don't have to solve every problem. Edge cases should not dictate a software systems design but too often they do.