I recently read Paul Graham’s essay on economic inequality and struggling to fit some of my thoughts into 140 characters on Twitter decided to write a more detailed perspective especially given the article’s core premise is a straw man argument. The core of Paul Graham’s message is captured in the introduction and

Since the 1970s, economic inequality in the US has increased dramatically. And in particular, the rich have gotten a lot richer. Some worry this is a sign the country is broken.

I'm interested in the topic because I am a manufacturer of economic inequality. I was one of the founders of a company called Y Combinator that helps people start startups. Almost by definition, if a startup succeeds its founders become rich. And while getting rich is not the only goal of most startup founders, few would do it if one couldn't.
...
You can't end economic inequality without preventing people from getting rich, and you can't do that without preventing them from starting startups.

With the above sentences, Paul Graham frames any complaints about income inequality in the United States as an attack on the culture and economic processes which have given us companies like Google, Facebook, Microsoft and Amazon which while minting a bunch of super-rich billionaires have also greatly improved the lives of their customers and employees. Since I work in the tech industry this sort of argument should naturally appeal to me but things are never that simple. Paul Graham has attacked a straw man and never really talks about why income inequality has been described as a problem.

Income Inequality: The Pie Fallacy

One part I did find particularly eloquent in Paul Graham’s essay was his description of the pie fallacy of income inequality which is excerpted below

The most common mistake people make about economic inequality is to treat it as a single phenomenon. The most naive version of which is the one based on the pie fallacy: that the rich get rich by taking money from the poor.
Usually this is an assumption people start from rather than a conclusion they arrive at by examining the evidence. Sometimes the pie fallacy is stated explicitly:

...those at the top are grabbing an increasing fraction of the nation's income—so much of a larger share that what's left over for the rest is diminished.... [1]

Other times it's more unconscious. But the unconscious form is very widespread. I think because we grow up in a world where the pie fallacy is actually true. To kids, wealth is a fixed pie that's shared out, and if one person gets more it's at the expense of another. It takes a conscious effort to remind oneself that the real world doesn't work that way.

He’s right, income inequality isn’t occurring because the rich are stealing a larger slice of the economic pie from the middle class and the poor. Growing income inequality is a natural aspect of the way capitalism works. This recently has come to the forefront of current economic thinking due to the book Capital in the Twenty-First Century by Thomas Piketty which spends hundreds of pages providing details of how this has occurred over the past few decades.

The chart below shows the savings rate of various income levels in the US broken into 20% buckets (i.e. quintiles).

The thing to note here is that the bottom 40% of earners in the US saved about a tenth of a percent of their income. Given that the median income in the US is about $50,000 it is unsurprising that people making less than that basically end up spending all of their income with nothing left over to save. On the other hand, the top 20% of earners are saving/investing about a quarter of their income while the top 1% are saving/investing pretty much half their income.

Over time, it’s quite obvious that the net worth of the rich will grow at a higher rate than the net worth of the poor & middle class who are not saving & investing the same proportion of their income. This is further perpetuated over time by the rich then handing off their wealth to their children via inheritance.

The bottom line is that income inequality growing over time is the natural consequence of the rich having more money to invest & save over time than the middle class and poor. One extreme example of this is that Bill Gates money is growing so fast due to growth of his investments is that he has more money now ($79B) than when he started promised to give away all his money in 2008 ($58B)

Again to repeat, the natural state of a capitalist system is that the rich will get richer over time at a rate faster than the poor & middle class given investments and the fact the stock market has effectively grown over time despite massive crashes every few years.

Why Income Inequality is a Problem

For some, the fact that income inequality exists is a sign that capitalism is unfair and fundamentally broken. One of the responses to Paul Graham have basically argued this point including using the example of the inherent unfairness of a school teacher struggling to pay rent while Candy Crush Saga generates billions for investors and shareholders. I’m not going to make that argument.

There’s a great interview with Thomas Piketty on the Wealth Divide where he addresses the specific question of why income inequality is bad

Q. What are the risks from allowing an ever-increasing concentration of wealth and incomes? Is there a point when inequality becomes intolerable? Does history offer any lessons in this regard?

A. U.S. inequality is now close to the levels of income concentration that prevailed in Europe around 1900-10. History suggests that this kind of inequality level is not only useless for growth, it can also lead to a capture of the political process by a tiny high-income and high-wealth elite. This directly threatens our democratic institutions and values.

This is a very important distinction. Income inequality is basically how capitalism works and overall the system is working as designed. However as we create a world where the super rich get even richer over time, there is an increasing risk of these rich people using their vast resources to subvert the political process to protect their interests. There are obviously tons of examples of this occurring in America today.

One example of this subversion is that income from investments (i.e. how rich people primarily make their income) is taxed at a lower rate than income from salaries (i.e. how the middle class and poor make their income). This is described fairly well in Mark Suster's response to Paul Graham's essay 

3. Both of these privileged, very small group of people in 1 & 2 [Ed: founders & investors], have much better tax rates than say, the third employee at a startup who might have joined 3 months after the founders. That employee was given “stock options,” which pay the exact same rate of taxes as income. In California considering state, federal and local taxes that can be as high as 56%. Think about it – if the first two employees work 6 years and sell a company while employee 3 works 5 years and 9 months … should they really pay grossly different tax rates? Of course if an employee “exercises” his or her options AND holds the stock more than one year then they are eligible to earn long-term capital gains. But this often requires relatively large sums of money and it implies writing a check in a company whose future is uncertain. That might actually seem fair. But ask yourself why employee three (and four and four hundred) has to write the check while employees 1 & 2 do not?

I wish founders, startup employees and VCs all paid the same rate of taxes. I also wish we paid the same amount of taxes as nearly any employee earning above-average income. But we all don’t and we’re not likely to fix any of that.

This dual tax system gets even more sophisticated for the billionaire class as described in this recent New York Times Article; For the Wealthiest, a Private Tax System That Saves Them Billions which begins

The hedge fund magnates Daniel S. Loeb, Louis Moore Bacon and Steven A. Cohen have much in common. They have managed billions of dollars in capital, earning vast fortunes. They have invested large sums in art — and millions more in political candidates.

Moreover, each has exploited an esoteric tax loophole that saved them millions in taxes. The trick? Route the money to Bermuda and back.

With inequality at its highest levels in nearly a century and public debate rising over whether the government should respond to it through higher taxes on the wealthy, the very richest Americans have financed a sophisticated and astonishingly effective apparatus for shielding their fortunes. Some call it the “income defense industry,” consisting of a high-priced phalanx of lawyers, estate planners, lobbyists and anti-tax activists who exploit and defend a dizzying array of tax maneuvers, virtually none of them available to taxpayers of more modest means.
…
“There’s this notion that the wealthy use their money to buy politicians; more accurately, it’s that they can buy policy, and specifically, tax policy,” said Jared Bernstein, a senior fellow at the left-leaning Center on Budget and Policy Priorities who served as chief economic adviser to Vice President Joseph R. Biden Jr. “That’s why these egregious loopholes exist, and why it’s so hard to close them.”

It’s an open secret that politicians consider courting the super rich and their money as key to winning elections. In many cases, the assumption is that getting money from the super rich is tantamount to winning an election. This conventional wisdom has been recently put to the test in the presidential elections with the surprising rise of Donald Trump as detailed in the article One year, two races: Inside the Republican Party’s bizarre, tumultuous 2015

“Shock and awe” is how it came to be called, to the chagrin of Bradshaw and others. Still, it was a genuine blitzkrieg. Bush’s advisers established Right to Rise, a super PAC that could accept unlimited contributions, and it vacuumed up big checks by the day. On Jan. 9, it received its first $1 million contribution, from Los Angeles investment banker Brad Freeman. By February, Bush was averaging one fundraiser a day and regularly headlining events with a minimum price tag of $100,000 a person, such as the Feb. 11 gathering at the Park Avenue home of private-equity titan Henry Kravis.

Longtime Bush family fundraiser Fred Zeidman recalled: “Everyone was enthusiastic, everyone was writing checks. That had always been the benchmark. Money has been the way you keep score.”

The intense early pace startled Bush’s likely opponents. “I think everybody was a little surprised as to not just the timing but how successful he was early on,” Wisconsin Gov. Scott Walker recalled later.

I could go on but the point should be clear enough. If you have more I’d also suggest reading Anatomy of the Deep State which gives a lot more food for thought on how the super rich can and have subverted the democratic processes in many parts of American life.

In summary, the primary problem caused by growing income inequality is that it perpetuates the creation of a separate class of people whose wealth allows them to control politicians and influence legislation in ways favorable to them and potentially unfavorable to the middle & lower classes. This manifests itself in lots of ways from obvious things like different tax policies for investments versus wages to using affluenza as a legal defense for crimes committed by children of the rich and more.

 

Piketty’s Solution to Income Inequality

Since Thomas Piketty deserves the credit for bringing these ideas to the forefront in recent years we should take a look at how he proposed addressing this problem. As covered in the Guardian’s review of his book his idea is straightforward

Piketty's call for a "confiscatory" global tax on inherited wealth makes other supposedly radical economists look positively house-trained. He calls for an 80% tax on incomes above $500,000 a year in the US, assuring his readers there would be neither a flight of top execs to Canada nor a slowdown in growth, since the outcome would simply be to suppress such incomes.

This is why I called Paul Graham’s argument a straw man. High tax rates on top earners are not the same thing as preventing people from becoming rich or stopping the creation of startups. The US tax rate was at 70% or higher between World War II and 1981 when the Economic Recovery Tax Act was made law. During this period of high tax rates on top earners a number of startups that went on to change the world were founded including Apple (1976), Intel (1968), Microsoft (1975) and Oracle (1977) as well as a bunch more which aren’t here today but had a huge impact during their hay day (e.g. Digital Equipment Corporation).

One could ask the question as to whether Mark Zuckerberg would still have created Facebook if he knew that his tax rate would be 80% (Piketty’s goal) if he became a billionaire versus 40%-50% (current tax rates) and  I suspect the answer for most founders would be Yes.

How Piketty’s Solution Impacts Tech Startups

That said, Piketty’s solution would have a material impact on one of the most important aspects of tech startups; hiring. Many high earning tech hires getto make the choice of working at a big established company like Microsoft, Facebook or Google versus working at an up and coming startup like Slack, Snapchat or Zenefits. As Dan Luu pointed out in his excellent post on Big Company vs. Startup Work and Pay it is not unusual for a top performer

The numbers will vary depending on circumstances, but we can do a back of the envelope calculation and adjust for circumstances afterwards. Median income in the U.S. is about $30k/yr. The somewhat bogus zeroth order lifetime earnings approximation I’ll use is $30k * 40 = $1.2M. A new grad at Google/FB/Amazon with a lowball offer will have a total comp (salary + bonus + equity) of $130k/yr. According to glassdoor’s current numbers, someone who makes it to T5/senior at Google should have a total comp of around $250k/yr. These are fairly conservative numbers¹.

Someone who’s not particularly successful, but not particularly unsucessful will probably make senior in five years².
...
If you’re an employee and not a founder, the numbers look a lot worse. If you’re a very early employee you’d be quite lucky to get 1/10th as much equity as a founder. If we guess that 30% of YC startups fail before hiring their first employee, that puts the mean equity offering at $1.8M / .7 = $2.6M. That’s low enough that for 5-9 years of work, you really need to be in the 0.5% for the payoff to be substantially better than working at a big company unless the startup is paying a very generous salary.

There’s a sense in which these numbers are too optimistic. Even if the company is successful and has a solid exit, there are plenty of things that can make your equity grant worthless. It’s hard to get statistics on this, but anecdotally, this seems to be the common case in acquisitions.

Moreover, the pitch that you’ll only need to work for four years is usually untrue. To keep your lottery ticket until it pays out (or fizzles out), you’ll probably have to stay longer. The most common form of equity at early stage startups are ISOs that, by definition, expire 90 at most days after you leave. If you get in early, and leave after four years, you’ll have to exercise your options if you want a chance at the lottery ticket paying off. If the company hasn’t yet landed a large valuation, you might be able to get away with paying O(median US annual income) to exercise your options. If the company looks like a rocketship and VCs are piling in, you’ll have a massive tax bill, too, all for a lottery ticket.

As someone whose talked to a number of friends and coworkers who’ve weighed the cost of switching from a my employer (Microsoft) to a startup, I have seen first hand that the cost of going to a startup versus staying at a big company is something like $50,000 – $100,000 per year in lost wages for people with 5 – 10 years of experience. So startups sweeten the pot by giving people stock options which if the company is reasonably successful, makes up for the lost wages.

For example, a senior developer making $250,000 at a company like Google would likely take a haircut to about $150,000 to work at Pinterest. Pinterest would have to give this developer enough equity such that if they stay about 3 – 4 years at the company, they’d earn back the $300,000 – $400,000 that was foregone plus some interest given they’d likely expect a promotion or two if they had stayed at Google. Note that this isn’t about getting rich, this is just breaking even on income. So it isn’t unusual for someone in this situation to get $500,000 – $1 million in options/restricted stock depending on their seniority and the potential the company sees in them over time.

If Piketty has his way then the tax bill on those options would shoot way up and makes sticking it out at a big company more attractive for top tech hires.

Stagnant Wages: A Related but Different Problem

One problem that is regularly conflated with income inequality as described by Piketty is stagnant growth of wages in America especially as companies are making record profits. With the rise of 401Ks, the influx of more money in the stock market from the Reagan tax cuts and the cult of maximizing shareholder value (Thanks Jack Welch) there has been a lot of pressure for companies to make as much profits/dividends for shareholders as possible while extracting as much as possible from employees. CEOs and executives are especially incentivized to do this since they get huge payouts as shareholders for managing these numbers. US companies have gone to great lengths from outsourcing & increased use of automation to union busting to avoid increasing expenditure on labor thus limiting wage growth.

Stagnant wages over the past few decades further exacerbates income inequality but are not the fundamental cause. Even if wages had been rising over the past few decades at the same rate as before, there is no historical precedent for them to have risen faster than the US stock market which means those with investments would still be seeing their wealth grow over time faster than those earning paychecks.

Now Playing: Charlie Puth & Lil Wayne – Nothing But Trouble (Instagram Models)

 

Recently I was reading an article titled When Facebook Knows You Better Than You Know Yourself which talks about all of the ways Facebook could predict our behavior thanks to to all of the information we have shared with the site with our likes, shared links and the various websites & apps we use that are connected to Facebook. This article then links off to Why Does Facebook Keep Suggesting You Friend Your Tinder Matches? which seems to present a conundrum

Maria Ledbetter has noticed six people she has met on Tinder in her Facebook suggested friends within the last few months, including one match who showed up so late to their date that she left. She said the suggested friends from Tinder often pop up within a week of getting her number, usually in cases where she hasn’t spoken with them since.

“It’s always people I don’t even talk to, have deleted their number, and have no friends in common,” she said. “It’s really frustrating.”

Emilio Ferrara, a data science and machine learning professor at Indiana University who studies social networks said the most obvious answer would be that these apps are collecting and sharing your information.

“It is likely that these social network companies are buying data from one another, which means that Facebook can acquire some information on user activity from other platforms,” he said.“If that’s the case, it would be very easy to cross match.”

“It could also be a coincidence,” he added. “But I don't believe very much in coincidences.”

The article goes on with a number of theories and quotes from other experts trying to understand the magic behind how Facebook seems to know who you’re talking to in dating apps when you’ve not added them as a contact on your phone nor have any friends in common. The answer is actually quite simple; your Tinder/OKCupid/Grindr dates have your phone number.

A long standing capability of Facebook is that it creates shadow profiles of its users. For example, if my email address known to Facebook was dare@example.com and phone number 555-1212 then when someone joins Facebook with that email address or phone number as a contact then Facebook knows that they know me. So Facebook has a “shadow” friend list of people who have my email address and phone number even if I haven’t added them as a friend nor have mutual friends in common with them.  Since there are lots of people who sync their phone contacts with Facebook there are likely dozens of people that Facebook knows you know even if you’ve not added them as friends.

Mystery solved. 

Now Playing: Chris Brown – New Flame (feat. Usher & Rick Ross)

 

Yesterday Twitter announced Fabric, a new mobile SDK for Android and iOS composed of four distinct pieces

• Crashlytics – an application analytics package that gives developers tools to measure how their apps are being used and measure app quality in the wild (i.e. crashes).
• Twitter Kit – an SDK that makes it add Twitter integration such as signing in with Twitter, embedding tweets or posting tweets from your app
• MoPub Kit – makes it easy to embed ads from Twitter’s ad network in your app so you can make money
• Digits – makes it easy for any app to build phone number based sign-in similar to what Skype Qik and WhatsApp have. This is quite frankly a game changer.

The response to this release I’ve seen online have swung between two extremes, fawning adoration from the tech press proclaiming that Twitter has moved beyond tweets into mobile services and skepticism from developers who don’t trust Twitter as represented in this tweet below

Did Twitter address in any way their platform trust issues today? Innovation in guaranteeing long term trustworthiness would be fascinating.

— kellan (@kellan) October 23, 2014

The root of this angst is Twitter’s tumultuous relationship with developers of Twitter clients which eventually led to their infamous quadrant of death post which effectively limited the growth of any app whose primary function was to be a replacement Twitter experience. This hurt many developers who had been working on Twitter reading experiences and in fact led to the CEO of Flipboard quitting Twitter’s board in disgust.

Thus it is a valid question for developers as to whether they can trust Twitter this time? The answer is Yes for a very simple reason. Twitter’s API moves in 2012 and yesterday’s announcements were borne from the same motives, to grow its primary business of selling ads tied to their mobile experiences. In 2012, they had to address the fact that their liberal exposure of their service via their API had created a situation where a huge slice of their user base were using the app through experiences Twitter could not effectively monetize. 

At the height of the 3rd party Twitter app boom almost half of their users were using official apps (42%) although that percentage dwindled as they stepped up their mobile app efforts and sent the message to app developers that they no longer wanted people to compete with them on providing mobile experiences.

 

Taking control of the primary user experience for Twitter was the smart business decision and is why they now  generate over a billion dollars a year as a business.

This brings us to Fabric. All four components aid Twitters core business of selling ads for mobile experiences.

• Twitter Kit increases engagement with Twitter by making it easy for users to consume and generate tweets from other apps without those apps being a threat to Twitter by becoming competing experiences.
• Digits allows Twitter to build a profile of users based on their phone number the same way Facebook builds a profile of users based on the apps and websites they visit that use Facebook Connect.
• Crashlytics + MoPub is the Trojan horse with a approach to Flurry which Yahoo acquired for $200 million. Crashlytics is a incredibly useful component that is valuable to all mobile apps since they all care about user behavior and crashes. Once you’re hooked on Crashlytics, it’s easier to upsell you to also using Twitters ad network and hence $$$.

All of these efforts help Twitter’s core business and it would be insanity for them to screw developers by abandoning them just as it would have been insanity for them to pursue an ad-based business model in a world where a huge chunk of their most active users were using 3rd party apps as their primary Twitter experience.

So go ahead, try out Fabric and judge it on its merits. I’m curious to hear what you think.

Now Playing: Chris Brown – Loyal (featuring Lil Wayne and Tyga)

 

I’ve read a number of articles about account security, passwords and secret questions this week for obvious reasons. Although I’ve seen a number of posts directed at end users as to how to better safeguard their accounts, I haven’t seen anything similar providing guidance to developers of online services on how to better safeguard their users in what is a very hostile environment.

Below are the top five (plus a bonus one) account security features that every competent online service should have implemented. None of these are ground breaking but it is quite clear that many services that we all use every day don’t implement even these basic security features thus putting our data at risk.

1. Strong passwords including banning common passwords: The most basic practice is requiring that users create a strong password often by requiring some combination of minimum length, at least one of upper & lower case character and encouraging the use of punctuation. Although this is a good first steps there are other steps services need to take to ensure their users are using hard to guess passwords. One such approach is to take a look at the common common choices of user passwords that have been observed as a result of website hacks. 

   Analysis of these lists show that people are quite predictable and you often find "password", "abc123", "letmein" or the name of the website being used by a sizable percentage of the users on your site. It thus makes sense to ban users from using any of these fairly common passwords which can then lead to successful drive-by hacking incidents. For example, a hacker can take the basic approach of trying to log-in to a bunch of users accounts using "password", "123456" as their email address and if past history is a judge can end up compromising thousands of user accounts with just this brain dead tactic.

2. Throttling failed password attempts: Regardless of how strong a user’s password is, it is trying to stop a bullet with a wet paper towel against a dedicated brute force attack if no protections are in place. Password cracking tools like John the Ripper can crack a strong eight character password in about 15 minutes. This means to fully protect users, online services should have a limit on how often a user can fail a password challenge before you put some road blocks in their way. These road blocks can include exponentially increasing delays after each failed attempt (wait 1 minute, if failed again then 2 minutes, etc) or requiring the person to solve a CAPTCHA to prove they are human.

   Another thing services should do is look at patterns of failed password attempts to see if broader prevention strategies are necessary. For example, if you are seeing hundreds of users failing multiple password attempts from a particular IP range you may want to block that IP range since given our previous discussion about weak passwords they probably have successfully hacked some of your accounts.

3. 2-factor authentication: Every online service should give customers the option to trade convenience (i.e. password only sign in) with more security. Two-factor authentication is typically the practice of combining something the user knows (e.g. a password) with something the user has (e.g. their smart phone or biometric data). Although more inconvenient than just providing a password, it greatly increases the security for users who may be desirable targets for account hijackings or when providing a service that holds sensitive data. This is why it is supported by a number of popular online service providers including Google, Microsoft and Twitter.

   A common practice to improve the usability of 2-factor authentication is to give users the option to only require it the first time the sign-in from a particular device. This means that once the user goes through the two step authentication process from a new computer, you can assume that that device is safe and then only require a password the next time they sign in from that device. 

4. Choose better secret questions or better yet replace them with proofs: Inevitably, users will forget the password they use with your service especially if you require strong passwords and have a policy that is incompatible with their default password choice (which hopefully isn’t “password1” ). A common practice, which has now become an Achilles heel of account security, is to have a set of back up questions that you ask the user if they have forgotten their password. The problem for account security is that it is often easier to guess the answers to these questions than it is to hack the user’s password. There is a great check list for what makes a good secret question at goodsecurityquestions.com with examples of good, fair and poor security questions.

   In general you should avoid security questions because most can be easily guessed such as what is your favorite color or sports team and for others their answers can be easily found on Facebook such as where the user went to high school or via social engineering your friends. A much better approach is to use a similar approach to 2-factor authentication where a user provides proof of something they have such as their smartphone (send an SMS) or alternate email account (send an email) to verify that they are who they say they are.

5. Show customers their sign-in activity: When all else fails, it is important to give your customers the tools to figure out for themselves if they have been hacked. A good way to do this is to let them know of sign-in attempts that have occurred on their account so they can that either failed or were successful. Google does this today via its last account activity feature. You can find this by going to security.google.com and click Recent activity under “Security” on the left. Microsoft provides this with its recent activity feature which you can find by going to https://account.live.com/activity.

Implementing these features isn’t a cure all for account security woes and should instead be treated as the minimum bar for providing a reasonable level of security for your users. 

 

Now Playing: Beyonce – Flawless Remix (featuring Nicki Minaj)

 

For the past couple of days, the tech press has been in an uproar from the news initially published in the AV Club that Facebook tinkered with users’ feeds for a massive psychology experiment in 2012. The money quote from the article is below

It shows how Facebook data scientists tweaked the algorithm that determines which posts appear on users’ news feeds—specifically, researchers skewed the number of positive or negative terms seen by randomly selected users. Facebook then analyzed the future postings of those users over the course of a week to see if people responded with increased positivity or negativity of their own, thus answering the question of whether emotional states can be transmitted across a social network. Result: They can! Which is great news for Facebook data scientists hoping to prove a point about modern psychology. It’s less great for the people having their emotions secretly manipulated.

The strange thing about the recent uproar is that the focus of the anger seems to be that Facebook ran the experiment. This is strange if you actually stop and think about what we actually know as humans.

1. People are influenced by what they see including what they see on social networks like Facebook. Remember all those, "Facebook makes you sadder" headlines from a year or two ago? How about the fact that just yesterday, the MayDay PAC raised $5 million from almost 50,000 people thanks to viral sharing on social media sites by people like George Takei? These are thousands of people being influenced to spend money to change how their government works based on what they saw in their news feed.

2. Facebook controls what you see in your news feed.

The second point can’t be emphasized enough. Remember when Facebook explicitly spelled out how Edgerank works?

Over the past few years, Facebook has made hundreds of tweaks to the news feed. Some we notice and others we don’t. The above image was from an article explaining one such tweak which caused posts by brands to start showing up much less in the news feed. Over the past few years Facebook’s news feed tweaks have caused our feeds to be filled with too much and then over time very few quizzes and polls, Zynga games like Mafia Wars & Farmville, articles my friends are reading from social readers, videos from social video sites like Viddy, Bitstrips comics and of course, Upworthy articles to name a few.

For each of these waves of content dominating our news feeds, some product manager decided to turn up or turn down the dial of said content based on our “engagement” with Facebook. There is no outside party vetting these changes nor is there even a way for such an interested party to even tell what these changes are. It is quite unprecedented in the history of the world for any entity (company or government) to control so much of the media that millions of people see daily without any visibility into its agenda or the content it is feeding to its subjects.

Most people who are still bloviating on this topic on Techmeme are upset that Facebook “manipulated people’s emotions without any oversight for an experiment” when the reality is that Facebook manipulates people’s emotions via tinkering with the news feed to increase their engagement (i.e. time spent on the site looking at ads) every minute of every hour of every day. 

That’s why Sheryl Sandberg gave this shrug as she responded that the major problem with the experiment is that it was poorly communicated. She’s right. Facebook does this every day. Manipulating your behavior by manipulating your news feed is their primary business. If anything, this experiment should be commended because it implies Facebook had at least at one point considered the impact of this manipulation on the psychological health of its users and wanted to understand it better.

Speaking of lack of oversight and transparency, one can’t help but wonder what subtle dampeners or viral boosts Facebook puts on sharing of content depending on the politics of the situation. For example, it’s interesting that George Takei posts still garner hundreds of thousands of likes each time they show up when other Facebook pages are seeing double digit percentage declines. With other media like Fox News or the Wall Street Journal, their agenda is understood by all and quite clear. On the other hand, Facebook editing which content from your friends or brands that you see, is driven by an unknown agenda while masquerading as serendipitous and organic content.

Maybe Facebook doesn’t manipulate your feed depending on politics. Maybe it did at one time then stopped. Maybe they will in the future. We don’t know and if it ever does happen we won’t even realize it.

So go ahead and freak out about one A/B test in 2012. That totally seems like the most worrisome thing about Facebook’s power over its users.

Now Playing: Rick Ross – Nobody (featuring French Montana)

 

As I write this the latest version of Skype for the iPhone has a 2 star rating as does Swarm by Foursquare. What these apps have in common is that they are both part of bold attempts to redesign a well-known and popular app which are being rejected by its core constituency. A consequence of my time working on Windows 8 is that I now obsess quite a bit about redesigning apps and determining what warning signs indicate that you are either going to greatly please or strongly offend your best users.

When I worked on Windows 8, there were a number of slogans that the team used to ensure the principles behind the work we were doing were understood by all. Some of them such as “content over chrome” were counterproductive in that slavish devotion to them led to ignoring decades of usability research by eschewing affordances and hiding navigation/controls within apps. However there were other principles from the Windows 8 era which I wish app developers took more to heart such as “win as one” which encouraged consistency with the overall platform’s UI model & working with other apps and “change is bad unless it's great” which encouraged respecting the past and only making changes that provided a noticeably better user experience. 

In addition to these two principles, I’ll add one more for app developer to keep in mind whenever the time calls for a redesign; “minimize the impacts of loss aversion”. For those who aren’t familiar with the term, loss aversion (aka the endowment effect) is the tendency for humans to strongly prefer avoiding losses to making gains. What this means for developers is that end users will react more strongly to losing a feature than they would to gaining that same feature. There are numerous studies that show how absurd humans can be in the face of loss aversion no matter how minor. My favorite example is how much people overreact to loss aversion when it comes to grocery shopping as taken from this blog post by Jon Geeting

There was a law set up last month in D.C. (passed unanimously by city council) to place a five-cent tax on paper and plastic bags at grocery stores, pharmacies and other food-service providers. So, basically, if I went shopping, my total came to $35.20, and I needed one bag to put it in, my total would then become $35.25. Similarly, if I needed two bags, my total would become $35.30, and so on — while if I simply bought reusable bags, I would be subject to no tax.

From what I hear from people in D.C., they absolutely hate it. Even though it’s just an extra five cents, they want absolutely nothing to do with it. They really want that nickel. So many people use less bags, bring their own, or just try to balance everything without one on their trip home. Think about how much less waste and pollution there is in D.C. now, because of a measly five-cent fee.

On the flip side, if you told people you’d give them 5 cents for each bag they brought from home they’d laugh in your face. Nobody is going to do an extra bit of work to be paid five cents even though they would do that work to avoid paying 5 cents. That’s loss aversion at work.

To recap, if you are redesigning an app you need to keep these three rules in mind

1. Win as one: Whatever changes you make must feel like a consistent whole both within the app and with the platform your app resides on. Swarm and Foursquare have completely different aesthetics and integrate in a fairly disjointed manner often with no way to easily jump back and forth between both apps. Skype for iPhone is pretty much a Windows Phone app in look and feel complete with pivot controls and cut off title text. This is a very jarring experience compared to everything else on iOS.

2. Change is bad unless it’s great: App developers need to be honest with themselves about whether a redesign is about solving a customer problem in a better way or is part of a corporate strategy. Facebook news feed is an example of a redesign which was actually driven by a need to solve customer problems which is why although it met with a massive user revolt at first, once people used it they loved it and the anger died down. Swarm exists because FourSquare now wants to compete with Yelp and needs to shed its history as a social check-in app which it sees as baggage as it evolves into a social discovery engine of things to do in your city. From an end user perspective, Skype for iPhone’s redesign is about making the app look and feel like a Microsoft metro-style app. Given these primary goals, it is no surprise that end users can tell that solving their problems came in second place as they review these apps.

3. Minimize the impacts of loss aversion: Coupling a redesign with taking away features means people will focus on the missing features instead of whatever benefits you have provided with the redesign. Foursquare took away badges, mayorships, social feed of your friends check-ins and points as part of the split that created Swarm. There are a large number of one star reviews of the Swarm app complaining about these missing features. Skype for iPhone’s initial release took away deleting & editing messages while making others harder to find. Even features that are used once in a blue moon seem mission critical once people find out they are gone. Taking away features will always sting more than the actual value of those features. Taking multiple features away as part of a redesign means any benefits of the redesign will be lost in the ensuing outrage about the missing features.

Now Playing: Rick Ross – The Devil is a Lie (featuring Jay-Z)

 

The most interesting news from Facebook’s F8 last week was the announcement of App Links. If you are unfamiliar with the announcement, watch the 1 minute video embedded below which does a great job of setting up the sales pitch. Using App Links, mobile app developers can put markup in their web pages that indicate how to launch that page in their application on Android, iOS and Windows Phone. For example, clicking on a link to a FourSquare check-in from the news feed will launch the FourSquare app on your phone and will open that specific location or event.  .

The interesting question is why is Facebook doing this? It boils down to the fact that Facebook is an advertising company which makes the majority of its revenue from those ads asking you to install Candy Crush and Clash of Clans in your news feed.

Facebook’s pattern at this point is well known. They give you something of value for free (traffic) and once you get hooked they dial it down until you have to pay. The world is littered with the ashes of various companies who were once media darlings because Facebook gave them a bunch of free traffic from liberal news feed algorithms and then turned off the spigot. Just ask Viddy, all those social readers, Zynga, or read that hilarious break up letter from those guys at Eat24.

Publishers who use app links will likely get a boost in the news feed algorithm likely under the pretext that they provide a better user experience to consumers. Early success stories will cause lots of developers to create app links and then get hooked on the traffic they get from Facebook. Eventually your traffic will start dropping and any complaints will be met with an elaborate mathematical formula which explains why your content isn’t that hot on Facebook anymore. But don’t worry, you can fix all that by buying ads.  

It’s obvious, devious and I love it. Especially since it does actually move the user experience of the mobile web forward even if the end goal is to make Facebook tons of money.

The other thing I give Facebook props for is holding a mirror up to the major search engines to see how silly we were being. Bing supports standards for app linking but it's only for Windows & Windows Phone apps. Google supports the same and again it only works for Android apps. Facebook is trying to say it doesn’t matter if you are on the web, Windows Phone, Android or iOS, links in the news feed should open in the native app on that platform. Google and Bing’s search engines on the other hand only supported the same when searching on the OSes from their parent companies. #strategytax

Hopefully Facebook’s move will bring more inclusiveness across the board from many online platform providers not just search engines. For example, I would love it if email providers also supported app links as well.

Now Playing: DJ Snake & Lil Jon – Turn Down For What

 

Chris Dixon has a fairly eloquent blog post where he talks about the decline of the mobile web. He cites the following chart

and talks about what it means for the future of innovation if apps which tend to be distributed from app stores managed by corporate gate keepers continue to dominate the web as the primary way people connect on the Internet.

Using HTTP Doesn’t Make Something Part of the Web

In response to Chris Dixon’s post I’ve seen a fallacy repeated a number of times. The most visible instance of this fallacy is John Gruber’s Rethinking What We Mean by ‘Mobile Web’ where he writes

I think Dixon has it all wrong. We shouldn’t think of the “web” as only what renders inside a web browser. The web is HTTP, and the open Internet. What exactly are people doing with these mobile apps? Largely, using the same services, which, on the desktop, they use in a web browser.
...
Yes, Apple and Google (and Amazon, and Microsoft) control their respective app stores. But the difference from Dixon’s AOL analogy is that they don’t control the internet — and they don’t control each other. Apple doesn’t want cool new apps launching Android-only, and it surely bothers Google that so many cool new apps launch iOS-first. Apple’s stance on Bitcoin hasn’t exactly kept Bitcoin from growing explosively. App Stores are walled gardens, but the apps themselves are just clients to the open web/internet.
...
The rise of mobile apps hasn’t taken anything away from the wide open world of web browsers and cross-platform HTML/CSS/JavaScript — other than supremacy. I think that bothers some, who saw the HTML/CSS/JavaScript browser-centric web’s decade-ago supremacy as the end point, the ultimate triumph of a truly open platform, rather than what it really was: just another milestone along the way of an industry that is always in flux, ever ebbing and flowing.

What we’ve gained, though, is a wide range of interaction capabilities that never could have existed in a web browser-centric world. That to me is cause for celebration.

The key point here is that the World Wide Web and the Internet are different things. The definition of the web I use comes from Tim Berners-Lee’s original proposal of a browsable information network of hyperlinked documents & media on a global network. The necessary building blocks for this are a way to identify these documents (URIs), the actual content of these documents (HTML/JS/CSS/media), how clients obtain these documents (HTTP) and the global network they site on (The Internet).

This difference is important to spell out because although HTTP and the Internet are key parts of the world wide web, they aren’t the web. One of the key things we lose with apps is public addressability (i.e. URIs for the technically inclined). What does this mean in practice

• Visiting a website is as simple as being told “go to http://bing.com” from any browser on any platform using any device. Getting an app requires the app developer to have created an app for your platform which may not have occurred due to technical limitations, policy limitations of the platform owner or simply the cost of supporting multiple platforms being higher than they want to bear.

• Content from apps is often invisible to search engines like Google and Bing since their information is not part of the web.

• Publishing a website simply requires getting a web host or even just hosting your own server. Publishing an app means submitting your product to some corporation then restricting your content and functionality to their rules & regulations before being made available to end users.

The key loss being that we are regressing from a globally accessible information network which reaches everyone on earth and where no publisher needs permission to reach billions of people to lots of corporate controlled fiefdoms and walled gardens.

I don’t disagree with Gruber’s notion that mobile apps have introduced new models of interaction that would not have existed in a web-browser centric world. However that doesn’t mean we aren’t losing something along the way.

Now Playing: The Heavy – Short Change Hero

 

This morning I saw the following tweet from Steven Levy, a Wired reporter who's written a number of interesting books about software people and the great companies they've built

I hate it when something I look at on Amazon turns up in a Facebook ad. It doesn't help me, it creeps me.

— Steven Levy (@StevenLevy) February 17, 2014

As part of my day job at Microsoft, I've begun to learn more about how advertising across the internet works on a technical level and it is quite interesting to learn how an image of a some head phones I looked at an e-commerce site ended up staring back at me from an ad on Facebook later that day.

The fundamental technology that makes this possible is Facebook Exchange (FBX). The infographic below provides an overview of how it enables ads from ecommerce sites to show up on Facebook and I’ll follow that up with a slightly more technical explanation.

Source: Business Insider

Facebook Exchange is a Real-Time Bidding platform which enables Facebook to sell ad slots on their page to the highest bidding advertisers in fractions of a second. Typically advertisers and publishers who own the pages where ads show up end up working together through an intermediary called a Demand Side Platform (DSP). A DSP such as AdRoll provides one of their retail partners such as American Apparel or Skull Candy with code to put tracking pixels on their site which allows the user to be identified and context such as what pages they’ve visited to be recorded. The retail partner then goes into AdRoll’s interface and decides how much they are willing to pay to show ads on various networks such as Facebook (via FBX) if a user who has visited one of their pages is shown an ad.

AdRoll then provides data to Facebook that allows the user to be uniquely identified within Facebook’s network. Later when that same user goes to Facebook, Facebook puts out a request on its Ad Exchange saying “Here’s a user who you might be interested in, how much are you willing to pay to show them an ad?”, AdRoll then cross-references that user’s opaque identifier with the behavioral data they have (i.e. what pages they were looking at on an advertiser’s site) and if there is a match they make a bid which will also include their ad for the page that piqued their interest. If the retailer wins the auction, then their ad is chosen and either rendered in the news feed or on the right hand side on Facebook’s desktop website. Each of these pieces needs to happen in fractions of a second but is still slow enough that rendering ads tends to noticeably be the slowest part of rendering the webpage.

In fact, there was a grand example of retargeting in action while I was researching this blog post. When I started writing this blog post I checked out AdRoll’s web page on how to use their service to retarget ads on Facebook. A few minutes later, this showed up in my news feed.

You can tell if an ad is retargeted on Facebook by hovering with your mouse cursor on the top right of the ad (on the desktop website) and then selecting the options. If the “About This Ad” link takes you somewhere outside Facebook then it is a retargeted ad.

Some ad providers like Quantcast provide an option to opt-out of retargeting for their service while others like AdRoll link to the Network Advertising Initiative (NAI) opt –out tool which provides an option to opt-out of retargeting for a variety of ad providers. Note that this doesn’t prevent you from getting ads and is instead just a signal to advertisers that you’d rather not have your ads personalized.

If you found this blog post informative I've begun a regular series of blog posts intended to answer questions about online advertising on Microsoft properties such as Bing & MSN and on industry trends. Hit me up on Twitter with your questions.

Now Playing: Ice Cube – Hood Mentality

 

Up until a few months ago, the term DevOps was simply another buzzword which filled my Twitter feed that evoked a particular idea but wasn’t really concrete to me. Similar to other buzzwords related to software development such as NoSQL and Agile, it is hard to pin down what the definitive definition of the term is just what it wasn’t. If you aren’t familiar with DevOps, a simple definition is that the goal of DevOps is to address this common problem when building online services

The Big Switch

A couple of months ago, my work group took what many would consider a rather extreme step in eliminating this wall between developers and operations. Specifically, Bing Ads transitioned away from the traditional Microsoft engineering model of having software design engineers (aka developers), software design engineers in test (testers) and service operations (ops) and merged all of these roles into a single engineering role. As it states in the Wikipedia entry for DevOps, the adoption of DevOps was driven by the following trends

1. Use of agile and other development processes and methodologies
2. Demand for an increased rate of production releases from application and business unit stakeholders
3. Wide availability of virtualized and cloud infrastructure from internal and external providers
4. Increased usage of data center automation and configuration management tools

All of these trends already applied to our organization before we made the big switch to merge the three engineering disciplines into a DevOps role. We’d already embraced the Agile development model complete with two to four week sprints, daily scrums, burn-down charts, and senior program managers playing the role of the product owner (although we use the term scenario owner). Given our market position as the underdog to Google in search and advertising, our business leaders always wants to ship more features, more quickly while maintaining high product quality. In addition, there’s a ton of peer pressure for all of us at Microsoft to leverage internal tools Windows Azure and Autopilot for as much of our cloud services needs as possible instead of rolling our own data centers and hardware configurations.

Technically our organization was already committed to DevOps practices before we made the transition that eliminated roles. However the what the organization realized is that a bigger change to the culture was needed for us to get the most value out of these practices. The challenge we faced is that the organizational structure of separate roles for developers, testers and operations tends to create these walls where one role feels their responsibility is for a certain part of the development cycle and then tosses the results of their efforts down stream to the next set of folks in the delivery pipeline. Developers tended to think their job was to write code and quality was the role of testers. Testers felt their role was to create test frameworks and find bugs then deployment was the role of the operations team. The operations team tended to think their role was keeping the live site running without the ability to significantly change how the product was built. No matter how open and collaborative the people are on your team, these strictly defined roles create these walls. My favorite analogy for this situation is like comparing two families who are on a diet trying to lose weight and one of them has fruit, veggies and healthy snacks in the pantry while the other has pop tarts, potato chips, chocolate and ice cream in theirs. No matter how much will power the latter family has, they are more likely to “cheat” on their diet than the first family because they have created an environment that makes it harder for them to do the right thing.

Benefits

The benefits of fully embracing DevOps are fairly self-evident so I won’t spend time on discussing the obvious benefits that have been beaten to death elsewhere. I will talk about the benefits I’ve seen in our specific case of merging the 3 previous engineering roles into a single one. The most significant change is the cultural change towards how we view automation of every step related to deployment and monitoring. It turns out that there is a big difference when approaching a problem from the perspective of taking away people’s jobs (i.e. automating what the operations team does) versus making your team more effective (i.e. reducing the amount of time the engineering team spends on operational tasks that can be automated thus giving us more time to work on features that move the business forward). This has probably the biggest surprise, although obvious in hindsight, as well as the biggest benefit.

We’ve also begun to see faster time to resolve issues from build breaks to features failing in production due to fact that the on-call person (we call them Directly Responsible Individuals or DRIs) is now a full member of the engineering team who is expected to be capable of debugging and fixing issues encountered as part of being on-call. This is an improvement from prior models where the operations team were the primary folks on-call and would tend to pull in the development team as a last resort outside of business hours.  

As a program manager (or product manager if you’re a Silicon Valley company), I find it has made my job easier since I have fewer people to talk to because we’ve consolidated engineering managers. No longer having to talk to an development manager separately from the manager of systems engineers separately from a test manager has made communication far more efficient for me.

Challenges

There are a number of risks with any organization taking the steps that we have at Bing Ads. The biggest risk is definitely attrition especially at a company like Microsoft where these well-defined roles have been a part of the culture for decades and are still part & parcel of how the majority of the company does business. A number of people may feel that this is a bait and switch on their career plans with the new job definitions not aligning with how they saw their roles evolving over time. Others may not mind that as much but may simply feel that their skills may not be as valuable in the new world especially as they now need to learn a set of new skills. I’ve had one simple argument when I’ve met people with this mindset. The first is that DevOps is here to stay. The industry trends that have had more and more companies from Facebook and Amazon to Etsy and Netflix blurring the lines between developers, test engineers and operations staff will not go away. Companies aren’t going to want to start shipping less frequently nor will they want to bring back manual deployment processes instead of automating as much as possible. The skills you learn in a DevOps culture will make you more broadly valuable wherever they find their next role whether it is a traditional specialized engineering structure or in a DevOps based organization.

Other places where we’re still figuring things out are best practices around ownership of testing. We currently try to follow a “you build it, you test it, you deploy it” culture as much as possible although allowing any dev to deploy code has turned out to be bit more challenging than we expected since we had to ensure we do not run afoul of the structures we had in place to stay compliant with various regulations. Testing your own code is one of topics where many in the industry have come out against as being generally a bad idea. I remember arguments from my college classes from software engineering professors about the blind spots developers have about their software requiring the need for dedicated teams to do testing. We do have mitigations in place such as test plan reviews and code reviews to ensure there are alternate pairs of eyes looking at the problem space not just the developer who created the functionality. There is also the school of thought that since the person who wrote the code will likely be the person woken up in the middle of the night if it goes haywire at an inopportune moment, there is a sense of self preservation that will cause more diligence to be applied to the problem than was the case in the previous eras of boxed software which is when most of the anti-developer testing arguments were made.

Further Reading

• DevOps on Wikipedia

• What is DevOps? by Damon Edwards

• Release Engineering at Facebook based on a talk by Chuck Rossi

• Ops, DevOps and PaaS (NoOps) at Netflix by Adrian Cockcroft

 

Now Playing: Eminem – Rap God