How many times has this happened to you? You’re on Facebook, browsing around, minding your own business, and you spy the ‘Trending Articles’ box in your news feed:

Or maybe, while checking up on a friend, you notice on their timeline an article they recently read:

You say, “Hey, that looks interesting,” and you click on the article. Surprise! Instead of the article popping up you get this lovely dialog box:

No article for you! With Facebook’s recent updates, it is now possible to see what your Facebook friends are reading and watching on many websites, including Yahoo, The Washington Post, MS NBC, Social Cam, The Guardian, and Viddy, right on Facebook. However, to actually view any of the things your Facebook friends are reading / watching yourself, many websites currently force you to install their Facebook apps. Once installed, these apps will then post whatever you watch or read on their sites to Facebook! Looking at some reading material or videos that you’re less than proud to say you viewed? To bad! On to your timeline it goes!

Obviously this user experience is less than ideal. I wanted to be able to click on any of these “Open Graph” links on Facebook without having to give anyone read access to my Facebook info, let alone write access! Luckily, I noticed that the urls of these facebook links always seem to contain the url to the actual piece of content they describe, even if they refuse to take you there unless you sign up for an app.

So, naturally, I made a Google Chrome extension that turns every Facebook Open Graph link into an actual link to the article – no signing up for apps required! This extension runs seamlessly in the background – just click any link on Facebook like you normally would, and you’ll skip the app install page and go straight to the link you clicked. Just so no one gets confused, this extension does not prevent Facebook apps you install from posting content to your Facebook feed. It only prevents the dialog box asking you to install the app from appearing.

I think its pretty cool, and if you dislike the many Facebook apps that force themselves down your throat as much as I do, I encourage you to try it out.

I know installing anything that accesses your Facebook introduces some privacy concerns, so I also open sourced the code behind this Chrome extension, so you can take a look and see its not doing anything malicious: https://github.com/jodoglevy/open-graph-redirect. If you have any improvements, feel free to issue a pull request!

So now when I click on an article about parrots on Facebook:

I get an article about parrots, not a sign up page! Huzzah!

Update: This plugin is now available for Firefox as well at https://addons.mozilla.org/en-US/firefox/addon/facebook-opengraph-redirect/.

After more than 24 hours of traveling, I’m finally home. Stepping off the plane at Chiang Kai Shek Int’l Airport in Taiwan, a far ways off from Duke University’s North Carolina campus, I whip out my iPhone. I hit settings -> Wi-Fi -> ON. Immediately I’m greeted by a plethora of free public wi-fi access points. Open Google Currents, and I get updates to Engadget, Gizmodo, Sci-Am, and all the other mainstream news sites I’ve been deprived of for the past 24 hours.

Loading… 1/4 Done… 1/2 Done … Loading… Loading… Any time now…

The connection hangs for more than 10 seconds - well beyond the average tech savvy person’s attention span. Annoyed, I force quit the app and pray my second connection attempt has better luck, and … win, it does. With higher traffic on public wifi on a normal day in Taiwan, one might not be as lucky as I was. Living in Taipei, I have had the opportunity to visit numerous countries in East Asia, all with advanced wireless network infrastructure. Some big cities like Taipei, Beijing, and Singapore have government sponsored free public wireless more or less all throughout the city, but the wireless infrastructure is nowhere nearly designed to handle large public loads – as such wifi becomes slower than 3G in crowded areas. Other places such as Hong Kong, Tokyo, London, and Belgium do not have prevalent free wifi, and, like in the US, most people rely on just 3G, which unfortunately has a bandwidth that doesn’t really cut it either. There are also countries that have limited or non-existent 3G infrastructures, which makes mobile browsing even more difficult on the go.

When I reach home, I settle into my comfy bed, phone in hand. Happy to finally be connected to fast WiFi and not having to deal with deathly slow loading times, I open up a website I browse a lot while in the US… loading…. loading…. laaaaaaaaaaggggg….. done. Wow that took a long time. Look at how much data its loading…no wonder it was taking so long.

Unfortunately, this is an all too common scenario for countless internationals trying to access United States-based mobile sites from outside the US. The experience is just always sub par. Many websites, in their quests for world domination, forget to optimize for the rest of the world!

Luckily, The Chronicle’s mobile site is not one of them. Duke University’s student body is 13% international, and so we make it a priority that the more than 1 in 10 students at Duke who try to view our site from a mobile device from outside the US have a pleasant experience. According to Google Analytics, The Chronicle website gets 4.4M pageviews a year, with 1 in 8 views from a mobile device. 8% of all views are from outside the United States. This amounts to 352,000 pageviews a year from outside the US and almost 600,000 on mobile devices, a significant number representing an audience that we cannot simply ignore as site developers. Thus, we have to create the best experience for the market outside the United States, including those that prefers to view our site on the go. But maintaining an internationally-optimized mobile website is not an easy task, and good engineering practices on the front and back end are paramount to sustaining a good mobile version.

Graphic: Map visualization of the number of visits to dukechronicle.com

Figure: Number of visits to dukechronicle.com by country

When I talk about “mobile”, the scope is not restricted to smartphones - tablets and netbooks have also gained in popularity - perhaps a better descriptor would be “non-traditional computers”. Due to varying resolutions, operating systems, touch capabilities, connection speeds, and processing power of devices nowadays, we’re starting to see a little repeat in history – developers are encountering the same problems as back when LCD monitors revolutionized computer displays, when desktop resolutions increased. Remember when we used to see “This site is best viewed in 800 x 600 resolution” on web pages? Now we’re seeing “This site is best viewed with iPhone”. On top of that, the physical distance between the United States and the rest of the world induces a higher latency for each packet, some countries have government-imposed firewalls that throttles transmissions, and many countries’ wireless traffic density is much larger than that of the United States. Put that all together and it means some sites should also be saying “This site is best viewed from the United States!”

Obviously, this is not how mobile should be done. A recent survey found that 57% of mobile web users had a problem in the past year when accessing a website [1]. It also found that 46% of mobile web users are unlikely to return to a website that they had trouble accessing from their phone, and 57 percent are unlikely to recommend the site to others. In addition, 34% said they’d likely visit a competitor’s mobile website instead. Simply put, you will lose users if you don’t provide them a great mobile experience. And if they’re outside the United States, it will be harder to provide them a great mobile experience.

Today I’ll be discussing good practices for international-optimized mobile web development, and how Chronline has approached the many issues surrounding mobile. Even if you are not trying to make your site optimized for the whole world, all these guidelines still apply to mobile sites in general, even if they are more important outside of the United States. There’s a few obvious guidelines every mobile site should follow, even if the correct implementation for them isn’t all that obvious. In terms of the guidelines, mobile websites should:

Minimize data transfer.

It is undisputed that the cost of wireless transmission is extremely high nowadays. High packet loss and bad connections result in slow page loads and higher device power consumption, due to TCP standard retransmit and waiting. Both of these things users HATE with a passion. In fact, a survey from market research group TNS of 7,000 mobile users in 15 countries (not just the United States!) found that over 75% of people placed better battery life as the main feature they want from a future smartphone [2]. Another survey found that 71% of users expect websites to load as quickly or faster on their mobile phone compared to the computer they use at home [1]. Now think about that. If your website that users see from their mobile device is bloated and takes 5 seconds to download even with good 3G connection, you’re going to have some unhappy campers. And if it takes 5 seconds to load in the US, it’ll take double that outside the US.

Load quickly.

Mobile users are most likely on the go, and the last thing that they would want is to wait for Windows 7 to boot. It would be wise for your site to be slightly faster than that. While this is similar to the minimize data transfer guideline, it is independently important as well. The data that is sent to the mobile device might be small, but if the source code renders a 3D environment in WebGL behind the scenes, your site might suffer from limited viewership. In addition, if you expect your site to be used all over the world, you should expect to geolocate your content all over the world.

Functionally similar to the regular site.

For Chronline mobile, the last thing we want a user to say is, “I could read about derp on the regular site, but how do I on mobile?” Lack of functionality discourages the use of the mobile version, and most people would rather save themselves the trouble. Users only have so much time when they’re on the go, and they will choose to look at the sites most functional as compared to their non-mobile counterparts. Make sure you’re one of those sites.

Opt out.

A lot of times minimizing data transfer means minimizing site design and/or functionality. Some people just might not be able to handle minimalist sites for one reason or another, or might want to use some discrete functionality only on your regular site. Let them opt-out at their own volition and view the regular site from their mobile device. Don’t force the hand that feeds you.

Design with every type of mobile device in mind

As with a regular website, your site should look and function great on different screen sizes, resolutions, browsers, and operating systems. And if you want a “worldly” mobile site, include operating systems popular in other parts of the world in that assessment, like Symbian and Bada.

With these guidelines in mind, let’s take a look at the network footprint of http://www.dukechronicle.com/m/. I pulled up Chrome Dev tools starting from a cold cache, and in Figure 1, we see that the front page of the mobile site downloads only 100kb of compressed data – nice and quick. JQuery and JQuery Mobile libraries (which I’ll talk about more later) take up 60kb, the site’s css sheets takes up about 12kb, and our own mobile javascript files take up 19kb. The ajax request that loads the actual content of the homepage is 3kb, which fits comfortably into one IP packet. You might think, “well duh, all you have is text, what about thumbnails?” We considered the pros and cons of having thumbnails with each list item. At the end of the day, we are looking to reduce data transfer, and thumbnails would quickly rack up bandwidth. Would thumbnails make the site more appealing? Given the resolutions and sizes of smartphone screens, would users even be able to see the image clearly enough to have it be worthwhile? Are they going to read or not read an article based on the tiny image associated with it? We thought no, but obviously you can weigh in your own opinion here.

Figure 1: Chrome Dev tools analysis of dukechronicle.com/m/ - part 1

Looking at Figure 2, from the moment I hit enter on the address bar, it takes approximately 2 seconds for the site to fully load in Taiwan. On an actual 3G benchmark from the US, loading takes approximately the same time. These numbers clearly decrease with caching of JQuery, JQuery mobile, and our site’s mobile javascript and css, which account for most of the data transfer. Moreover, page transitions are smooth - once again, articles take about 2 seconds to load on average. One might think of ways to reduce page loading ti mes, and the first idea that pops into mind is prefetching. However, before you happily code away, let us be reminded of the first good practice - minimizing data transfer. Using similar logic from before, one would more or less conclude that prefetching data to smoothen user experience is unfeasible.

Figure 2: Chrome Dev tools analysis of dukechronicle.com/m/ - part 2

At the same time, we must not forget that JS and CSS are cached on mobile. This reduces the load times even more. When the cache is warmed, the main page and article pages now load in just over a second in Taiwan. Just to compare another benchmark, on wifi in the US, the mobile site loads in approximately 800ms cold cache, and even less than that cached.

Finally, Chronline mobile has organized subsections similar to that of the original site, a search bar for any specific searches, and finally, the almighty op-out button on the bottom right corner, which Balrog, “you shall not press!”

Figure 3: dukechronicle.com/m/

So how do we maintain versions of the mobile site to support all the different mobile platforms? The trigger happy developer might immediately jump into hardcoding cases for each type of mobile device, but asides from that being bad coding practice, it’s also quite a pain to have to test all those different Android models and such (yay fragmentation!). Which calls for a layer of abstraction… Enter: JQuery Mobile.

JQuery Mobile (JQM) is a cross-platform web framework that supports virtually any smartphone use case seamlessly. Its syntax is extremely clean (you’re writing minimal HTML) - just by giving tags certain attributes, JQuery Mobile instructs the mobile browser to render the window to the developer’s specifications. From what I’ve witnessed, it works beautifully on both Android and iPhone, and it claims to support much, much more. JQM comes with a whole bunch of presets that a web developer would want - lists, buttons, forms, toolbars…etc. If you don’t like the default themes? Slap on a custom style sheet and you’re done. Just like that, the first iteration of Chron Mobile was made in a single hour, and it worked on every mobile OS.

So what does JQM mean for mobile web dev? If all mobile sites were to use JQM, mobile users will not have to download each site’s ginormous custom JS file because lots of mobile sites would all share the same javascript (read: smaller data transfer). For the user, pages will be clean, accessible, and more or less display-glitch free. Smaller pages mean higher portability, increasing the chances that a page would load fully in low connectivity areas. For the developer, easy learning curve, and countless hours saved from having to deal with cross platform display issues. JQM protects us from reliving another IE 6/7 compatibility nightmare, this time in terms of mobile.

Of course, at the end of a day, Chronline Mobile is nowhere near perfect, and there’s much more we can work on to optimize the site. For example, from the benchmarks I did from Taiwan, my user experience is 3 times slower than in the US. Just by physical separation distance, the speed of light imposes an unavoidable 100ms ping between Taiwan and US servers (almost halfway around the world). However, if we use a content delivery network such as Amazon CloudFront to geolocate our static content, the many Duke students and Duke fans in all corners of the world can access our mobile site almost as quickly as those in the US. We recently packaged our mobile CSS files to do this, and have seen a whopping 33% reduction in overall load times in Taiwan, and an 11% reduction in the United States, due to faster CSS download (Figure 4 shows before CDN in Taiwan). We still need to do this with our mobile javascript, however, but jQuery hosts JQuery and jQuery Mobile on their own CDNs, so they experience geolocation benefits. We should also take this a step further and combine our various mobile Javascript files, to minimize the number of HTTP requests (for speed and battery life benefits) the mobile phone has to make to view our site.

Figure 4: Latency from Taiwan pre-CDN

In summary, when developing for the internationally-optimized mobile web, one cannot think in terms of traditional websites. One must remember that mobile devices have limited processing power and screen space, and more importantly, expensive network transactions. And users outside the United States have a whole bunch of other network-related issues to deal with. Mobile phones and the global web have certain limitations, and we must develop with those limitations in mind in order to provide the best experience possible.

Sources

1. http://www.gomez.com/resources/whitepapers/survey-report-what-users-want-from-mobile/
2. http://www.informationweek.com/news/171100095

Update: Since the writing of this post, the design and implementation of the Chronicle’s mobile site has changed. It is currently live at http://m.dukechronicle.com.

The below is an essay about Google’s anticompetitive practices in search that I wrote last semester for a computer science class. Given the recent news about Google’s actions in taking payment for better placement in Google ‘Hotel Finder’ and ‘Flight Search’ results, I felt it important to highlight some of Google’s other recent anticompetitive practices, as well as compare how Google’s current practices and ideologies stack up against those of the Google of yester-year.

Now don’t get me wrong, I love Google. I use them for search and email everyday, and I have an Android-powered smartphone. But I think it is important that we look long and hard at what Google’s been up to recently, and, if we find that Google is turning too “evil,” we help push the company back towards the right path. Obviously only one side of the argument is presented here, but I think its a compelling one.

Introduction

Fifteen years ago, google was just a number. A number so large, in fact, that no one ever used it. Nothing of such gargantuan size existed in the real world, so google remained an abstract concept. It was never used in math. It was never used in chemistry. It was never used in economics. It was never used, in the real world, by anyone, for anything. If google had disappeared from the Earth, never to be seen again, life would have gone on just as it had before, uninterrupted.

Fast forward to today. Google is no longer just an abstract number, but rather the key interface to our knowledge base, our society, and even our own identity. Anyone with a computer interacts with Google on a daily basis. We use it for answering questions, sending emails, finding directions, editing documents, watching videos, maintaining schedules, and even translating languages. Google is, and for the foreseeable future will be, our primary, trusted, interface to information.

Because of this, if there’s one company that the phrase “with great power comes great responsibility,” applies to, it’s Google. With the amount of power Google has over people’s access to information, what would happen if Google suddenly betrayed our trust, and put its preferences for search results over users’ preferences? What if Google decided to display certain websites more prominently than the websites most relevant and useful to users’ searches? What if Google simply stopped displaying websites in search results that it felt were competing with it’s own products? Our society’s access to the best and most pertinent information would be diminished, and we wouldn’t even know it, because we trust Google to provide the best results. Its a good thing that Google is impartial in its search results, or else searching Google could become equivalent to asking a Nissan salesman for information on cars, or even Google’s Android marketers for information on mobile phones, without the knowledge that these information sources are biased.

Except that Google is not impartial anymore. It has abandoned the core mantra of ‘users over profits’ on which the company was built and which allowed Google to rise to such epic ubiquity. In its place are search results that favor Google’s own services over those most useful and relevant to users. The free and open access to information we as Google users expect from Google is being constricted, as are the many businesses that compete with any of the Google services favored by Google’s search results. Google is operating a monopoly, and its anticompetitive business practices must be regulated.

The Bullying of Yelp

One can clearly see Google’s anticompetitive practices by following its dealings with Yelp over recent years. Yelp, which allows consumers to find businesses in their area, competes closely with the Google service ‘Google Local.’ From 2005 to 2007, Google licensed Yelp’s content to be used by Google Local, but in 2010 Google “began incorporating the content that it indexed from its competitors into Google Local without permission” (6). You see, in order for a site’s pages to appear in Google’s search results, the site must let Google “see” the content on every page they want to appear in search results, so that Google can properly place the page. So, Yelp, like most sites, let Google see all of its content. But in addition to using Yelp’s content to learn where to place Yelp in search results, Google also directly incorporated Yelp’s content into its own competing product, Google Local, without Yelp’s permission. While Google had previously shown it knew it needed a license to use Yelp’s content, “it was now using [the content] without permission to prop up its own, less effective product” (6). When Yelp protested Google’s actions, Google “informed [Yelp] that it would cease the practice only if [Yelp] agreed to be removed from Google’s web search” (6). On the surface, this might seem like a valid solution. As one site specializing in “local search expertise” notes, all Yelp has to do to stop Google from getting its content is change a file on yelp.com to “disallow Googlebot from crawling” its site and seeing its content, since “no one in [Google’s headquarters in] Mountain View is forcing Yelp” to let Google see its pages (9). But Yelp, like most sites, receives a large amount of its traffic from the Google search engine, and so cannot afford to not show up in Google’s search results. Basically, because of “Google’s dominant position in the [search] market,” Yelp would have immediately lost a large portion of its traffic if Google stopped showing its pages. Yelp had no choice but to play by Google’s rules, and continue giving Google access to its content (6). But the FTC ruled Google’s actions as anticompetitive, and told Google to “cease misappropriating content from its competitors” (6). While Google publicly agreed, a week later, Google Local “was continuing to use content from services like Yelp to power its own local business search product,” against the FTC’s wishes (6).

In addition to the above monopolistic practices, Yelp has had additional trouble competing with Google Local because Google’s search results specifically place Google Local “in the most prominent position” on the search results page, “regardless of whether the [Google search] algorithm has actually determined that [Google Local] has the most relevant content” (6). Even if Yelp results are more relevant or useful to users, Google will still show the Google Local results at the top of search results, where the most relevant results usually go. In this way Google is “steering users toward its own products” rather than towards the best sources of information (8). As Jeremy Stoppelman, cofounder and CEO of Yelp, notes, “when Google artificially promotes its own properties regardless of merit,” Google is “not helping consumers get to the best information” but is rather just using its dominant position to “generat[e] more revenue” (6). In fact, Google’s practices have been so unfair that Stoppelman wonders “if [he] would have been able to start Yelp today given Google’s recent actions” (6).

Anticompetitive Practices in Search

Its hard to argue the above practices aren’t anticompetitive, but if its just local search services being affected, then its not so bad right? Problem is, local search is just the tip of the iceberg of unethical practices currently employed by Google. For example, a group of Facebook and Twitter employees recently published a website dedicated to highlighting Google’s unfair search bias towards its own social networking service, Google+, over more popular and widely used social networking services like Twitter or Facebook (7). As the website notes, when one searches ‘cooking’ on the Internet, Google displays renowned chef Jamie Oliver as a relevant search result. But “rather than linking to Jamie’s Twitter profile, which is updated daily, Google links to [Jamie’s] Google+ profile, which was last updated nearly two months ago” (7). The website also describes how this is a purposeful decision made by Google, shown by the fact that if ones searches Google for ‘Jamie Oliver’ directly, “his Twitter profile is the first social result that appears,” and his Google+ profile “doesn’t even appear on the first page of results” because it is not useful or relevant to users (7). What this shows is that Google purposefully promotes Google+ over more relevant social results in some searches, even though the Google search algorithm knows these Google+ results are less relevant than those from Twitter or Facebook. And just like in the Yelp case, Google displays these Google+ results, regardless of relevance, in a very prominent position on the search results page.

These are not ‘slip-ups’ or independent events. They are part of Google’s current strategy to use its monopoly in search to dominate other industries by promoting its own services in “ways that suggest to consumers that they are natural search results, rather than links to Google sites in which Google has a direct economic interest” (8). In fact, the Fair Search Coalition recently sent a 44-page paper to all 50 U.S. Attorney Generals, entitled “Google’s Transformation From Gateway To Gatekeeper: How Google’s Exclusionary And Anticompetitive Conduct Restricts Innovation And Deceives Consumers,” which details many ways in which Google is misusing consumer trust in advertising, display, search, and mobile, including by “manipulating its search algorithm to exclude or penalize competing sites, effectively ‘disappearing’ them from the Internet” and “stealing content developed by other websites…without permission and displaying that content on its own pages, sometimes even without attribution” (8). As Stoppelman noted in his 2011 submission to the U.S. Senate Subcommittee on Antitrust, Competition, Policy, and Consumer Rights, “allowing a search engine with monopoly market share to exploit and extend its dominance hampers entrepreneurial activity,” because “when one company controls the market, it ultimately controls consumer choice” (6). Stoppelman’s words could not be clearer. It is imperative that Google become a neutral party in search, for the sake of both businesses and consumers.

The Hypocrisy of a Search Giant

Competitors aside, even Google’s founders would say Google is being anticompetitive if they stood by the standards established when they started the company. In 1998, the same year Google was incorporated, it’s founders published a paper advocating against “advertising funded search engines” – those that gain a profit by advertising external companies or the search engine company itself – because they are “inherently biased towards the advertisers and away from the needs of the consumers” (3). While Google eventually went on to use advertising as its primary mode of revenue, it’s founders tried to ensure that Google would always put search neutrality ahead of profits by incorporating certain beliefs into Google’s official corporate philosophy right from the beginning. Still written in the web giant’s manifest today, the “core principles that drive [Google’s] actions” are always putting the user first, written as “focus on the user and all else will follow,” and being socially and ethically responsible, written as “you can make money without doing evil” (1). In fact, when Google’s 23rd hire, senior engineer Paul Buchheit came up with the “don’t be evil” phrase (a shortened version of the “you can make money without doing evil” mantra) as “a bit of a jab at a lot of other companies, especially [Google’s] competitors” who were “exploiting the[ir] users,” it was quickly written down on every white board used throughout the company, so every employee would always remember that Google was there to serve its users, not the other way around (2).

Google’s founders believed that by instilling a “Don’t be evil” culture, the corporation would be able to establish a baseline for honest decision-making that disassociates Google from any cheating and corner-cutting. This, the founders believed, would enhance the image, trust, and usefulness of the corporation in the eyes of its users (3). And this philosophy has served Google very well – until recently when they stopped abiding by it. In order to make sure searches were as relevant and accurate as possible, from its inception Google refused to sell ad placement in its search results, and rather only showed ads on the side of the page, under an ‘Ads by Google’ header (1). Similarly, Google wouldn’t “allow people to pay for a higher ranking” in the search results, keeping results as accurate and unbiased as possible (1). By employing these user-over-profits policies, Google was able to provide the most useful, most accurate, and most relevant search results. For these reasons, by 2004, Google served more than 84% of all search requests on the Internet (4), and today represents a 66% market share of all Internet search – with its closest search competitor at only 15% market share (5).

But that was then. Google used ethical search results to gain users’ trust, and now that it has that trust, it is abusing it. Google’s recent operations are in clear violation of its own core mantras – with current CEO Larry Page, one of the founders of Google, in clear violation of his own ethical standards. Yes, Google may not sell ad placement in its search results, or allow people to pay for a higher ranking in search results, but as discussed above, it does present its own services in response to searches in prominent positions at the top of search results – regardless of their relevance or usefulness as determined by the Google search algorithm. So while Google does not allow any other companies to manipulate the Google search results to their favor, Google is allowing its own services to manipulate the search results to their favor – a clear monopolistic practice, and an especially dangerous one when Google is so widely trusted by users and its services extend to so many business segments.

Conclusion

Google’s founders had a dream – to create a search engine that provided the best results and put user’s interests over those of shareholders. And Google built an empire on that dream. But recent anticompetitive practices by Google, including its prioritization of search results of Google Local over Yelp and Google+ over Facebook and Twitter, break both Google’s core mantra as well as US law. Google must be held accountable to present the most relevant search results over those results which benefit its own services, in order to keep our access to information, and other businesses’ ability to compete, safe. The search for search neutrality must continue.

Works Cited

1. “Corporate Information – Company Overview”. Google. http://www.google.com/corporate/
2. Livingston, Jessica (2007). Founders at Work: Stories of Startups’ Early Days. Apress Publishing.
3. Brin, Sergey; Page, Lawrence. “The Anatomy of a Large-Scale Hypertextual Web Search Engine”. Stanford InfoLab. http://infolab.stanford.edu/~backrub/google.html
4. “Can Google Go Glossy?”. Businessweek. http://www.businessweek.com/magazine/content/05_50/b3963130.htm. (2005).
5. “comScore Releases December 2011 U.S. Search Engine Rankings”. ComScore.com. http://www.comscore.com/Insights/Press_Releases/2012/1/comScore_Releases_December_2011_U.S._Search_Engine_Rankings
6. “The Power of Google: Serving Consumers or Threatening Competition?” Stoppelman, Jeremy. http://www.judiciary.senate.gov/pdf/11-9-21StoppelmanTestimony.pdf
7. “Focus on the User”. FocusOnTheUser.com. http://www.focusontheuser.com
8. “Google’s Transformation From Gateway To Gatekeeper: How Google’s Exclusionary And Anticompetitive Conduct Restricts Innovation And Deceives Consumers.” The Fair Search Coalition. http://www.fairsearch.org/wp-content/uploads/2011/10/Googles-Transformation-from-Gateway-to-Gatekeeper.pdf
9. “Yelp vs Google in Congress”. MIHMORANDUM. 11/21/2011. http://www.davidmihm.com/blog/google/yelp-vs-google-congress/

I recently had the pleasure to graduate from Duke University. At one of my graduation ceremonies, I ran into the parent of a friend who asked me what I was doing after graduation. This parent was no tech-fanatic, so I summarized my cloud and virtualization automation gig at Microsoft by saying I worked on a product that allowed operating systems to run other operating systems as applications, so one could run Linux on their Windows machine, Windows on their Mac machine, etc. Of course, this is only a tiny sliver of the true description and value of virtualization to computer systems, but to a parent who probably had never even heard of BootCamp or VMWare, let alone hypervisors or virtual machines, it seemed like an acceptable summary.

Of course, the parent said how cool that was and how proud they were of me, the same thing all parents say to their children’s friends. But what was really interesting was what she said next. “I could actually really use that software,” she said. “I run a business on Windows machines and I just get so much spam! I’d love to switch to Macs without having to buy new computers for my business.” I kind of just scratched my head for a second. I tried to explain to her the fallacy of her argument – that email is a completely separate module from a computer operating system, and no matter what OS she used, spammers could still send email to her email address. Sure a different email client might filter the spam out better or automatically delete it, but that has nothing to do with the operating system. There’s a variety of clients for Mac, Windows, and in the browser that do a good job filtering spam. My comments seemed to go in one ear and out the other. “But Macs are a closed system. They can’t get viruses or malware or spam,” she said. You can’t make this stuff up…

Now before I go on, I’d like to note that Macs definitely have their uses. For one, their hardware is great. They’re built on top of Unix, so they have a great terminal which makes development easy. They work great as general purpose computers. And they look pretty. But Macs are not a safe-haven when it comes to security, and the fact that Apple advertises them as such actually makes Mac users even less secure because they have a false sense of security. Because of this false sense of security, many Mac users click on questionable links on the internet, install sketchy programs, and don’t install antivirus software. Apple has always stated things to this effect, such as this snippet from their website:

with virtually no effort on your part, OS X defends against viruses and other malicious applications, or malware

Literally making users believe they have to put zero effort or thought into defending themselves on a Mac! If this were the case, would security vendor ESET have found that Mac users lost more money on average than PC owners did in phishing attacks? Would the recent Flashback Trojan have infected 1% of all Macs in the world – a larger percent infection than the “most dangerous malware to attack Windows-based computers” ever, Conficker? Would Mac vulnerabilities in WebKit, Samba, DNS, MDNS, Apache, and Java have gone unpatched by Apple for many months after all other platforms had fixed the same vulnerabilities in their software? Despite all this, another ESET survey found that more than 50% of computer users thought of PCs as very or extremely vulnerable, whereas only 20% of computer users thought of Mac that way. If there’s one thing Apple does better than making computers, its marketing!

Until recently, Windows usage was much greater than Mac usage, and for this reason virus makers always targeted Windows PCs in their attacks, because they simply had more users to exploit. Macs were more secure than PCs, not due to the operating system’s security, but to the lack of interest by malicious parties. Today, however, Mac market share has grown enough that malicious parties are starting to target the platform, and while PC users know they must be mindful of what they click on and download to their computer, because they have had to be mindful in the past, Mac users have been lulled into the naivety that they are safe. The problem is even worse considering social engineering attacks like phishing scams work in the browser and so are cross platform, and are almost impossible for OS protection software to prevent due to their social nature. For example, if someone emails you saying they are Facebook and they need your password emailed to them to confirm you are who you say you are because someone recently tried to hack in to your account, is anything built into the computer operating system going to delete this email or warn you its fishy? Nope. Maybe, and only maybe, an email client will. But social engineering attacks happen in so many ways that the only way to truly defend against all of them is to recognize that the Internet is a dangerous place, and to take things with a grain of salt before you click on or install them. The best defense one can have in the world of ubiquitous Internet isn’t something on the computer, but something in the brain.

Think of it this way – if you were wearing an impenetrable suit of armor from head to toe, why not run into the middle of a battlefield? The problem is, if that armor isn’t as safe as its maker says, you’d actually be putting yourself in way more danger than even someone wearing no armor, because you are willing to put yourself into much more dangerous situations with no thoughts of the consequences, whereas the defenseless person knows not to do that. Where this metaphor isn’t completely sound is that, in sword fighting, it is a no brainer to all of us that you should take precaution no matter what defenses you’re wearing. But this isn’t the case in computer security. Staying away from an enemy with a sword is a much more straightforward “instinct” than the lessons people need to learn about defending themselves on the Internet, where malicious attacks are disguised or socially engineered. That’s the problem with Mac security. It is good, just as Windows security is good, just as Linux security is good. But Mac security is no where near as good as its marketing presents, and it never will be, because no amount of secure software can replace the basic understanding that one must be alert to danger on the Internet at all times.

So next time you log on to your computer, regardless of what OS you use, or how secure you think it is, make sure to proceed with caution.

I’d always found foursquare a bit peculiar. What could possibly compel a person to get up early after a Friday night of partying and hurry to the other side of campus to click a button on their phone and ‘check in’? Why were foursquare locations popping up for the most arbitrary of things…Classroom 6 in Old Chem, a local frat house, or even my dorm’s 3rd floor bathroom. I noticed that foursquare was gaining popularity and I really couldn’t figure out why. Yes, the service lets you tell everyone where you are every second of every day, and yes, it lets you brag to your other foursquare pals when you become the ‘mayor’ of a location both of you were fighting over – whether it be the Duke Chapel or the dorm bathroom. To me, foursquare was a stupid game - one that I thought could easily be cheated.

I originally had no intention of hacking foursquare - I just wondered whether I could do it. I knew that HTTP had no inherit location header that foursquare could take advantage of, so they would have to get the user’s location one layer up – the API level. And I knew foursquare had a public api. This lead me to the hypothesis that I could probably pass false GPS coordinates to the foursquare API, and check in to any location near that location. I did a quick google search to confirm. It turned out to be even easier than that. According to KrazyDad in his blog post Mayor of the North Pole, you didn’t even need to pass coordinates to foursquare’s api – you simply pass a venue id (a location’s id) to the check-in api url, and foursquare will credit you as having checked in and been there.

I now knew, in theory that it could be done. But I wanted to actually get a script working that could do it. Not just a script that could commit check-in fraud, but one that could commit mayorship fraud. A hop, skip, and a jump later, this PHP script was born. Feel free to test the script yourself…all it requires is a PHP server (with curl installed) and a browser window. The venue id of a location can be found by going to the locations page on foursquare. For example, the White House’s venue id is 3945, which I found via its foursquare url: https://foursquare.com/venue/3945. I will probably be publishing an entirely javascript version soon so anyone with a web browser can run the script.

In under 40 lines of code, I had created a script that would, every day, automatically check me in to a number of locations, eventually taking the mayorship of those locations. Further testing of the script showed that you can check in to any location as long as it is somewhat near the last location you checked in to. Checking into a bar in Durham, NC and then an hour later to the White House in Washington, DC. worked. Rome and then Washington, DC in an hour didn’t. Presumably, if you wanted to check in to many places throughout the country, you just create a ‘check in path’ from the first check in location you want to the last, in geospatial order.

Yes there were fraud detectors, but I got around them. Like I said above, check ins too far away from one another over too short of a time period would be seen as invalid. Checking in to a location more than once a day was seen as invalid. But with patience, over the course of one or two months (depending on the locations popularity) this script would make me the mayor of that location, with no effort on my part, no matter where in the world I am, and keep that mayorship for however long I wanted it.

Wanting to expose Foursquare’s flaws to its many worshipers on campus, I put the venue ids of Duke’s top locations into my script – the Duke Chapel, the Duke Gardens, Cameron Indoor Stadium, and even Shooters. Within a month I owned them all. People came up to me, and said, how did you do that? Ironically, stealing away foursquare mayorships was a great way to meet girls – they were the vast majority of foursquare users at Duke. Not wanting to offend any lovely ladies, I didn’t tell them I was hacking. The free wings Hooters offered to foursquare users every fourth check-in also encouraged me to keep my mouth shut.

One day, Duke’s own CS professor, Owen Astrachan, stopped me in the halls and asked me how I’d managed to steal the mayorship of his favorite on campus eatery while he and another CS professor had been constantly battling back and forth for the mayorship for over 2 months. Surprised the CS professors had not seen the fault in the service they used so frequently, I spilled the beans to him. He laughed, asked me some questions, and walked away. Months later, I heard Duke foursquare hacker ‘Joe L’ was part of the course topic ‘Ethical Hacking’ in Astrachan’s 400 person intro CS course: ‘Technical and Social Analysis of Information Science’. Then another 2 weeks went by. I heard from a friend on the ‘foursquare committee’ (a group that promotes foursquare’s use on campus at Duke) that the group was trying to figure out how to stop the infamous Joe L from basically halting foursquare’s use at Duke. The end of the month came around, and I went to a Chronicle event. At the event, I was approached by my boss’s boss’s boss, a very angry, 90-pound girl, who had just learned I had stolen all of her Chronicle mayorships by cheating. This girl had plenty of reason to be upset – she came to the office every day to make sure the Chronicle paper came out on time in top quality. She wanted recognition for that, which she sought by being the mayor of the Chronicle offices on foursquare. My script had stolen that from her. Like this girl, many people became aware of how I was doing what I was doing, and got a little angry. Wanting to retain my job at the Chronicle, wanting to cool people down, and feeling I had made my point, I deleted all the Duke locations from my script. Slowly, my name faded from the mayorship position on over 30 Duke locations.

But there was one location for which I kept the script running – the White House. I wanted to prove this script could truly disrupt foursquare, so I went after one of the most checked-in to places in the country – the president’s pad. And after about 2 months, I was unrightfully crowned mayor of the White House…a position almost as prestigious as President. Two days after becoming mayor, I was banned from foursquare, after almost 5 months of abusing the system:

Hi there,

Our system has flagged your account due to frequent check-ins when you’re not actually at a place. While we love that you’re an enthusiastic user, it’s important to follow our guidelines in order to ensure the best experience for everyone! Please only check in when you’re spending time somewhere and your check-ins will be legitimate.

Thanks! Parker

I don’t think its a coincidence that I was banned right after becoming the mayor of such a high profile location. My best guess is the former mayor of the white house reported me, as I had gone so long without getting caught I doubt I was automatically ‘flagged’ by the system. This leads me to believe that the fraud detection system at foursquare is not very good – even if they did automatically catch me after 5 months of abusing the system, 5 months is a long time. Even after the ban, there was nothing stopping me from creating a new account, putting its credentials into my script, and doing it all again.

I would love to provide foursquare screenshots of all this, but when I was banned my account was deleted so I can’t take any screenshots of it. But I do have some twitter posts from that time.

A few weeks ago in the middle of the night I received a call from Pennsylvania. ‘Are you the dude who hacked foursquare and ran ****?’ the drunk person on the other end of the phone said. He proceeded to call me the next Mark Zuckerberg because of my ability to do this. While I liked the compliment, I was getting way more credit than I deserve. Foursquare makes it very easy to abuse their service, and actually have no technological way to prevent location fraud if they want to use a public API. People don’t know just how easy it is to abuse the service. So I wrote this post. People need to know its not worth it to wake up early and run across campus to retain your mayorship, when a kid anywhere in the world can use a script like mine to beat you while he sleeps. Merchants need to know that they shouldn’t offer discounts or incentives to people who check-in, or their mayors, because a script like mine can get any number of abusers free stuff whenever they want, at any of the places that give foursquare discounts.

Many people will probably dislike that I am providing a means for others to cheat. That is true, I am. But really what I am doing is providing full disclosure to a system that does not work. A system consumers, merchants, and VCs should not get over invested in because of how easy it is to manipulate. Feel free to keep using foursquare…but do so knowing it is a game with many loopholes.

Update: Since foursquare implemented OAuth, the script linked to in this post no longer works. I was able to get around this and automate checkins with a new script, but that’s a story for another day…