Published August 16, 2013
Note from Huw Morgan, VP Research Strategy, Info-Tech Research: James McCloskey, Info-Tech's Senior Consulting Analyst on Security projects, is currently working on a Guided Implementation Blueprint called "Secure your Critical Systems and Intellectual Property Against APT's (Advanced Persistent Threats)". If you are interested in participating in this research (e.g. sharing experiences, sharing best practices), please contact me at hmorgan@infotech.com.
Note from Mark Anderson: Something interesting has been happening in the world of cyber attacks, and very few outside the security establishment have noticed it – yet – although it involves everyone reading this Special Letter.
Not long ago, security officials reported that hackers working for the Iranian government had mounted a series of extensive DDoS (distributed denial of service) attacks on Western banks. The motivation seemed clear: a response to sanctions against Iran over its ongoing nuclear program; and the results, to the public at least, predictable: a hassle.
DDoS attacks, after all, were historically considered the sledge hammer in a mostly surgical hacker toolset. Using botnets to array hundreds, or thousands, of servers to send simultaneous requests to a single Web server usually had the intended function: it overwhelmed the server, or at least made it difficult-to-impossible for the victim company's customers to access its services online. As time progressed, companies have found increasingly better ways to deal with these problems, and while they remain a threat, they have dropped down the ladder of worry.
But that was then.
Since around the time of the Iranian attacks, a new concern has emerged among those who make a living studying cyber crime and espionage: What if DDoS attacks are just a part of larger attacks?
Today, one level of the state of the art in cyber offense is to use DDoS's as both a distraction and a way of softening a target's defenses, so that while the victim company is busy dealing with the massive onslaught of fake Web requests, one or more additional teams are working on breaking into the company's network and stealing secrets.
Indeed, such multisource attacks are now becoming common, and the use of DDoS's as just one of the tools in a single attack is no longer in doubt as a technique. As if that were not enough to worry about, DDoS's have another benefit to offer would-be cyber attackers: they can put so much stress on the applications layer that integrity is compromised and new entry points for the bad guys magically appear.
In this week's Special Letter, longtime member and serial Valley entrepreneur Kevin Surace describes this problem in clear detail and offers a much-needed new solution set. The good news: that software you bought that beefs up firewalls and known virus attacks is doing pretty well on those. The bad news: the new attacks will go right around your defenses.
As I write, our INVNT/IP site – specifically focused on stopping nation-sponsored theft of IP, including by cyber theft – has been under DDoS attack for 24 hours.
Thanks to prudent planning, there isn't any private data there to be stolen. And we were careful as we began this work to set this server up separate from all our other servers, physically and technically.
Thanks to our terrific CTO, the site has not gone down or been inaccessible to our users since the onslaught began. But thanks to Kevin Surace, we're now aware that the real attack may be happening on a different level.
I know that our members will want to learn about, understand, and be prepared for this new type of attack before it comes to them. – mra.
-----
All Your Data is Being Stolen As You Read This:
The New Security Threat We All Face
By Kevin Surace
Our data is being stolen? Yes, it is. Really. I have seen it first hand (not your data... but the activities). And I don't have a long security-space background. Your apps, not your networks, have huge vulnerabilities, especially when put into stressed conditions. And the bad guys know this and are now exploiting it daily.
We have heard Mark Anderson talk brilliantly about China's theft of intellectual property. (Watch his video at Strategic News Service and Mark Anderson on China Intellectual Property Theft on "Economic CyberWar and the New Security Mandate.") It's striking. It's shocking. He is right: this is possibly the greatest transfer of value in the history of human civilization.
There was a great panel and working session on this very topic at the 2013 Future in Review Conference. Even at the recent RSA conference, app performance vulnerabilities was a featured topic.
And we have heard about bad guys – or supposed 'ok' guys, like Anonymous – playing havoc with apps and sites. And we have spent untold billions on network security. Yet somehow the data gets out, at an even faster rate today than a year ago. And the worst thing is you often don't know it was taken.
There is some good news here, because we now have ways to close up these holes in all your sites and apps. But they are all open today as you read this.
To some extent, this article describes an awakening I had about where many of the issues are, how they are being exploited, and moreover, why it isn't talked about a lot. At least until now. It's great that we all know there are app layer security issues, but we need to have solutions to close them – and then act.
A Bit of Background
My story starts with a friend of mine, Frank Cohen, a masterful engineer and product guy who was side by side with Peter Norton at the very beginnings of security and virus awareness. The concept of viruses was so foreign... they nearly (but didn't) had to invent some to prove their existence to the world. Peter Norton Computing eventually was bought by Symantec, and the rest, as they say, is history.
Frank approached me in 2012 asking for my help. He had a successful open-source company, basically spun out of Sun Microsystems years earlier, which had become the dominant open-source product in app performance testing, with three million downloads and growing. The help he requested was to create, and dramatically expand on, a commercial company and offering from that work. I must have said No a dozen times before finally saying Yes. It was called PushToTest; we later renamed it Appvance (of which today I'm CEO).
Appvance counts some of the largest companies in the world as clients: PepsiCo and BestBuy and many financial institutions and insurance companies and others. But I noticed, almost by chance, that we were also executing a particular security test service which utilized our performance technology. To me, they didn't seem to go together, so I asked what was going on and why. The answer surprised me: a few large companies had realized that break-ins were occurring in unusual operating conditions of their sites and apps, in places where they would not normally test. And in those places, a lot was exposed.
As I soon found out, everyone we tested was essentially "bare naked" in this place – yet they had all passed standard security screens and had substantial network security installed. And I mean, they were really exposed. We didn't even have to try to see secure data; it just showed up on the screen. A regular user could see it. In one example with a large multinational bank, at these usage conditions anyone logging in to their own account would see and gain access to someone else's account. Oops. Yet they had passed all security screens before we discovered this.
Network vs. Apps
Over the past many years, Cisco, Juniper, and others have brought great hardware and systems to market to protect the network from intrusion. In fact, they're so efficient that network break-ins have gone way down.
In the 1990s, most security issues occurred at the network layer. By 2005, however, Gartner estimated that 75% of security breaches were now at the application layer. In 2009, Verizon found that 79% of breaches were at the app layer. And in 2011, the National Institute of Standards and Technology found that 92% of all security vulnerabilities wee now considered application vulnerabilities and not network vulnerabilities.
While not a completely scientific assessment, using the above data sources and others, the trend shown in the chart below is clear: breaches are occurring at an alarming rate at the app layer, and far less at the network layer.
The main reason (I presume) for this is the excellent job that network administrators have done locking down the network. This is both good and bad. The spend on network security has been huge. According to a recent presentation by IBM, that spend now dwarfs that of the spend on app-level security, as the chart below illustrates.
So the good news is that we have plugged up the network so tight that hackers don't appear to bother with it any longer. The bad news is that they haven't gone away. As we have built and deployed apps, both outside-facing and internal, faster than at any time in history, the invaders have just moved to exploit our apps. And it appears that they are having great success.
There are now (arguably) millions of app developers, all over the world, and our largest corporations are farming out app development to others every hour. From digital agencies creating user engagement contests to the small one-person shop writing a corporate app to access specific corporate functions, more apps, including mobile and web, are built and deployed than ever before – by some estimates over 100,000 a week. And they provide access to more and more proprietary data. So... why try and break into network systems when these apps already have access to the data? And moreover, few if any of the app developers have any expertise in app security.
Expressed Concerns
In a white paper he wrote for Intekras, a security consulting firm focused on government clients, cyber security expert Donald J. Fergus states so eloquently (emphasis mine):
According to the most recent Ponemon Institute's fifth annual U.S. Cost of a Data Breach Study, the cost of a data breach is $204 per compromised record, with the average total cost of a data breach amounting to $6.75 million. So, even for a small organization with 5,000 records, that's over one million dollars. And the fines and legal actions associated with data breaches can put businesses in jeopardy. Additionally, compromised sites and databases can frighten users and lead them to seek out alternative providers. And yet most organizations are not focusing on securing their applications. The main reason continues to be lack of understanding and knowledge, and a false sense of security. Many IT professionals still believe that having a network firewall, IDS, SSL certificates, and so on will protect them from hackers attacking their web sites or databases. It's like having locks on your front door but leaving your windows and side doors wide open and hoping that burglars will only try to come through the front door."
Cisco recently stated:
Organizations must also consider flaws in their design and implementation. Hackers looking for security flaws within applications often find them, thereby accessing hardware, operating systems and data.
The Open Web Application Security Project (OWASP) states on its site:
Systems and network administrators in [the] last 5-10 years (end 1990s to early 00s) have achieved significant maturity on controlling OS and network level attacks. Strong OS hardening/patching procedures coupled with well managed firewalls provides sufficient surety to the business that these layers are secure and not easy to penetrate.
This is yet not true for applications, especially web applications. Web applications provide a logical tunnel from outside/Internet to the backend databases inside the enterprise. Web applications are complex pieces of code with a mix of customized business logic, third party libraries, back-end database routines and integration to multiple other applications. Complexity increases potential points of failures. A recent study by penetration testers shows that more than 95% of web applications have some sort of vulnerability.
And Symantec was quoted in an article by Sam Shead of ZDNet in October 2012:
Symantec has detected a new type of disguised attack that uses a distributed denial-of-service (DDoS) to draw a business's attention away from a more important security breach. The multi-vector attack includes the DDoS as a bluff so it can quietly target another vulnerability, the company said at the RSA Europe 2012 conference in London. "It's an attack where multiple seemingly different attacks are launched by an adversary on a target," Francis deSouza, Symantec's head of enterprise products and services, said during a keynote speech. "DDoS's have gone from being a blunt-forced attack to being a sophisticated diversionary attack to disguise another attack." DeSouza said financial services companies handling vast amounts of data are most susceptible to these tactics.
In the past year, for example, phishing attacks have been directed at IT administrators at European banks, he noted. These eventually enabled malware to penetrate the banks' systems and steal login credentials. As soon as the criminals had the login details, they launched the DDoS attacks against the banks. This was carefully timed so that it occurred on a Friday afternoon when IT departments were thinly staffed. "Once the attack was launched, the IT department predictably moved resources to deal with DDoS attack," said deSouza. While this was happening, the cybercriminals launched the real attack, which allowed them to grab and clone private data that could be used to steal money. The cybercriminal gang drained $9m in 48 hours from a selection of accounts in cities across the world.
A Bit of Tech Talk
As a quick primer: a DDoS attack is one method of overloading an app. Someone simply sends more requests to the app than the servers were meant to handle. This is often done by using slave machines around the world to send simultaneous requests. They can look like legitimate requests from legitimate users. It's not easy to stop if well orchestrated, and it's easy for the perpetrators to do. It gets IT scrambling to add more servers or deny responses to certain sources, when all the while it's actually a ruse to steal data when no one is watching. It's easy, it's brilliant, and it's happening every day – not in order to take a site down, but to exploit its vulnerabilities and give up corporate secrets.
Apps are built to handle a maximum load by their designers. Now, in my experience at Appvance over the past year, not a single app actually met its goals the first time through performance testing. Testing involves creating hundreds or thousands (or millions) of virtual users to purposely load the app and systems to the breaking point and beyond, so when a large consumer company launches a new app and gets 3M registrations in three hours, it knows it can handle the load. It turns out this is much harder than one might think. It isn't just about adding more servers – it gets into client-side code issues and third-party services that break down and database access and caching, etc. There are always substantial code rewrites for days or weeks after a proper load test.
But this simply tests the app's ability to scale and support user volume with a good user experience rather than a degraded one. Unfortunately, most apps also go untested for load. I am surprised at how many organizations just run out of time or don't bother. They do functional testing, get it working, and send it to the app store or release it to their employees.
In a perfect world, one would create software in an agile workflow that includes functional testing, security testing, and load testing at every major build, so by the time of release, the app has been through that cycle dozens of times. But in most cases this process simply isn't followed. Functional testing is certainly performed. But security is often ignored, as is load. And if they do perform security and load testing, it is typically one time, at the end of the project, if there is time and budget. So apps are released which have full access to important data, with little to no security or load testing. Happens every minute.
But let's get to the heart of the matter: app layer security screens, when they are performed, traditionally have happened at functional test. That is Stage 1 of the app performance regime, as depicted in the graph below:
Stage 1, standard white-hat [nonmalicious hack-tested] security screens are quite comprehensive, and the bad guys know this. Trying to break into an app and find a weakness with a few users on it is nearly pointless if all the holes at Stage 1 have been closed.
But what happens at Stage 2 (app under stress, or heavy load) and Stage 3 (app in failure mode = some servers have crashed)? Well, here is the rub. A lot happens, and in pretty difficult-to-predict ways. The new breed of hackers (and thieves) are targeting compromised apps and sites at load. Apps behave differently at high levels of load, and these hackers seek to hold the software at a compromised condition in order to further breach security. A compromised app at load is a frightening thing to behold: private user data is accessible, system functions are available to misuse, and database integrity is compromised.
Traditional white-hat security screens operate when the app and systems are running as designed and at limited user load levels. That used to be enough, but no longer; the black hats have figured out new ways to break in as the app is struggling under heavy load and stress. While your IT group is busy trying to restore basic functionality from an attack – which is actually the smokescreen for the real attack – the hackers are able to steal passwords, credit cards, customer info, Social Security numbers, and proprietary company data.
Mark Anderson and others have been loudly proclaiming that we have a problem. Recently, much of the current cause (Stages 2 & 3) has been identified. It affects everything we have ever built. Consumer apps. Corporate apps. Everything. Old and new... all bad. So, given the knowledge about the issue, everyone must be addressing it, right? Wrong.
Most companies complain that they have no budget to test for, identify, and fix the issues. I hear this every day.
Case in point: As I was writing this article, I took a call from a large government agency with lots of data to protect; there are 150,000 users of this particular service. I asked about security. The answer was, essentially: "We will do one white-hat sweep at functional test time [Stage 1]. There is nothing in the budget to do further security testing [Stages 2 & 3], and even if we did and there were problems uncovered, we don't have the budget to go back and fix them." This agency, like many others, didn't include this in its original budget estimates, so now it isn't an option.
So does the security of data, intellectual property, and that of nations come down to budget? As new potential issues are found, what is the process for going back and finding the problems in long-launched sites and apps? Oh... There is none. So I can only surmise that an app built five years ago, which allows (for example) employees to access certain corporate data and functions, has zero hope of being re-tested and fixed. Who budgets to go back in time? Yet the data is there as much as it is in anything new built today.
One may think that this is all about large-scale apps. It isn't. A small-scale app, meant to scale to only 20 concurrent users, is the easiest to break into. Small-scale apps tend not to be carefully watched, and one can easily ramp an attack to 50 users, far overwhelming that app into a territory it has never been in before, exposing all kinds of goodies. For example, apps that are intended for building operations are built to have little scale in terms of Web users – just a few, usually. That's plenty, when they control temperature, fresh air, lighting, and so forth. Even though there is little interesting data there, it's pretty easy to get a building to ridiculous temperatures, turn off all the lights, and then crash the server. Or a financial system meant to house important private corporate data, also meant for only 100 (or fewer) users logged in at a time. Pretty easy to bring that to its knees and potentially expose sensitive information. Even if these are third-party systems, they generally have not been tested for security at Stages 2 & 3.
Yes, there are some companies that do Stage 2 & 3 security testing. Google. eBay. Amazon. No doubt those whose main business is net-centric have dealt with this for some time. But net-centric or not, the potential data exposed at banks, healthcare institutions, government agencies, and large corporations is just as critical – maybe even more so – yet they are far less tested and protected.
Where Do We Go from Here?
It isn't enough any longer to just talk about the rampant theft of supposedly secure information. IBM and others have shown that we are spending substantial sums on network security, while app security, particularly in loaded conditions, gets almost no attention or money. It's almost an afterthought. Amazing.
Spend needs to be aligned to today's threats, not yesterday's. IT budgets need to rapidly shift spend from network security to app security. It's critical to fully test all apps, old and new, for security issues at Stages 1, 2 & 3. Arguably 90% of the security spend should be in this area, given that is where the vulnerabilities are and where the hackers have moved. Of course, this is always a moving target, and this article should seem dated in a few years, as companies close these holes and the bad guys find new ones.
There are certainly many other security issues; I am not attempting to cover them all or, frankly, any others. Just one big one we have found in virtually every app at every company we've tested, and one which we are helping those companies close up tight.
Is this the largest problem today, the high-order bit? Maybe. The stats suggest so. But in any case, this little missive may raise awareness of one area of security concern we can do something about today. And if you didn't know about it or how to deal with it until today, don't feel bad. Now you do.
If you can't get the budget to find and then fix the holes, that's ok. The bad guys are peeking into those holes right now. And you won't even know.
Sleep well.
-----
About the Author
Kevin Surace is CEO of Appvance in San Jose, California. He sits on five boards and is a serial entrepreneur, having started a dozen tech companies. He has been recognized as Entrepreneur of the Year by Inc. Magazine, Tech Pioneer by the World Economic Forum, and Innovator of the Decade by CNBC, and inducted into the Innovation Hall of Fame at RIT. He has been awarded 23 patents.
He can be contacted at ksurace@appvance.com.
-------
I would like to thank Kevin for ringing the bell on the issue of applications as the security failure point, for all of our members. Through our INVNT/IP Global Consortium, we are working to improve communications with potential victims of the massive, well-organized attacks set up by nations today, as they strive to grow through the theft of the same ideas that drive our members' corporate (and personal) success.
Those interested in learning more about these and related problems may wish to go to our reference library on this question, which we update daily, at www.invntip.com. Here, you can learn not only about attacks and thefts, by country and year, but also see the larger picture through white papers, institute studies, and related academic and industry views. Finally, if your firm is attacked, you can report it, anonymously if you wish, on this site as well.
No wonder it gets attacked.
My thanks also goes out to Sally Anderson, who does such a great job of making the rest of us look good. – mra.
Your comments are always welcome.
Sincerely,
Mark R. Anderson
