On Sunday 30th March Boston’s Head of HPC David Power wrote a great article in 'Business Technology', featured in the Sunday Telegraph. The feature discusses how companies are relying on Big Data to turn information into business results - and how Boston’s solutions can help! Read the full article below.
Big data has become a big talking point these days, but there are many different interpretations, contrasting views and conflicting arguments surrounding it. Dan Ariely of Duke University provides what is in my opinion, a great analogy:
"Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it."
At Boston we have been providing HPC (High Performance Computing) solutions for more than 20 years. HPC has always involved big data for things like large-scale analysis and huge storage requirements. Boston has provided systems to a range of users from various scientific backgrounds, all of whom needed to generate, store and analyse immense volumes of data. One such customer was CERN, which was generating 15 petabytes of data on a daily basis (roughly 250 million MP3 songs). Producing data at this scale can create numerous technological challenges and requires some highly innovative HPC solutions that push the boundaries of science and technology.
The SKA Project (Square Kilometer Array) is another venture Boston is currently engaged with. This project aims to analyse approximately one exabyte of data a day (an exabyte is 1,000 petabytes - you can do the math on the number of MP3s this equates to!). These are just two examples of the current scientific challenges with regards to generating and analysing data - but still not necessarily what people mean when they speak about Big Data.
Technology is becoming more and more ubiquitous in our society. This has been the main driving factor in the evolution of big data. Our standard means of communication has been digital for some time (email, text messages) - however, thanks to mobile technology and an increasing number of smart devices (TVs, watches, games consoles, cars, some domestic appliances), more of our daily lives are being carried out online.
This sort of information may seem worthless on an individual scale, but if you start to look at it from a holistic point of view or as an aggregated data set, there is huge opportunity for extracting information that can have a global impact. Using Google flu trends as an example, something as simple as an individual searching for a flu remedy online when combined with geolocation and time stamp information regarding similar search terms across multiple states and countries, can suddenly become hugely impactful. This sort of big data analysis could potentially alert us to, and help stop the spread of, more powerful epidemics and pandemics besides the flu.
I sat in on a talk recently which was conducted by the University of Warwick, who used big data techniques to identify correlations between internet search terms and fluctuations in financial markets by taking the same approach as Google flu trends, but mapping search sentiment to movements in the stock market. The long and short of their findings is that they were able to outperform the basic "buy and hold" approach (where people buy stocks and hold onto them over a period of time) by almost 20-fold over a number of years with simple strategies such as selling when the search term "debt" appeared as a top search term. I've hugely oversimplified their sophisticated algorithm, but the point is they used big data techniques to take everyday data that was freely available and create value from this information.
If we look into the future, the Internet of Things is going to have a further impact on big data. Not only will we have mobile phones generating data, but personal sensors, smart watches, cars, homes, fridges - in fact, anything that you have in the house could potentially be a smart device in the near future.
One can only begin to imagine a world where your fridge texts you about the low stock of alcohol available having reviewed your calendar and noticed a big sporting event coming up. The sporting event in question could have athletes wearing sensors and GPS tracking devices, which will be automatically collecting measurements and statistics to feed back to coaching and medical teams. The team that wins in the future may be the team with the best analytical capabilities rather than just the best team on paper. This has already been proven in the baseball arena with Oakland Athletics, who employed statistical methods to put together the optimal team, on about 30 per cent of the budget that the top teams had (check out the book or film Moneyball for all the details). Devices being able to query, analyse and make intelligent decisions will be a big part of the future - these will all be powered by big data solutions.
I have a one-year-old daughter - she already has a larger online presence than I do thanks to my wife posting pictures of her either in a new outfit or using her dinner as face paint! She'll have all this information logged and recorded and available at a moment's notice. I've no idea what I did on this date 10 years ago. In 10 years' time, however, she will. At this stage, I'm not entirely sure as to whether this eventuality should fill me with intrigue or fear… or both. This lifelong digital presence will become the norm for our children and our children's children. They'll have the ability to access all their data, anytime, anywhere - from pictures posted by parents when they were young to videos of their life achievements captured via Google Glass, and everything in between. One thing is for certain: big data is a truly exciting space to be in and is going to become increasingly applicable and desirable to companies in the future, regardless of their size or budget.
Business leaders are rapidly facing up to the implications of this growing phenomenon. If they have not already done so, they are upgrading their data management approaches and processes in order to meet the challenges. They are changing the way they work and they are re-tasking people within their organisations to take advantage of the insight and business intelligence that can be generated by these critical numbers. If your business is starting to look at big data solutions or is planning to in the future, Boston and Intel have the tools, optimised server solutions, software platforms and the in-house expertise to ensure your big data project is a success.