3.2. Legacy Systems Issues and Concerns Questions
This section captures answers about legacy systems and data, and the advantages, disadvantages, and factors influencing legacy systems and data in the business intelligence community. The following provides a description of the survey results.
What kind of data does your organisation have?
Most organisations we surveyed were working on organisational transactional and historical data. All of the respondents from financial organisations believed that they had good practices in place to manage data, processes, and infrastructure for detecting fraud. Similarly, all of the respondents believed that data was the biggest asset of their organisation. A share of 48% of the respondents believed that the data generated within the organisation fitted into the characteristics of Big Data and if analysed effectively can provide a competitive edge to the organisation. The drawback was that the existing technology inhibited the organisation from exploiting this volume of data. Our survey identified that the kinds of data used include structured data, unstructured data, sensor data, log files data, big data, time stamped data, machine data, spatiotemporal data, open data, real-time data, operational data, and unverified outdated data.
shows the responses relating to the various kinds of data used by our respondents. Our survey identified that all organisations had structured, unstructured, sensor, log file, time stamped, machine, and operational data. However, only 5.21% had social media (Facebook, Twitter, and LinkedIn) data. The share of unverified outdated data sitting within organisations was 22.30%. This data was collected and stored within organisations, and no employees had knowledge about the data, its relevancy, or whether it could be put to use. A share of 63.91% of respondents believed that data silos are one of the biggest identified challenges to be addressed and diminish the power of Big Data. These respondents believed that if data was stored in different data sources, different data systems, and different organisational units, without some link between them, then no complete insights can be generated because the available data was not integrated at the back end.
What do you think are the benefits of legacy systems and data in your organisation?
Most respondents believed that their organisations used legacy systems for day-to-day operations. People were very comfortable using these legacy systems as they were familiar with them. All respondents believed that business continuity was important, and a legacy system that worked and kept everyone on the same page was good for the organisation. A share of 64.94% of the respondents believed that using a legacy system was easy to manage and within the control of the organisation. All respondents believed that using existing systems was less complex as over time people gained confidence in using it. Globally, data is being retrieved from legacy systems for business intelligence and decision making. However, the insights available from existing data have become more meaningful with the advancement in tools and technologies. Legacy systems are considered as the life-blood of organisations, such as Supply Chain Management (SCM), Human Resources Systems (HRS), Customer Relationship Management (CRM), and Learning Management Systems (LMS). Respondents admitted that legacy systems were very important to the running of their organisations. All respondents believed that the legacy systems running in their organisations contained significant and invaluable business logic of the organisation. These respondents further believed that the business logic embedded in the legacy systems was very important for the running of the organisations and organisations could not afford to throw them away. All respondents believed that the “legacy systems cannot be replaced because of obvious reasons and hence Big Data solutions need to be integrated to legacy systems”. All respondents believed that “there is a need for integrating Big Data with their legacy systems”.
What do you think are the main disadvantages with reporting when using legacy system and data?
A share of 56% of respondents believed that existing systems and data could not fulfil the requirements of changing technology, which should be treated as being of prime importance if organisations wished to gain and maintain a competitive edge in today’s world. Business requirements change continuously. The disadvantages of legacy systems were identified as: costly, outdated, limited flexibility, cannot generate reports in real time, store only organisational data in a structured format, immobile systems, and data integrity problems. Data and processes are not scalable and cannot be cross referenced. Data redundancy exists, together with the inability of systems to share data with each other. Vital information is lost as a result. Legacy systems support certain types of reporting and can supplement legacy system data with other data sources, thereby improving insights. A share of 57.73% of the respondents believed that their customer contact information was incorrect. One of the respondents mentioned that “if you’ve got a database full of inaccurate customer data, you might as well have no data at all”. A share of 57.73% of the respondents believed that data silos can be eliminated by integrating data. They also believed that, in their organisation, accessing legacy system data takes a long time. Business decisions become totally dependent on accessing legacy data. Business users cannot access legacy systems directly, so the request to extract data from legacy systems has to go through the IT department for processing. This takes a long time as too many processes are involved. Organisations need to reduce these unwanted processes by integrating and eliminating data silos in today’s world of BI.
Where and when do you use legacy systems and data for decision making?
Legacy data are used for historical analysis and operational analysis to understand how to improve the future based on historical evidence. Data is analysed for descriptive and predictive analytics. Respondents want to have prescriptive analytics; however, they do not have the skill sets, tools, and technologies to run prescriptive analytics. Our respondents required the use of legacy systems for five key themes; information access; insight; foresight; business agility; and strategic alignment. All of the respondents and their organisations were using data generated from systems such as enterprise resource planning systems, attendance tracking systems, and e-commerce systems. The data generated from these systems were stored in data warehouses, data marts, and database management systems. These technologies have existed in various forms for many years. These are large amounts of data and our respondents believed that these data fit into the meaning of Big Data. However, Big Data is not only about storing and retrieving semi-structured and unstructured data.
3.3. Big Data Initiatives and Implementation Questions
This section captures answers about finding the proper fit of a Big Data solution and technologies for an organisation. The section discusses the results we found from the survey.
In your opinion what is the range (high; moderate and low) of data processing in your organisation in relation to volume, velocity, variety, and veracity?
Most of the respondents believed that their organisations were contributing towards one or other of the stated characteristics of Big Data (volume, velocity, variety, and veracity) in the range of high, moderate, and low. Figure 4
shows the responses from our respondents about data volume, velocity, variety, and veracity in their organisation.
Does your organisation have information management big data and analytics capabilities?
A share of 72.16% of the respondents believed that their organisational data was measured in gigabytes and that applying new tools for business intelligence would help their organisation. However, they also believed applying new tools should not disrupt their existing running systems. Organisations require different types of analytics for different purposes, so new technology should help them with these changing requirements. Organisations having gigabytes of data do not have any Big Data project in their organisation. However, data analytics is enhanced using machine learning to provide more insight into the data and make use of the data that the organisations are already collecting. A share of 27.83% of the respondents believed that their organisations had petabytes of data and Big Data projects were underway to identify real time fraud and detection. These included finance, insurance, and service provider organisations. The aviation industry is establishing a framework to use Big Data. Higher education systems are using Learning Analytics on already collected data. This fits into Big Data, as the analysis of discussion forums, emails, etc., uses unstructured data. A share of 70.1% of the respondents stated that Big Data had primarily been used to drive profits. All of the respondents believed that Big Data analytics can provide deep insights into customer behaviour and help in gaining a 360° view of their customers, by analysing and integrating existing data. One of the respondents stated that “Big Data analytics is all about understanding the customer, and that means harnessing all resources not just analysing all contacts with the organisation, but also linking to external sources such as social media and commercially available data. For the digital supply chain, it is about collecting, analysing and interpreting the data from the myriad of connected devices”.
Do you think that integrating a big data solution will benefit your organisation?
All of the respondents believed that integrating Big Data solutions would bring benefits to their organisation in different forms. This question had interesting answers, in which higher education respondents highlighted that contract cheating, which is a form of fraud, can be detected using Big Data analytics. Fraud detection is very common in financial, service provider, and insurance organisations. One of the respondents stated that “if you can obtain all the relevant data, analyse it quickly, surface actionable insights, and drive them back into operational systems, then you can affect events as they’re still unfolding”.
Has your organisation developed a big data strategy?
Organisations with gigabytes of data had not developed any Big Data strategy within their organisation. They were using advanced analytics, pattern recognition, and deep learning techniques to identify any irregular patterns. Organisations with petabytes of data had a Big Data strategy in place as these organisations were already working on a Big Data framework. However, their reporting systems were not integrated to their legacy systems. According to our survey, 86.62% of the respondents using Big Data (that is, 27.83% of all respondents) believed that Big Data framework implementation requires organisational efforts. These respondents believed that one of the disruptive facets of Big Data is the use of a wide range of Big Data tools and technologies for innovative data management to support different analytics.
What kind of analytics is used in your organisation?
All of the respondents believed that they were using descriptive analytics. A share of 53% believed that they were using predictive analytics and 27% believed that they were using prescriptive analytics. Organisations on the forefront of money management were using prescriptive analytics. The ecosystem of Big Data is daunting and confusing. A share of 10.3% of the respondents believed that there should be guidelines with requirements on how to use Big Data tools, technology, and architectures. One of the respondents stated that “the most practical use cases for Big Data involve the availability of data, augmenting existing storage of data, as well as allowing access to end-users employing business intelligence tools for the purpose of the discovery of data”.
How do you generate reporting for business intelligence?
All of the respondents were using legacy systems, such as CRM, SCM, LMS, etc., to generate reports for business intelligence. A share of 27% of the respondents believed that they were using a Big Data framework to generate reports. However, the frameworks did not include data from legacy systems. All of the respondents believed that they used legacy systems and data for business intelligence and decision making.
What approaches does your organisation use to integrate legacy systems and data? What problems are associated with it?
All of the respondents believed that there was no framework implemented to integrate Big Data solutions with legacy systems. However, they were generating reports from legacy systems, generating reports from Big Data frameworks, and combining the reports for final analysis. Respondents cited a lack of experience slowing project progress (48%), struggling to keep up with new data sources (58%), and issues with constantly changing business requirements (44%) as their top challenges.
When making a business decision in your organisation what do you mostly rely on?
Our survey identified that business decisions are made at different levels, which can be classified as functional level, business unit level, and corporate level. At all three levels, people rely on data and reports generated by existing organisational systems. As identified by our respondents, 100% of the respondents believed that information is the key success factor influencing the decision-making process. A share of 19.58% of the respondents believed that Big Data analytics was integrated into the decision-making process. The remaining 80.42% respondents believed that Big Data analytics was not used in the decision-making process.
Do you see value in integrating big data solutions into legacy systems and data in your organisation?
All of the respondents believed that integrating a Big Data solution with legacy systems and data in their organisation would bring benefit to their organisation. However, 87.62% of the respondents also stated that their organisation must have a strategic plan for integrating data from multiple data sources. This was required for Big Data integration for receiving holistic information from different data sources, including legacy systems and data. Integrating new datasets into existing pipelines (72%) was cited as a primary obstacle to Big Data projects. This was shown as the biggest concern that would hinder an organisation’s progress towards a Big Data solution. These respondents believed that their organisation had hundreds of systems. This means that to receive all relevant data, the data must be extracted from many different sources, and the volumes could be overwhelming. In addition to the volume, the variety of sources also needs to be considered for integration purposes.
Does your organisation use any big data technology? Please specify the technology in the text box below.
A share of 12.38% of the respondents said that their organisation was using some kind of Big Data technology and tools. The most common tools used were Hadoop, Apache Spark, Apache storm, Cassandra, RapidMiner, MongoDB, R programming, and Neo4J. A share of 3.09% of the respondents believed that Hadoop was not suitable for social networking. For large volumes and graph-related issues, such as social networking or demographic patterns, Neo4j, a graph database, may be a better choice. Neo4j is one of the Big Data tools that is widely used as a graph database in the Big Data industry.
What is the biggest challenge in your organisation for collecting, accessing, storing, processing, and analysing data?
While Big Data offers many benefits, implementation of Big Data also has many challenges. The Big Data landscape is vast, making it even more challenging and complex for organisations to implement Big Data. Business users do not have enough understanding and knowledge of how Big Data can be utilised within organisation. Some of the commonly identified issues include inadequate knowledge about the technologies involved, data privacy, and inadequate analytical capabilities of organisations. A share of 85.56% of the respondents believed that their organisation lacked the skills of Big Data implementation in the workforce. Employees are not trained enough to handle Big Data technologies with confidence. Not many people are actually trained to work with Big Data, which then becomes an even bigger problem. Volumes, velocities, and varieties of data are growing continuously, giving organisations a large number of opportunities to gain insights that might otherwise be hidden in their available raw data. A share of 87.62% of the respondents were looking to increase their data team headcount to support Big Data solutions, but 85.56% also said that it was difficult to find professionals with the right skills and experiences within Big Data. Organisations were struggling to satisfy the requirements of implementing Big Data.
What are your goals of adopting big data projects?
Our respondents believed that there were several goals for adopting Big Data projects within their organisations. Following are the listed goals for adopting Big Data projects according to our identified nine organisational categories.
Research Organisation: Goals are to process and analyse a high variety of data to generate more insight from the data.
Service Providers: Goals are to collect, process, and analyse high volume and variety of data so that consumers’ insights can be utilised and recommendations can be built.
Higher Education: Goals are to collect, process, and analyse a high variety of data so that it can be used to enhance good practices in learning and teaching. Educators and learners should be able to take control of informed decisions. Teachers’ performances can be fine-tuned and measured against student numbers, subject matter, student demographics, student aspirations, and behavioural classification.
Financial Organisations: Goals are to collect, process, and analyse a high volume and high variety of data for early fraud detection and mitigation, and anti-money laundering.
Energy Sectors: Goals are to collect, process, and analyse data from smart meters so that energy consumption can be analysed for improved customer feedback and better control of utilities use. Big Data analytics in the energy sector plays a crucial role in reducing energy consumption and improving energy efficiency. Through Big Data analytics, energy utilities can optimize power generation and planning.
Supermarket Organisations: Goals are to collect, process, and analyse data to optimise staffing through data from shopping patterns and local events.
Insurance: Goals are to collect, process, and analyse data derived from social media, GPS-enabled devices, and CCTV footage for fraudulent claims.
Aviation: Goals are to collect, process, and analyse data to strengthen customer value, relationships, and loyalty.
Others: Goals are to collect, process, and analyse data for optimising resources within the organisation.
If you do not have big data framework in your organisation, how beneficial will it be for your organisation? Do you see any value in having big data framework in your organisation?
A share of 12.38% of the respondents believed that they had a Big Data Framework within their organisation; 87.62% of the respondents believed that they did not have any Big Data Framework within their organisation. However, they did believe that a Big Data Framework would be beneficial to their organisation as a framework provides structure to achieve long term success. A Big Data framework concerns structure, technology, capabilities, and skilled people. The respondents believed that a Big Data framework can provide a structure for organisations that want to start with Big Data or aim to develop their Big Data capabilities further. Respondents also suggested that the Big Data framework should be vendor independent and applied to any organisation regardless of choice of technology, specialisation, or tools. It should be able to provide a common reference model that can be used by any organisation depending on their requirements of Big Data analytics and solutions. All of the respondents believed that having a Big Data framework would add value where organisations were struggling to embed a successful Big Data solution in their organisation. The respondents also believed that a Big Data framework would be useful in integrating a Big Data solution with legacy systems, as it will dictate the architecture of Big Data and help in developing a Big Data strategy.