In the digital era in which we live, data has taken on a central role in our daily lives. The growing amount of information generated every day has led to the emergence of a field of study and application known as “Big Data.” In this article, we will explore the history of how Big Data emerged, evolved, and shaped our world.

The roots of Big Data

The history of Big Data begins long before the digital era, when the first attempts to collect and analyse information on a large scale were made. The first censuses, for example, are examples of efforts to collect data on a large population. However, the real turning point came with the advent of computers and automation. In the 1960s, mainframe computers began to be used to manage large amounts of business data. These systems allowed companies to process and store information more efficiently than traditional methods. However, the idea of Big Data, as we know it today, was not fully developed until the 2000s.

The explosion

The explosion of Big Data can be mainly associated with the 2000s onwards, although the roots of large-scale data collection and analysis date back to previous years. During the 1990s, with the exponential growth of the World Wide Web and the increase in online activities, a significant accumulation of data began. However, it was in the 2000s that the term “Big Data” began to gain popularity and took shape as a distinctive concept.

During this period, several factors contributed to the explosion of Big Data. The spread of the Internet and the increase in online activities generated huge amounts of data, including text, images, videos, and more. At the same time, data storage technologies improved significantly, making it more practical and cost-effective to store large amounts of information.

In addition, the advancement of distributed processing technologies, such as the Apache Hadoop framework, enabled companies to manage and analyse large datasets more efficiently. Organisations began to recognise the strategic value of Big Data in optimising operations, making informed decisions, and identifying customer behaviour patterns.

So, although the roots of large-scale data collection are older, we can consider the 2000s as the period when Big Data truly exploded and began to significantly transform various sectors.

The Big Data revolution: from analogue to digital

The real turning point for Big Data came with digital transformation. With increased connectivity and the spread of smart devices, we began to generate data continuously and automatically. From financial transactions to social media interactions, every online activity contributes to the vast network of information that constitutes Big Data.

Companies quickly adapted to leverage this wealth of data. Business models were redefined through data analysis to better understand customer behaviour, optimise operations, and make informed decisions. The healthcare sector benefited from the ability to analyse large datasets to identify patterns and trends, improving the diagnosis and treatment of diseases.

Phases in the history of Big Data

• Origins (1960s–1970s): the first companies began collecting and storing large amounts of data to improve their operations. We recall the project that lasted from 1952 to 1963, involving 12,000 cryptologists employed to create an automated system for analysing information transmitted during the Cold War.
• Development (1980s–1990s): the growth of the Internet and the advent of Web 2.0 led to an increase in the quantity and variety of available data.
• Explosion (1990s–2000s): the growth of social media, mobile devices, and IoT led to an exponential increase in the quantity of available data. In 1995, the first supercomputer was built. In 2005, the term BIG DATA was coined by R. Mougalas, and in the same year, Hadoop by Yahoo was created based on Google’s MapReduce. In 2009, the Indian government collected iris scans, fingerprints, and photographs of all its inhabitants, creating the largest biometric database.
• Maturity (2010s–present): the development of new technologies for managing and analysing big data has led to an increase in applications for this data. In 2010, Schmidt stated at the Techonomy Conference that the amount of data produced from the beginning of humankind to 2003 is now generated in just two days. The McKinsey report on BIG DATA was published in 2011.

Characteristics of the phases

• Origins: big data is generated from a limited number of sources, such as companies and government institutions.
• Development: the quantity and variety of data increase, but the technologies for managing it are still limited.
• Explosion: the quantity and variety of data increase exponentially, making the development of new technologies necessary.
• Maturity: technologies for managing and analysing big data evolve, making it possible to extract value from ever larger amounts of data.

What is Big Data used for?

Big data can be used for a wide range of applications, including:

• Business intelligence: big data can be used to improve business decisions, for example, to forecast customer demand or optimise the supply chain.
• Marketing: big data can be used to create personalised customer profiles and target marketing campaigns more effectively.
• Healthcare: big data can be used to improve the diagnosis and treatment of diseases.
• Security: big data can be used to improve public safety, for example, to prevent crimes or identify terrorist threats.
• Environment: big data can be used to monitor the environment and reduce environmental impact.

The possible future of Big Data

As we reflect on the brief history of Big Data, it is clear that we are only at the beginning of this revolution. With the advent of emerging technologies such as artificial intelligence and machine learning, Big Data will continue to evolve and shape our world in ways we cannot yet fully imagine. Big data is destined to play an increasingly important role in our future. Technologies for managing and analysing big data are evolving rapidly, making it possible to extract value from ever larger amounts of data.
In the future, big data will be used to improve our lives in many ways, including:

• Health: big data will be used to develop new medical treatments and improve disease prevention. Big data can be used to improve diagnosis and treatment of diseases, develop new drugs, and improve disease prevention.
• Environment: big data will be used to monitor the environment and reduce environmental impact.
• Education: big data will be used to personalise learning and improve the effectiveness of education while reducing school dropouts.
• Security: big data can be used to improve public safety, prevent crimes, and identify terrorist threats.

Of course, the use of big data also raises a number of challenges, including:

• Privacy: the use of big data can raise ethical issues, such as privacy and discrimination.
• Social impact: the use of big data can have a significant impact on society, both positive and negative.

In conclusion, the history of Big Data is a story of transformation, innovation, and impact. From humble beginnings to a global driving force, Big Data has proven to be a powerful catalyst for progress in society, the economy, and science. It remains to be seen how this story will unfold in the coming years and how Big Data will continue to influence the way we live and work.