Big Data#
Have you ever wondered, “What is big data?” Let’s dive into a real-world example that vividly paints the picture of this phenomenon. Target, one of the largest retail networks in America, has had an intriguing experience demonstrating the power of big data.
A confused father walked into one of their stores, frustrated and baffled. His complaint? His 15-year-old daughter had been receiving mail advertising baby products. Upon hearing this, the manager assured the man that it was likely a system error and promised to look into the matter.
However, to everyone’s surprise, the same man returned a few days later, now visibly embarrassed. He confessed that he was unaware of his daughter’s pregnancy, raising an astounding question: “How did Target know?”
The response to this question reveals the power and precision of data science. Target’s data scientists explained that their system detected a change in the young woman’s buying patterns. Seven months ago, she purchased a pregnancy test from Target, but since then, she hadn’t bought any sanitary products. These patterns alerted the system, increasing the likelihood of her being pregnant.
Target’s ability to predict customer needs and preferences is awe-inspiring. If you’ve been a regular customer for years, Target can predict major life events, such as pregnancy, and cater its marketing and product suggestions accordingly. The system will send super discount coupons for new clothes just when you’re beginning to realize you might need them.
Target’s ability extends beyond anticipating customer needs. It can also infer personal circumstances. For instance, suppose you’re a long-time customer who’s been purchasing baby breakfast items. If you suddenly start buying beer, Target’s system would infer a potential shift in your life situation, perhaps a separation, and adjust its product suggestions accordingly.
This uncanny predictive power isn’t magic; it’s the power of data.
In just a few years, the companies that can effectively harness the power of the data they collect can overtake their market rivals. Indeed, the game has shifted. It’s no longer just about oil, it’s about data.
Every company, irrespective of size, needs to equip itself with data engineers and invest in affordable cloud platforms. Small businesses, too, must recognize this reality. If they underestimate the power of data, they could potentially lag.
Take, for instance, the recent Data+AI Summit held in San Francisco. With around 70,000 data professionals attending both online and offline, the event underscored the industry’s emphasis on data science. Major companies, including Amazon, Microsoft, and JP Morgan, demonstrated their high-value approach towards data science, indicating the exponential growth in the field.
Technological developments are happening at a breakneck pace. Not long ago, data lakes and data warehouses were the latest breakthroughs. Today, these technologies are already making way for lakehouses, demonstrating the relentless evolution in the field. Scripts are now being written in Python and SQL, turning raw data into engaging visualizations and useful insights.
Just to emphasize the vastness of data we’re discussing here - JP Morgan processes 500 PB of data per day. The magnitude is such that if you tried to copy this data onto CDs, the stack would tower to a height 100 times that of Mount Everest!
Without the assistance of cloud technology like Databricks, managing such vast volumes of data would be unfeasible. This underscores the reality we are living in - the era of big data.
In conclusion, we’re not merely living in the information age. We’re living in the age of data-driven insights and predictions. Welcome to the century of dataizm!
