There was an uneasy feeling in the meeting room called Gabbar — named after the famous dacoit played by Amjad Khan in the movie Sholay. Sachin Bansal, the chief executive of Flipkart, was not a happy man that Monday evening.
Unlike in Sholay, in which four people beat Gabbar's army of dacoits, in less than 24 hours, an army of online buyers were going to come crashing through the gates — hoping to lay their hands on a few thousand Xiaomi phones that were on sale. For Flipkart, the company that is up against the techsavvy Amazon, it was more important than ever to prove that they could handle the influx. Hugo Barra, a former Googler who is now a top executive at Xiaomi, was in the room. The flash sales model had worked very well for Xiaomi in other markets and they'd picked Flipkart for the sale in India.
While Flipkart was prepared to handle the spike in traffic, a couple of its core services had buckled under stress the previous week during the flash sale. Amod Malviya, Flipkart's engineering head, and Sameer Nigam, vice-president of engineering, were also present.
At about 8 pm, Bansal asked Malviya if he could rework the selling process for the flash sale that would last a few seconds next day. "It was almost like rewriting all of what we had created in the last three years in 15 hours," recalls Malviya.
Buyers were waiting for the Xiaomi sale for days and the clock was ticking away. Frenzied Flipkart engineers geared up for the last-minute code rejig. And then dawned the big day.
On Tuesday the same day Flipkart announced that it raised a massive $1 billion in funding — thousands of people hit the site at 2 pm to reserve 10,000 Xiaomi phones that were up for grabs. The sale lasted all of 5 seconds. "It seemed surreal that everything was gone. So we sat down to figure out if it was a mistake," recalls Sameer Nigam, vice president of engineering. Thankfully for Flipkart, it had not.
Amod Malviya, the engineering head, climbed on to a desk placed in the middle of the floor and cheered the team while Hugo Barra shot a video of it. But something bigger was around the corner and Malviya knew in the back of his mind that it would be nothing like they'd done before: the Big Billion Day sale on 6 October 2014.
READY FOR THE LONG HAUL
Armed with over a billion dollar in venture capital, Flipkart is readying its engines for the long haul in the Indian market. As smaller competitors snap at its feet, the threat of Amazon's quickly growing muscle in the market also looms on the horizon.
Malviya knows that if there is one thing that will be key to Flipkart's success, it is the technology being built and the team of engineers who are working round the clock to keep the site running.
After finishing school in a small town in Uttar Pradesh, Malviya went to study engineering at IIT- Kharagpur. The engineering college is the alma mater of some of India's best minds — from Sundar Pichai, the rising star at Google, to former RBI governor Duvvuri Subbarao. He graduated in 2002 and moved to Bangalore to work for various technology companies.
In 2010, he joined a 3-year-old startup called Flipkart, barely known outside the book-loving community in Bangalore. The company founded by former Amazon employees Sachin and Binny Bansal had managed to raise $1 million from Accel Partners. Fast forward to 2014, and Flipkart is leading the charge in India's ecommerce revolution. As of August, the site had over 6 million visitors and was shipping over 5 million products a month.
When a user lands up on the company's website, over 150 services need to run in a carefully orchestrated manner to make sure that the user not only buys the product but also gets it on time. Services need to make sure that the user finds what he is looking for, gets great recommendations, pays without a hitch, gets products delivered on time, is able to return a product, track it and that no one is trying to scam Flipkart.
In the technology world, the software architecture that Flipkart uses is called a service-oriented architecture — successfully implemented by its rival Amazon many years ago. The idea is to have different services performing different tasks, drawing from a single database so that it is easier to scale up parts of the whole system without disturbing everything else.
Of the 14,000 people who work for Flipkart, nearly 800 are in the technology team. Six months ago, there were only 400. By March next year, there will be 1,200 engineers working for the company.
Just a few days before the Big Billion Day, Flipkart had moved into a larger building a few kilometers away from its old office. About 500 of the engineers had been working hard to make it happen. Like a startup, there were mattresses, pizzas and round-the-clock activity in the new office.
"There was as much chaos as there was energy," recalls Malviya. The Big Billion Day was at least 10 times bigger than the Xiaomi sale. Once again, a lot of services had to be built from scratch and each of them had to be tested to see if they worked well on a big scale. "We had never done something like this before," said Malviya. The team had identified nearly 80 things that could go wrong. And sure enough, a few did go wrong.
A BILLION HITS
On the day of the sale, which was preceded by heavy marketing by Flipkart, there was a surge in traffic. It spiked at about 8 am and stayed that way until evening. Flipkart said it saw more than a billion hits across its properties.
In a few minutes, one service that was written from scratch had started to rot. In programming parlance, software rot or bit rot is the slow decay of software that eventually renders it useless. "The good thing is that we had a contingency plan. Otherwise, we would have found it difficult to recover from the initial issues," said Malviya.
Around 25-30% of the times, the service which was to act as the single source of truth for all of Flipkart's product data was faulty. During the time, customers would see error messages. All this meant only one thing: there was no time to stop and if something went wrong, it would have to be fixed on the fly.
"The plan was that a set of people would keep the service alive in some way or the other and another set would focus on finding the root cause," said Malviya. At about 11.30 am, the service was permanently fixed.
Meanwhile on social media, customers had gone on an overdrive--complaining mostly that there weren't any great deals as advertised and occasionally about the errors on the website. The next day, Sachin & Binny Bansal apologized to them for not living up to their expectations. Flipkart, however, had created ecommerce history in the country. Goods worth $100 million were sold on that day, the company said.
"With a game of these high stakes, it is easy to cave in but our engineers kept calm and stuck to the playbook," said Malviya. "We did end up stumbling, to be honest," he said.
To be fair, Flipkart's main rival Amazon has had its share of outages too. In January last year, Amazon was reportedly down for 49 minutes and again in August for 25 minutes. According to estimates, the Seattle-based online retailer has over 80 million monthly visitors.
MIXED FEELINGS
He recalls going through a mix of emotions throughout the day, finally being left with a sense of relief and enough lessons for the next few months. "It was excitement, then a tinge of fear, disbelief, a sense of pride and a bit of disappointment," said Malviya.
A week after the Big Billion Day, Malviya and his team are busy analyzing the issues that cropped up and how they can be avoided the next time. He pointed out that having set out with a goal that was too ambitious was probably the right thing to do when it comes to technology. "Setting ambitious goals helps us grow exponentially," he said.
For Malviya, the biggest lesson was to always plan for failure. "If we hadn't done that, the whole day would have been a disaster." While keeping the site running without a hitch is one of the things that takes much up his time, said Malviya, who is investing heavily into what he calls the "data engine," and "systemic intelligence."
"We've just started exploring some of these themes. It is early...but this is very exciting to me," said Malviya, who is a fan of technologist Stephen Wolfram, the maker of Wolfram Alpha and chief architect of Mathematica, a software widely used by mathematicians, scientists and engineers.
A data engine is much like an automotive engine. Only here, data is the new oil. In simple words, the engine needs to be fed with data to be able to generate more business. To build one from scratch is no mean task.
Computational engine Wolfram Alpha, which sifts through data sets that can produce answers to questions such as what was the weather in Bangalore during October 2013, comprises of over 15 million lines of code and uses more than 10,000 processors.
Flipkart has data on its customers (26 million), the products (20 million) it sells and data that come from its supply chain which is one of the largest in the country already.
With customer data, Flipkart can tell if you are a parent or not. Or if you are an impulse buyer or not. The online retailer has 500-1000 data points about each user coming to the site. To use it effectively, the data engine needs to be capable of correlating seemingly discrete information and trigger actions that can make the customer's life easier and help him buy more.
UNDERUTILIZED RESOURCE
Malviya feels that data are an underutilized resource in most businesses including ecommerce. Commerce data include information about customers and their behaviour, their likes and dislikes both offline and online. "We are probably the only company which is marrying offline and online insights," he said.
Marrying product attributes with customer behaviour tells him why a customer did not buy a particular phone or why he did. In all, Flipkart generates between 5-10TB of data every day. That is about a petabyte of data every three months. All the data Flipkart generates goes into what they call the Bigfoot, a single source of data for its different services.
The bigger idea is that of creating systemic intelligence. In other words, creating a system that learns on its own. While intelligent systems aren't easy to make, Flipkart has started taking baby steps towards creating them. Take, for instance, the process of searching for something on the website. "We are currently trying to interpret the query better. That is, what is it that you want to search for," said Malviya.
The idea is to figure out the user's intent behind the search than to just use the text filled in by the user for searching. Malviya has put two of his best men on the job: Kaushik Mukherjee from Yahoo! and Parag Dhanuka, who started his career at Google.
The other areas where Malviya wants to build systemic intelligence are fraud detection and recommendations. Detecting fraud is a big priority for ecommerce and payments companies. For instance, in 2008, PayPal acquired Israeli company Fraud Sciences Ltd $169 million. "People will game the system if I talk too much about it," said Malviya.
As Flipkart grows, the scope to use data grows with it. "The problem space keeps growing. The sheer size of the landscape makes me salivate. It's beautiful," said Malviya, who spends most of his free time learning about new technologies. "We have solved some problems but there are a lot many more to solve," he said.
Unlike in Sholay, in which four people beat Gabbar's army of dacoits, in less than 24 hours, an army of online buyers were going to come crashing through the gates — hoping to lay their hands on a few thousand Xiaomi phones that were on sale. For Flipkart, the company that is up against the techsavvy Amazon, it was more important than ever to prove that they could handle the influx. Hugo Barra, a former Googler who is now a top executive at Xiaomi, was in the room. The flash sales model had worked very well for Xiaomi in other markets and they'd picked Flipkart for the sale in India.
While Flipkart was prepared to handle the spike in traffic, a couple of its core services had buckled under stress the previous week during the flash sale. Amod Malviya, Flipkart's engineering head, and Sameer Nigam, vice-president of engineering, were also present.
At about 8 pm, Bansal asked Malviya if he could rework the selling process for the flash sale that would last a few seconds next day. "It was almost like rewriting all of what we had created in the last three years in 15 hours," recalls Malviya.
Buyers were waiting for the Xiaomi sale for days and the clock was ticking away. Frenzied Flipkart engineers geared up for the last-minute code rejig. And then dawned the big day.
On Tuesday the same day Flipkart announced that it raised a massive $1 billion in funding — thousands of people hit the site at 2 pm to reserve 10,000 Xiaomi phones that were up for grabs. The sale lasted all of 5 seconds. "It seemed surreal that everything was gone. So we sat down to figure out if it was a mistake," recalls Sameer Nigam, vice president of engineering. Thankfully for Flipkart, it had not.
Amod Malviya, the engineering head, climbed on to a desk placed in the middle of the floor and cheered the team while Hugo Barra shot a video of it. But something bigger was around the corner and Malviya knew in the back of his mind that it would be nothing like they'd done before: the Big Billion Day sale on 6 October 2014.
READY FOR THE LONG HAUL
Armed with over a billion dollar in venture capital, Flipkart is readying its engines for the long haul in the Indian market. As smaller competitors snap at its feet, the threat of Amazon's quickly growing muscle in the market also looms on the horizon.
Malviya knows that if there is one thing that will be key to Flipkart's success, it is the technology being built and the team of engineers who are working round the clock to keep the site running.
After finishing school in a small town in Uttar Pradesh, Malviya went to study engineering at IIT- Kharagpur. The engineering college is the alma mater of some of India's best minds — from Sundar Pichai, the rising star at Google, to former RBI governor Duvvuri Subbarao. He graduated in 2002 and moved to Bangalore to work for various technology companies.
In 2010, he joined a 3-year-old startup called Flipkart, barely known outside the book-loving community in Bangalore. The company founded by former Amazon employees Sachin and Binny Bansal had managed to raise $1 million from Accel Partners. Fast forward to 2014, and Flipkart is leading the charge in India's ecommerce revolution. As of August, the site had over 6 million visitors and was shipping over 5 million products a month.
When a user lands up on the company's website, over 150 services need to run in a carefully orchestrated manner to make sure that the user not only buys the product but also gets it on time. Services need to make sure that the user finds what he is looking for, gets great recommendations, pays without a hitch, gets products delivered on time, is able to return a product, track it and that no one is trying to scam Flipkart.
In the technology world, the software architecture that Flipkart uses is called a service-oriented architecture — successfully implemented by its rival Amazon many years ago. The idea is to have different services performing different tasks, drawing from a single database so that it is easier to scale up parts of the whole system without disturbing everything else.
Of the 14,000 people who work for Flipkart, nearly 800 are in the technology team. Six months ago, there were only 400. By March next year, there will be 1,200 engineers working for the company.
Just a few days before the Big Billion Day, Flipkart had moved into a larger building a few kilometers away from its old office. About 500 of the engineers had been working hard to make it happen. Like a startup, there were mattresses, pizzas and round-the-clock activity in the new office.
"There was as much chaos as there was energy," recalls Malviya. The Big Billion Day was at least 10 times bigger than the Xiaomi sale. Once again, a lot of services had to be built from scratch and each of them had to be tested to see if they worked well on a big scale. "We had never done something like this before," said Malviya. The team had identified nearly 80 things that could go wrong. And sure enough, a few did go wrong.
A BILLION HITS
On the day of the sale, which was preceded by heavy marketing by Flipkart, there was a surge in traffic. It spiked at about 8 am and stayed that way until evening. Flipkart said it saw more than a billion hits across its properties.
In a few minutes, one service that was written from scratch had started to rot. In programming parlance, software rot or bit rot is the slow decay of software that eventually renders it useless. "The good thing is that we had a contingency plan. Otherwise, we would have found it difficult to recover from the initial issues," said Malviya.
Around 25-30% of the times, the service which was to act as the single source of truth for all of Flipkart's product data was faulty. During the time, customers would see error messages. All this meant only one thing: there was no time to stop and if something went wrong, it would have to be fixed on the fly.
"The plan was that a set of people would keep the service alive in some way or the other and another set would focus on finding the root cause," said Malviya. At about 11.30 am, the service was permanently fixed.
Meanwhile on social media, customers had gone on an overdrive--complaining mostly that there weren't any great deals as advertised and occasionally about the errors on the website. The next day, Sachin & Binny Bansal apologized to them for not living up to their expectations. Flipkart, however, had created ecommerce history in the country. Goods worth $100 million were sold on that day, the company said.
"With a game of these high stakes, it is easy to cave in but our engineers kept calm and stuck to the playbook," said Malviya. "We did end up stumbling, to be honest," he said.
To be fair, Flipkart's main rival Amazon has had its share of outages too. In January last year, Amazon was reportedly down for 49 minutes and again in August for 25 minutes. According to estimates, the Seattle-based online retailer has over 80 million monthly visitors.
MIXED FEELINGS
He recalls going through a mix of emotions throughout the day, finally being left with a sense of relief and enough lessons for the next few months. "It was excitement, then a tinge of fear, disbelief, a sense of pride and a bit of disappointment," said Malviya.
A week after the Big Billion Day, Malviya and his team are busy analyzing the issues that cropped up and how they can be avoided the next time. He pointed out that having set out with a goal that was too ambitious was probably the right thing to do when it comes to technology. "Setting ambitious goals helps us grow exponentially," he said.
For Malviya, the biggest lesson was to always plan for failure. "If we hadn't done that, the whole day would have been a disaster." While keeping the site running without a hitch is one of the things that takes much up his time, said Malviya, who is investing heavily into what he calls the "data engine," and "systemic intelligence."
"We've just started exploring some of these themes. It is early...but this is very exciting to me," said Malviya, who is a fan of technologist Stephen Wolfram, the maker of Wolfram Alpha and chief architect of Mathematica, a software widely used by mathematicians, scientists and engineers.
A data engine is much like an automotive engine. Only here, data is the new oil. In simple words, the engine needs to be fed with data to be able to generate more business. To build one from scratch is no mean task.
Computational engine Wolfram Alpha, which sifts through data sets that can produce answers to questions such as what was the weather in Bangalore during October 2013, comprises of over 15 million lines of code and uses more than 10,000 processors.
Flipkart has data on its customers (26 million), the products (20 million) it sells and data that come from its supply chain which is one of the largest in the country already.
With customer data, Flipkart can tell if you are a parent or not. Or if you are an impulse buyer or not. The online retailer has 500-1000 data points about each user coming to the site. To use it effectively, the data engine needs to be capable of correlating seemingly discrete information and trigger actions that can make the customer's life easier and help him buy more.
UNDERUTILIZED RESOURCE
Malviya feels that data are an underutilized resource in most businesses including ecommerce. Commerce data include information about customers and their behaviour, their likes and dislikes both offline and online. "We are probably the only company which is marrying offline and online insights," he said.
Marrying product attributes with customer behaviour tells him why a customer did not buy a particular phone or why he did. In all, Flipkart generates between 5-10TB of data every day. That is about a petabyte of data every three months. All the data Flipkart generates goes into what they call the Bigfoot, a single source of data for its different services.
The bigger idea is that of creating systemic intelligence. In other words, creating a system that learns on its own. While intelligent systems aren't easy to make, Flipkart has started taking baby steps towards creating them. Take, for instance, the process of searching for something on the website. "We are currently trying to interpret the query better. That is, what is it that you want to search for," said Malviya.
The idea is to figure out the user's intent behind the search than to just use the text filled in by the user for searching. Malviya has put two of his best men on the job: Kaushik Mukherjee from Yahoo! and Parag Dhanuka, who started his career at Google.
The other areas where Malviya wants to build systemic intelligence are fraud detection and recommendations. Detecting fraud is a big priority for ecommerce and payments companies. For instance, in 2008, PayPal acquired Israeli company Fraud Sciences Ltd $169 million. "People will game the system if I talk too much about it," said Malviya.
As Flipkart grows, the scope to use data grows with it. "The problem space keeps growing. The sheer size of the landscape makes me salivate. It's beautiful," said Malviya, who spends most of his free time learning about new technologies. "We have solved some problems but there are a lot many more to solve," he said.
 |
No comments:
Post a Comment