Cloud Automation: PayPal’s OpenStack Secret Sauce
PayPal makes online payments secure and painless for millions of customers per day. But PayPal’s internal customers found the pain of deploying services so great, the company’s operations team knew it had to do something. PayPal is one of the world's largest financial institutions, processing $300,000 in payments every minute, in 25 different currencies, for 128 million active registered customers in 193 markets. Given that kind of scale, you'd think PayPal would be an exemplar of cloud automation — indeed, of all types of automation. But the company was so mired in technical debt, it was having a hard time innovating quickly in an increasingly competitive market. An example: It took more than 100 tickets and hours of meetings to launch even a small service, according to Saran Mandair, senior director of site operations. Under pressure to bring new features to market faster than competitors, PayPal developers demanded higher agility with 99.9999 percent availability. The operations team faced three major challenges:
- Agility at scale. It took three weeks to provision 50 nodes due to manual processes and hand-offs.
- Configuration drift. Development, QA, staging and production environments were all different, causing frequent failures.
- Siloed teams and tools. Deploying a service required hand-offs between systems, network and storage teams, significantly slowing time to production. The systems team was already using some orchestration and automation tools, but these were not portable across network and storage.