Source of the article: http://searchsolidstatestorage.techtarget.com/news/450400665/Flash-storage-decreases-build-times-for-Wargaming-DevOps-engineers
Author: Dave Raffo
After builds began taking close to two hours to complete with disk storage, Wargaming installed a Kaminario all-flash array to keep DevOps engineers busy
To bring build times for its DevOps engineers to an acceptable level, online game publisher Wargaming Seattle ditched its hard disks and installed an all-flash array.
Wargaming develops multiplayer online games, including World of Tanks. Wargaming Seattle was previously known as Gas Powered Games before Belarus-based Wargaming acquired the company in 2013.
Calsoft Whitepaper: Amalgamation of 3G Mobile Services, Cloud Services & Storage Technology
Wargaming has around 40 engineers developing games on the company’s Kaminario K2 all-flash array.
Wargaming Seattle’s DevOps engineers use Jenkins software for continuous integration (CI) development. But while working on the company’s next game — the title is a secret to the outside world — the developers experienced build delays that made CI difficult. CI requires developers to add code into a shared repository so they can immediately test and report on the results throughout the day. The process allows for rapid feedback to catch any defect introduced into the code base so it can immediately be rolled back. But Wargaming’s developers found their CI efforts delayed while they waited for builds to complete, sometimes for hours.
Kevin Folks, Wargaming Seattle’s IT manager, said the developers also found themselves waiting longer for data transfers and updates. Wargaming Seattle was using an EMC VNX5300 storage array with all hard disk drives but the performance just wasn’t good enough, Folks said.
“Builds started to creep up to about two hours for a full build,” he said. “We were trying to implement continuous integration so every time someone checks something in on the back end, it automatically creates a new build.
“It started to back up to the point where builds weren’t getting created. So, we had to start doing select builds, which defeats the purpose of continuous integration. We started to look at where the bottlenecks were, and found the largest bottleneck was disk speed. That caused really long read times.”
So, Wargaming’s IT team went hunting for an all-flash array to stoke performance. Folks said they tested NetApp All Flash FAS, EMC XtremIO and Kaminario K2. The team conducted tests for two months, evaluating performance and price.
“Kaminario was the best performer of them all,” he said. “It hit our workloads better than the others.”
Wargaming Seattle put its K2 into production early in 2016. Running the CI system with a cluster of virtual machines and flash storage greatly reduced build times for the DevOps engineers.
“Our latency on K2 is very low,” said Justin Thoemke, Wargaming Seattle’s senior DevOps engineer. “We never spike over four to five milliseconds. We had spikes with the VNX at heaviest build times at 200 milliseconds.”
He said the K2 reduced his developers’ build time to a low of 14 minutes and 51 seconds, from close to two hours with the VNX.
“The engineers were ecstatic,” Thoemke said.
So were the accountants. Folks said the quicker build times save the company a significant amount of money by eliminating wasted engineering time. “An engineer on average builds the game three times a day,” he said. “Waiting two hours for a part to build wastes a couple of millions of dollars in engineer time over the course of the game build, if you look at salaries and the time we save.”
Wargaming started with a K2 array with 7 TB of raw capacity, which stretches to about 24 usable TB with data deduplication.
Thoemke said the flash array also saves time outside of the development process.
“As part of the project we’re working on, we’re spinning up new servers that need to load data quickly into memory,” he said. “We use the K2 to do a fast read of that data so the server’s not stopped.”
Wargaming retained its VNX5300 for archiving and applications that don’t require high performance.
Thoemke said the Wargaming DevOps engineers are exploring using containers, and he would expect K2 to help there. “It would be the same as us launching a game instance that needs to be read with faster IOPS,” he said.
Calsoft DevOps Expertise
Calsoft has in-depth expertise in almost all globally used DevOps tools and technologies. Our DevOps consultants assist our customers in their release and deployment processes by consulting and using best DevOps practices and implementation methodology to produce increased speed, efficiency, better quality and reduced cost.
[Tweet “Flash Storage Decreases Build Times for Wargaming #DevOps Engineers”]